The baseline results (Haiku + Copilot was the only one that improved with the skill included in the agents context).
Original skill vs. no skill
• Claude Code / Sonnet: time +7.6% | tokens +17.7%
• Codex / Sol: time +11.8% | tokens unknown*
• Claude Code / Haiku: time +6.2% | tokens +14.1%
• Codex / Luna: time +27.0% | tokens +75.8%
• Pi / Sol: time +10.4% | tokens +25.8%
• Pi / Luna: time +21.7% | tokens +31.1%
• Copilot / Luna: time +21.3% | tokens +45.2%
• Copilot / Haiku: time −18.7% | tokens −14.8%
Could a small revision improve these results?
We increased the difficulty of the tasks assigned to these agents in order to get some concrete failures and traces from those failures, and then revised the skill to emphasize doing an additional check on db migrations
Compared with the original skill in that revision round:
• Tokens decreased in 3 of 4 pairings.
• Time decreased in 3 of 4 pairings.
Revision 1 vs. original skill — separate revision round
• Codex / Luna: time −2.6% | tokens −3.4%
• Claude Code / Haiku: time −3.4% | tokens +11.9%
• Copilot / Haiku: time −8.6% | tokens −9.1%
• Pi / Sol: time +7.8% | tokens −0.1%
As always, this emphasizes that context matters when working with agents, and not to just blindly add skills to your agentic workflows.
The baseline results (Haiku + Copilot was the only one that improved with the skill included in the agents context).
Original skill vs. no skill • Claude Code / Sonnet: time +7.6% | tokens +17.7% • Codex / Sol: time +11.8% | tokens unknown* • Claude Code / Haiku: time +6.2% | tokens +14.1% • Codex / Luna: time +27.0% | tokens +75.8% • Pi / Sol: time +10.4% | tokens +25.8% • Pi / Luna: time +21.7% | tokens +31.1% • Copilot / Luna: time +21.3% | tokens +45.2% • Copilot / Haiku: time −18.7% | tokens −14.8%
Could a small revision improve these results?
We increased the difficulty of the tasks assigned to these agents in order to get some concrete failures and traces from those failures, and then revised the skill to emphasize doing an additional check on db migrations
Compared with the original skill in that revision round: • Tokens decreased in 3 of 4 pairings. • Time decreased in 3 of 4 pairings.
Revision 1 vs. original skill — separate revision round • Codex / Luna: time −2.6% | tokens −3.4% • Claude Code / Haiku: time −3.4% | tokens +11.9% • Copilot / Haiku: time −8.6% | tokens −9.1% • Pi / Sol: time +7.8% | tokens −0.1%
As always, this emphasizes that context matters when working with agents, and not to just blindly add skills to your agentic workflows.