Does the agent add-on help? The only honest answer is: test it on your job.
We ran one minimalism add-on against its own baseline in two coding agents, on the same frozen task, three runs per arm. The two experiments reached opposite conclusions. That is the useful result, and it is also the reason to run the test yourself.
You leave with The five requirements of a defensible add-on test, both measured results side by side, and the protocol to rerun on your own repository
The short version
Twelve runs, two harnesses, one add-on, one frozen fixture. Every run passed the verifier. In Claude Code the add-on was an efficiency win and left the implementation the same size. In Codex it cut the implementation to less than half, and the median run took longer and used more tokens.
Two fixtures cannot rank an add-on, and neither result is a prediction about your repository. What both experiments do supply is a protocol cheap enough to copy.
What a defensible add-on test requires
The published material on agent add-ons is mostly method with almost no results attached to it. Five things separate a test from a testimonial, and every one of them is visible in the two comparisons below.
Setup failures belong in the record too. The Claude setup had two pre-model failures before the measured batch, and the first Codex batch failed closed in all six attempts because its separate code-mode helper was not mounted. Those runs are excluded from the comparison and published anyway, because setup reliability is part of choosing a workflow.
The same add-on, two opposite answers
Both tables are the published medians from three runs per arm. Every figure below is read directly from the comparison records, not restated.
Measured September 3, 2026 — three runs per arm
Claude Code + Opus 5: with vs. without Ponytail
Both arms passed 3/3 and every label-blind artifact scored 10/10. Ponytail lowered median elapsed time and reported list cost on this tiny task; it did not reduce median implementation size.
| Median, n=3 per arm | Claude Code only | + Ponytail 4.8.4 | Delta |
|---|---|---|---|
| Task success | 3/3 | 3/3 | none |
| Artifact score | 10/10 | 10/10 | none |
| Elapsed time | 38.425 s | 27.231 s | -29.1% |
| Turns | 9 | 6 | -33.3% |
| Output tokens | 2,698 | 1,844 | -31.7% |
| Reported list cost | $0.2686005 | $0.191269 | -28.8% |
| Implementation lines | 7 | 7 | none |
Only changed variable: Ponytail 4.8.4 in full mode was mounted for one arm only.
Held constant: Recipe commit 864230d8099302b48c074e4b5cebcba4c5caa79e; Same prompt and fixture hashes; Claude Code 2.1.259 with claude-opus-5 at high effort; Same $1 list-cost cap, tools, permissions, container, verifier, and artifact rubric; Three fresh runs per arm.
The account used a subscription; dollar amounts are Claude Code list estimates. One tiny task cannot establish a universal winner.
Measured September 3, 2026 — three runs per arm
Codex + GPT-5.6-sol: with vs. without Ponytail
Both arms passed 3/3 and every label-blind artifact scored 10/10. Ponytail cut median implementation size from seven to three nonblank lines, but the median run took longer and used more tokens.
| Median, n=3 per arm | Codex only | + Ponytail 4.8.4 | Delta |
|---|---|---|---|
| Task success | 3/3 | 3/3 | none |
| Artifact score | 10/10 | 10/10 | none |
| Elapsed time | 57.055 s | 68.281 s | +19.7% |
| Input tokens | 125,343 | 154,389 | +23.2% |
| Output tokens | 1,420 | 1,632 | +14.9% |
| Command executions | 6 | 7 | +16.7% |
| Implementation lines | 7 | 3 | -57.1% |
Only changed variable: Ponytail 4.8.4 in full mode was added as a repository-local Agent Skill for one arm only; Codex read it in all three runs.
Held constant: Recipe commit 864230d8099302b48c074e4b5cebcba4c5caa79e; Same prompt and fixture hashes; Codex CLI 0.152.1 requesting gpt-5.6-sol at high effort; Same container, permissions, code-mode helper, verifier, and artifact rubric; Three fresh runs per arm.
Codex CLI did not report a dollar cost, so none is estimated. The requested model was gpt-5.6-sol. One tiny task cannot establish a universal winner.
The side-by-side reading of both experiments, including why the tradeoff may have flipped, is in the field note on the two results.
What twelve passing runs do not show
They do not explain the cause
The runs establish that the tradeoff differed between the two harnesses. They do not establish why. Different control loops, different built-in instruction stacks, and a task already close to its natural minimum are hypotheses, not findings.
They do not rank the two harnesses
Each experiment compares an add-on against its own baseline inside one harness. Claude Code reported list-cost estimates and Codex CLI reported none, so no cross-harness cost or speed comparison is made here.
They do not rank the add-on
Two fixtures are two data points. Both comparisons carry the same caution in their own record: one tiny task cannot establish a universal winner.
They do not transfer
Another repository, model, effort level, permission set, or add-on version is a different experiment. Nothing here is a reason to install or remove anything on your repository.
They do not separate the arms on correctness
Every arm passed 3/3 and every label-blind artifact scored 10/10. On this fixture the evidence cannot distinguish the arms on whether the work was right, only on what it cost.
Run the same test on your own repository
The protocol is the reusable part. It costs a few hours and it answers the question for your job rather than for ours.
- Pick a job you actually repeat, and freeze the input plus a checkable acceptance artifact.
- Pin the harness, requested model, effort, permissions, tools, budget, and add-on version.
- Run fresh workspaces in randomized order and repeat each arm; three runs per arm is what made the variance visible here.
- Score correctness and regressions before revealing which arm produced which artifact.
- Keep time, tokens, cost, intervention, and output size as separate numbers; do not collapse them into one score.
The pinned fixture, the deterministic verifier, and both benchmark batches are public, so you can copy the harness before writing your own fixture. The method itself is written up as a free recipe.
Get the next measured run
One maintained workflow at a time. Explicit opt-in, no vendor-paid ranking, unsubscribe any time.