Measured September 3, 2026 — three runs per arm
Codex + GPT-5.6-sol: with vs. without Ponytail
Does the same minimalism add-on change correctness, implementation size, and execution efficiency in Codex?
Held constant
- Recipe commit 864230d8099302b48c074e4b5cebcba4c5caa79e
- Same prompt and fixture hashes
- Codex CLI 0.152.1 requesting gpt-5.6-sol at high effort
- Same container, permissions, code-mode helper, verifier, and artifact rubric
- Three fresh runs per arm
Only changed variable
Ponytail 4.8.4 in full mode was added as a repository-local Agent Skill for one arm only; Codex read it in all three runs.
Measures
- Acceptance-test success
- Regressions
- Human intervention
- Elapsed time
- Reported tokens
- Command executions
- Lines and dependencies added
Measured result
Both arms passed 3/3 and every label-blind artifact scored 10/10. Ponytail cut median implementation size from seven to three nonblank lines, but the median run took longer and used more tokens.
| Median, n=3 per arm | Codex only | + Ponytail 4.8.4 | Delta |
|---|---|---|---|
| Task success | 3/3 | 3/3 | none |
| Artifact score | 10/10 | 10/10 | none |
| Elapsed time | 57.055 s | 68.281 s | +19.7% |
| Input tokens | 125,343 | 154,389 | +23.2% |
| Output tokens | 1,420 | 1,632 | +14.9% |
| Command executions | 6 | 7 | +16.7% |
| Implementation lines | 7 | 3 | -57.1% |
Codex CLI did not report a dollar cost, so none is estimated. The requested model was gpt-5.6-sol. One tiny task cannot establish a universal winner.
Inspect the protocol and exact artifacts ↗