In September 2026, 50 realistic backend tasks from finance, healthcare, and insurance practice were given to five frontier models from five vendors, each task twice: once from the plain prompt, once with a 267-word filled Specification Frame in front of it. One attempt per output, no retries. The hypotheses and the results that would have refuted them were written down before the first call. Nine deterministic checkers scored the code, and Bandit, a security scanner that knows nothing about the Frame, scored it independently.
What it found
- Defects in nine named classes: 148 in the bare arm, 23 with the Frame.
- Every model improved. None got worse in any domain. Where the two arms differed, the Frame won 95 of 100 times.
- Bandit, run independently, moved from 53 medium-or-high issues to 11, in the same direction for every model.
- The largest gains came from the models with the weakest unprompted defaults: the Frame supplies the discipline a model lacks.
- Two tasks whose text already carried the constraint showed no Frame effect at all, which is what a specification account predicts.
What it does not claim: it is Python, single turn, nine defect classes, one run per arm. Functional correctness was not scored. Sixteen of the 500 outputs were truncated in transport; a sensitivity analysis excluding them holds for every model, and the paper says so. The paper's threats-to-validity section is the honest version of this list.
The paper
The dataset
Inside the archive: the 100 prompts with checksums, the filled Frame, all 500 model outputs exactly as received, the nine checkers with their unit gauntlet and 50 task fixture pairs, the scoring and statistics code with seeds, the Bandit results, the adjudication table, the run and verifier logs, and the pre-registration with its four dated addenda. License: CC BY 4.0 for data and documents, MIT for code.
Replicate or refute
The archive is built to be re-scored without trusting the author. Unzip it, run python3 tests/test_checks.py (it must end ALL GREEN), then run the scoring and statistics code; every figure in the paper re-derives from the raw files. Re-running the generation itself costs about $20 in API calls and ten minutes per model with the harness in the repository. If you find a defect the checkers missed, a checker that fires wrongly, or a result that does not reproduce, write to sandeep@acuity.press or open an issue in the repository. Refutations will be published in the errata with the same prominence as the result.
The instrument
The Specification Frame is the four-part preamble (role, context, task, constraints) that the book is built on. The exact 267-word filled Frame used in the study is in the archive as spec/FRAME_FILLED.md; a Spec-Kit-compatible constitution and the machine-readable ten laws are in the repository. How the Frame works →