The generation benchmark

“Leading LLM programming language” is an empirical claim: a model receives a task, writes POLYTONE, and ptc test judges — deterministically, no human in the loop. Two numbers per task, both first-class: pass@1 (the first candidate passes) and pass@2e (after a failure the model sees the actual error and gets one repair attempt — POLYTONE’s thesis is that its errors teach).

Methodology

• The judge is ptc test: candidate + hidden tests → one file → pass/fail. The judge never trusts the model — tests are appended after generation.
• Capability-clean: effects appear only through mocks (mock_fs, fixed_rng, fixed_clock, mock_env, mock_http) — a run touches no network, disk, clock, or entropy.
• A CI gate keeps every reference solution green against its own hidden tests — the corpus cannot rot. CI never calls a model; runs are local and BYO-key.
• The report always shows every task × every attempt — no cherry-picking.
• The system prompt is the frozen IDE language card. Since Sprint 252 the harness sends the EVALUATED card — byte-identical with the IDE and the Pro CLI, pinned by a gate; runs 1–7 sent the raw source form (doubled backslashes). Tokens-to-green stays directly comparable only between runs of the same card form.
• Tier L (since Sprint 265) is multi-module: the task names every file, the model answers in the IDE loop's fence grammar (one block per file with path=), and the judge verifies EVERY file as its own entry — modules first, then the entry, each with its own hidden tests (the Sprint-244 rule). A path outside the project is dropped, never sanitized; a missing file teaches before ptc runs. One judge path (corpus.mjs) for run.mjs and the harness.
• Comparability across corpus sizes and generators: runs 1–6 measure corpus 33, runs 7–9 corpus 36, run 10 corpus 48 (36 + 12 × tier L), run 11 corpus 52 (48 + the four Sprint-306 tasks), and since Sprint 326 the corpus holds 58 (52 + six everyday tasks over the forms Sprints 319–320 opened — unmeasured until Baseline 12). A headline number compares directly only with runs over the same corpus; a 48-run's line on the 36 older tasks is the number that compares with runs 7–9 (run 10: 33/36 · 35/36 — the run-8 level). Tier L is reported on its own. Runs 1–10 were generated by claude-fable-5; run 11 (Sprint 314) ran on claude-opus-5 because Fable's credits were exhausted — the model is on every record, and numbers compare directly only within one generator (run 11 on the 48 run-10 tasks: 43/48 · 46/48 against Fable's 43/48 · 47/48; on the 36 older tasks 33/36 · 35/36, as run 10).
• Tasks added after the newest run are listed in the corpus table as not yet measured (since Sprint N) — the next run covers them; the coverage gate exempts them until then (the four Sprint-306 tasks trait_travel, culled_parts, model_page, label_card were first measured by run 11: 4/4 cold).

Results

The first delta proof (replays)

The phase's first failure dataset is the corpus-authoring session itself (Sprint 133) — an LLM writing POLYTONE cold, its first attempts recorded verbatim. Sprint 138 replays them against the hardened toolchain, CI-gated (gen_replays.rs):

• content_tag: failed on the missing Text.slice — now passes verbatim (the language grew to meet the model, Sprint 137).
• json_pluck: the error read like nonsense — now teaches the qualified form (the repair signal pass@2e depends on).
• log_scan: still fails (a semantics miss no compiler error can prevent) — but the assertion diff shows the actual values, and Card v6 teaches the rule.

The tier-L delta (Sprints 267–268)

Run 10 (corpus 48) had five first-attempt failures; two of them are the first multi-module replays (pkg_check_attempt1/, tune_mix_attempt1/ — directory form, every file its own entry) and web_toc_attempt2_run10.pt the residual. Sprint 268 rewrote two teaching errors from them (a partial variant pattern shows the exact named form; an element method on a list names the way to an element) and measured COLD over the same five tasks: 3/5 pass@1 (0/5 in run 10). The residual named the last gap — there was no way to write an arm that does nothing — so pass became the empty statement; the same candidates re-judged under it read 4/5, the repair round 5/5. All three replays are pinned in gen_replays.rs.

The run-11 dataset (Sprint 314)

Run 11 (corpus 52, claude-opus-5) had five first-attempt failures, three healed in one round (audio_probe sample_rate for the listed rate — run 10's class on a different model; json_pluck fields for entries — the run-9 residual, named by the error; model_parts shape lines without their indentation — the codec names the rule). The two residuals are ONE class, stacked twice: both TOML tasks first invented the shape of toml.parse (a document with .tables; a Value.Table variant), the repair fixed exactly that and hit the second error behind it — the guessed field entries on toml.Table (pairs). Two stacked errors cost the task under one repair round. Five replays (config_port_attempt{1,2}_run11.pt, pkg_check_attempt{1,2}_run11/, model_parts_attempt1_run11/) are pinned in gen_replays.rs.

The hardening from the run-11 dataset (Sprint 315)

The stacked class got two answers: a field read on a List[Record] now lists the ELEMENT's fields and the way to one (the Sprint-268 method error's twin) — the second error is visible in the first round; the task texts config_port/pkg_check name the shape of toml.parse the way the 266 corrections name every import, model_parts shows its shape line indented. Measured cold over the same five tasks (generator claude-opus-5 = run 11's): 3/5 pass@1 (0/5 in run 11), the repair round 5/5; the two untouched tasks failed on their exact one-round classes (stdlib field names the error names — no card candidate). The three cold candidates are pinned verbatim (*_cold_run11delta).

Fresh pass@1/pass@2e runs against frontier models stay local and BYO-key — this section shows only what has actually run.

The corpus (58)

21 × tier S · 21 × tier M · 16 × tier L (multi-module: one file per module, each verified as its own entry) — each task: a prompt, a hidden judge, a reference solution.