The generation benchmark

“Leading LLM programming language” is an empirical claim: a model receives a task, writes POLYTONE, and ptc test judges — deterministically, no human in the loop. Two numbers per task, both first-class: pass@1 (the first candidate passes) and pass@2e (after a failure the model sees the actual error and gets one repair attempt — POLYTONE’s thesis is that its errors teach).

Methodology

• The judge is ptc test: candidate + hidden tests → one file → pass/fail. The judge never trusts the model — tests are appended after generation.
• Capability-clean: effects appear only through mocks (mock_fs, fixed_rng, fixed_clock, mock_env, mock_http) — a run touches no network, disk, clock, or entropy.
• A CI gate keeps every reference solution green against its own hidden tests — the corpus cannot rot. CI never calls a model; runs are local and BYO-key.
• The report always shows every task × every attempt — no cherry-picking.
• The system prompt is the frozen IDE language card. Since Sprint 252 the harness sends the EVALUATED card — byte-identical with the IDE and the Pro CLI, pinned by a gate; runs 1–7 sent the raw source form (doubled backslashes). Tokens-to-green stays directly comparable only between runs of the same card form.

Results

The first delta proof (replays)

The phase's first failure dataset is the corpus-authoring session itself (Sprint 133) — an LLM writing POLYTONE cold, its first attempts recorded verbatim. Sprint 138 replays them against the hardened toolchain, CI-gated (gen_replays.rs):

• content_tag: failed on the missing Text.slice — now passes verbatim (the language grew to meet the model, Sprint 137).
• json_pluck: the error read like nonsense — now teaches the qualified form (the repair signal pass@2e depends on).
• log_scan: still fails (a semantics miss no compiler error can prevent) — but the assertion diff shows the actual values, and Card v6 teaches the rule.

Fresh pass@1/pass@2e runs against frontier models stay local and BYO-key — this section shows only what has actually run.

The corpus (36)

17 × tier S · 19 × tier M — each task: a prompt, a hidden judge, a reference solution.