How this site was built

spec → tests → capture → verify · July 10–12, 2026

Three apps, one Worker each, one repo: this hub, a model arena, and a daily guessing game. The default everywhere is replay. Recorded model output ships as fixtures checked into the repo, live calls are the exception, and the gate below decides who gets one. Every cost figure on the site comes from the provider’s reported usage on a recording; when a number renders, a capture log backs it.

The pipeline

Spec first, then tests, then one capture pass, then verification. The tooling is part of the product:

• A drift alarm re-scores every recorded answer, recomputes its cost from raw usage, and fails when either budges. A scorer edit can’t quietly rewrite an old result.
• The winner rule, correct → cheapest → fastest, is enforced when fixtures load. Not just written down; checked.
• A banned-words lint (38 terms, word-boundary) fails the build when marketing vocabulary reaches public copy. This page had to pass it.
• A commit-message hook keeps tool sign-offs and generated-with boilerplate out of the git log.

Fixtures are captured once and never hand-edited. The game’s 18 candidate prompts were cut to 15 by rules written down before the capture ran; the kill log ships in the repo.

The live-run gate

A live run on the arena walks this list, in order. Any failed check lands on replay.

What the build cost

The thesis, applied to the site itself. Two capture passes on July 12 recorded every fixture served here, and this receipt is the entire model-API bill for the content you can see.

Three recordings worth naming

• sort-second: three models, one identical wrong answer. All alphabetized by first letter when asked for second, and returned the same eight words, character for character. Nobody won.
• letter-count: the answers came back 6, 11, 8, and 15. Only 15 was right, and it came from the lane that spent 215 output tokens getting there.
• numbers-1-50: the cheapest lane was right. The lane costing 74× as much spent its whole 400-token budget thinking and never reached fifty.

[fact pending]

How the work split between me and the models is on the about page.

Check the receipts yourself

The token counts on this site are the providers’ own — lifted from each response stream, not re-counted here. The dollar figures are those counts times the per-million prices OpenAI and Anthropic publish, divided out in one small function; a drift alarm recomputes every recorded cost from raw usage at each build and fails if a figure moved. Estimates are labeled as estimates. Below is the code that does the counting, then the tally it produces.

The functions that read tokens and price them

The same interpreters run in the capture harness and on the live path. Token counts come off the provider’s own stream; the dollar line is arithmetic on the published price. These blocks are quoted from the repo, and a build-time check fails if any of them drifts from the source.

  // Arrives on the final chunk when stream_options.include_usage is set.
  const usage = o["usage"] as Record<string, unknown> | null | undefined;
  if (usage && typeof usage === "object") {
    if (typeof usage["prompt_tokens"] === "number") update.usageIn = usage["prompt_tokens"];
    if (typeof usage["completion_tokens"] === "number") update.usageOut = usage["completion_tokens"];
  }

    case "message_start": {
      const usage = ((o["message"] as Record<string, unknown> | undefined)?.["usage"] ?? null) as
        | Record<string, unknown>
        | null;
      if (usage && typeof usage["input_tokens"] === "number") update.usageIn = usage["input_tokens"];
      break;
    }

    case "message_delta": {
      const delta = (o["delta"] ?? null) as Record<string, unknown> | null;
      if (delta && typeof delta["stop_reason"] === "string") update.stopReason = delta["stop_reason"];
      const usage = (o["usage"] ?? null) as Record<string, unknown> | null;
      if (usage && typeof usage["output_tokens"] === "number") update.usageOut = usage["output_tokens"];

const laneCost = (code: ModelCode, inTok: number, outTok: number): number => {
  const m = MODELS_JSON.models[code];
  return (inTok * m.in_per_mtok + outTok * m.out_per_mtok) / 1_000_000;
};

All ten races, from the committed recordings

This table is built at render time from the same fixture files the arena replays — the numbers are computed from the recordings, not typed in. Across the ten, the cheapest lane won 7 of 10, with cost spreads from 13× to 648×. For ordinary tasks, cheaper usually suffices; the premium lane earns its price on the hard ones. The letter-count race is that case — the priciest lane was the one that returned the right answer — and sort-second is the null, where every lane failed alike and the winner reads nobody.

Winner recomputed from each recording by the arithmetic rule (correct → cheapest → fastest); spread is the priciest lane’s cost over the cheapest; verdict counts are the recorded pass/fail per lane. Full timelines and prompts: the arena.