The best model is the one that ships your work.

Measured — flagship-default A/B · verified August 2026

Personal CFO build — which model should be the default?

Scored on Price 50% · Quality 30% · Speed 20%

One spec, three models, same harness (srooter-agent, pinned, iterate-to-pass): a Personal CFO app — email login, income/expense tracking, and a conversational CFO that answers “can I afford this?” with a grounded, machine-checkable verdict. Every model shipped a working 8/8 app, so the ranking is the operator’s price×quality×speed score, not pass/fail. Cost is tokens-to-first-8/8 × list price.

• All three ship a working 8/8 app — so the score, not pass/fail, decides. Under the operator’s Price 50 / Quality 30 / Speed 20 weighting the three land within ~2.5 points: co-equal, glm-5.3 slightly ahead.
• glm-5.3 is the new default (ADR-079): it’s the cheapest of the three AND one-shot the build in a single iteration, where glm-5.2 needed two. This is the direct test that confirmed the flip — glm-5.3 measurably beats the model it replaced.
• No free lunch on speed: kimi-k3 builds 2× faster (9 vs 19 min) but at ~5.5× the per-token price, so it lands costlier overall. Route for speed → kimi-k3; for cost-per-shipped-build → glm-5.3.

Caveat. Single run per model (n=1) — token counts vary run-to-run, so read this as “co-equal, glm-5.3 ahead,” not a blowout. glm-5.3 is priced at glm-5.2’s blended tier ($0.0011/1k) pending confirmation of Z.ai’s flagship list price; since price is 50% of the score, a higher real price would tighten the gap.

Methodology. Same agent-build harness as the July study, on the isolated bench org: srooter-agent pinned via x-srooter-pin so the gateway served each model verbatim; iterate-to-pass against an 8-point Playwright acceptance suite (login → income → expense → month summary → affordability verdict), capped. Score = 0.5·price + 0.3·quality + 0.2·speed, each normalised to the best run; cost = tokens-to-first-8/8 × config list price. August 2026.

Measured — agent-build benchmark · verified July 2026

Nine-model coding build-off

One agent, one full app — email login, a three-panelist "council" plus synthesis, persistent threads, and full-text search. Two axes: (1) 8 browser acceptance checks (does it work?), (2) a hand-graded 0–5 polish score of the finished build (is it any good?). Eight of nine passed all 8 checks, so pass/fail saturates — polish is what separates them. Each model pinned (served verbatim, not routed); iterate-to-pass, capped. Times are wall-clock; tokens are output. Rows sorted by polish, then efficiency.

• Six of eight built a working app to a perfect 8/8 — so pass/fail can’t rank them. We hand-graded each finished build 0–5 on layout, role design, and graceful-degradation. That’s where the real spread lives.
• Polish tracks model class: Opus, Fable, and Sol shipped 5/5 builds (color-coded panelist roles, graceful “no API key” fallbacks); DeepSeek and Kimi passed the same checks but shipped barer UIs (3/5, 2/5). The 8/8 ceiling hid exactly this difference.
• The value leader is new: Kimi K3 hits the same 5/5 polish as Opus / Fable / Sol, but at ~$3.60 / build — half of Opus, a third of Fable — and reached 8/8 in the fewest iterations (2). Its one trade-off is speed (1208s; the reasoning-max flagship is slower than Opus’s 652s). No single winner — route top-craft where quality matters to K3 (cheapest 5/5) or Opus (fastest 5/5), and high volume to fast-and-good-enough (DeepSeek / GPT-5.5).

Methodology. A distinct July 2026 run — nine models incl. Kimi K3 (vs the June “Five models” block below) on an improved harness: empty-turn failover + tool-loop fixes since resolved June’s GPT-5.5 build failure, and a per-iteration watchdog + port cleanup were added. Each model was pinned (served verbatim, not routed); iterate-to-pass capped at 5 rounds; times are wall-clock, tokens are output. The 0–5 polish score is a subjective, hand-graded review of each finished build — layout, panelist-role design, and graceful degradation on missing config — from identical-viewport screenshots + source, not an automated metric. Acceptance is run-to-run variable on the cosmetic panelist-count check (why Opus and GLM-5.2 can land 7/8 one run, 8/8 another). These are this run’s numbers and do not supersede the separate June block.

The pricing view · cost to build the same app

The pricing view — what polish costs

Same app, cost to build it on each model at list price (constant ~1M input + 40k output — the build-off average). Pips are the hand-graded polish. The question a router answers per request: the cheapest model that clears your quality bar.

Kimi K3 reset the top tier: same 5/5 polish as Opus / Fable / Sol at $3.60 / build — the cheapest 5-star of the nine (half of Opus, a third of Fable). The other flagships are a price band for a slower or pricier path to the same bar: gpt-5.6-sol costs the same as Opus but runs 3–4× slower; gpt-5.5 charges flagship rates for a 4/5 build. Pick the cheapest model that clears your bar — that routing, per request, is srooter.

List price, uncached. Real agentic builds cache the input prefix (~10× cheaper in), so absolute $ run lower — the ordering holds. GLM & Kimi shown at the gateway’s blended config rate (no public in/out split).

Measured — first-party benchmark · verified June 2026

One-shot full-app build

Build a complete app in a single autonomous pass: email login, a multi-perspective "council" answering with three panelists plus a synthesis, persistent threads, and full-text search. Same spec, same harness; measured on quality, cost, and speed.

• Per-token price is the wrong number: Fable-5 costs ~2× Opus per token but used ~2.3× fewer tokens — identical total (~$3), 25% faster. Optimize tokens-to-done.
• srooter delivered the same 8/8, build-passing app through governed cheap-model routing at roughly a half to a third of the direct-frontier cost — trading wall-clock speed for cost + governance (budgets, audit, policy).
• srooter cost is measured at the gateway audit: client-side meters price routed backends at the requested model’s rates and can overstate real spend by 5–10×.

Methodology. One spec, one harness: each configuration got a single autonomous `claude -p` run (no retries, no human help) against the same written spec with 8 acceptance criteria. Quality was verified by a headless Playwright E2E driving every criterion in a real browser plus `npm run build`; cost and tokens come from the run artifacts and the gateway audit; UX was reviewed from identical-viewport screenshots. June 2026.

Measured — multi-model benchmark · verified June 2026

Five models, same app — coding & routing tiers

Every model built the same Next.js 15 + TypeScript + Tailwind app ("Council Brainstorm") end-to-end through srooter-agent — our own CLI — iterating until an 8-point Playwright acceptance suite passed (capped at 4 rounds). Models were pinned so the gateway served each one verbatim. Scored on accuracy, speed, output tokens, cost, and UI polish.

Coding tier — full app build

Routing tier — small bounded tasks (all 100% correct → speed decides)

• No single "best" model: GLM-5.2 won on build quality (only 8/8 with the cleanest UI), DeepSeek-V4 Pro won on value (also 8/8, 2.9× faster, lowest cost), Cerebras won the routing tier on speed. That spread is the entire reason a routing layer exists.
• The cheaper model won outright: GLM-5.2 built the same app and scored higher than Claude Opus (8/8 vs 7/8) — at a small fraction of the cost. On a like-for-like basis (GLM’s input tokens weren’t metered, so its $0.12 is output-only) that’s ~25× cheaper than Opus’s ≈$25.73; on output tokens alone, far more. Per-token sticker price is the wrong number.
• GPT-5.5 failed through our converted serving path (empty turns); with a model pinned, the gateway’s empty-turn failover is off, so it surfaced raw — keep failover on for it. A real reliability finding, not a billing blip.

Methodology. Run entirely on our own stack — srooter gateway + srooter-agent CLI (dogfooded). Accuracy via Playwright (8 acceptance criteria); UI polish judged from identical-viewport screenshots. Cost is list-price-from-tokens (the truer number is the gateway audit); some models’ input tokens weren’t metered by the client, so their cost is output-only and understated — compare directionally, not to the cent. Models pinned via x-srooter-pin so each served verbatim instead of being routed. June 2026.

Use-case × model fit

Models reference

The models srooter routes across, with versions and where each shines. SWE-bench Verified figures are external evidence (approximate — see note).