Why We Stopped Updating Docs After Every Frontier Model Change
Our evaluation loop moved from static snapshots to rolling model comparisons with per-run spend and output-quality deltas.
Published 2026-06-27
Why We Stopped Updating Docs After Every Frontier Model Change
TL;DR: We generate evaluation documents with reproducible per-run deltas so model changes don’t force full rewrites.
The Context
We run hands-on comparisons for every tool under evaluation. With models changing rates and tiers weekly, manual rewrites became an entire week’s work each cycle.
What We Tested
| Approach | Verdict | Why |
|---|---|---|
| Static quarterly rewrites | ❌ | Already stale when updates hit |
| Rolling per-run deltas | ✅ | One template, repeatable arithmetic |
The Pivot Point
A 04:00 UTC rate change invalidated three published comparisons within 24 hours. That was the last time we shipped snapshot docs for a volatile model tier.
What We Use Now
We compute spend deltas from webhook-level exports and insert them via template field substitution. Docs still reflect the latest view without rewriting prose.
When You’d Choose Differently
If the tool stack is stable and vendor changes are rare, classic rewrites work fine. Our environment is not that environment.
Tool Crucible Rating
7 / 10 — Overall 8 / 10 — Ease 7 / 10 — Value 7 / 10 — Support
This is part of our AI dev tools evaluation series. See full comparison: [link]
Last reviewed 2026-06-27. See our methodology and affiliate policy.