Files
ai-agent/docs/usage-calibration.md
2026-10-13 00:13:55 +02:00

137 lines
7.5 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# What 1 % of a Claude plan window is worth
Claude Code's `/usage` shows how full the subscription's windows are (rolling
5 h, 7-day week), as whole percentages. The transcripts show what was spent, in
tokens. `scripts/usage-calibration.py` lines the two up and fits the exchange
rate, so a dollar figure on the dashboard can be read as "% of my week", and
the plan's ceiling can be expressed in tokens.
```bash
uv run scripts/usage-calibration.py # personal seat, last 9 days
uv run scripts/usage-calibration.py --since 2026-09-13 --json /tmp/cal.json
uv run scripts/usage-calibration.py --account work --prefix sensor.claude_work_usage
```
## Inputs
- **Usage %**: Home Assistant long-term statistics (hourly min/max) of the
sensors that the homelab's `claude-usage-notifier` pushes (`<prefix>_5h`,
`_week`, `_fable`). The script reads the recorder SQLite db read-only, so it
goes back to the sensors' creation, beyond the 10-day state history.
- **Tokens**: the account's conversations listed by `/api/conversations?account=`
(the backend resolves each conversation's account), parsed from the archive,
subagents included. Records are de-duplicated on (message id, request id) and
priced at **list price** (the script's own table; see "Pricing" below).
## Method
The data is binned by hour. A window's % only rises between resets, so each
bin's increase is the usage spent in it. Bins that contain a reset are dropped,
and so are 5 h bins above 100 % (overage). The fit is `Δ% = k·cost + c`, by
non-negative least squares. `c` soaks up usage the archive can't see
(claude.ai, the desktop app, a Mac that isn't paired as a worker). It came out
at ≈ 0 once both Macs were paired.
Unweighted list-price dollars leave a drift. Through a Fable-heavy week the
cumulative $/% climbed from $13.5 to $20 as the mix moved to Opus. A **Fable
multiplier** removes it. It is chosen by grid search on the combined R² of the
5 h and week fits, and with it the cumulative ratio stays flat at $21–23/%.
Other weightings were tested and lost: cache reads free, cache reads at 0.1×
input, cache reads at 1× input, Opus 5.5 priced as Opus 5, and raw token
counts.
Use 1 h bins. With 3 h bins most 5 h resets fall inside a bin, which then gets
dropped, and the 5 h fit collapses.
## Calibration — personal Max plan (€200/month), 2026-10-12
Window: 2026-10-04 18:00 → 2026-10-12 (Opus 5.5 / Fable 5.1 era), 1 h bins.
Intervals are 90 % CIs from a 300-draw day-block bootstrap.
| | API $ at list price | tokens at the real Opus 5.5 mix¹ |
|---|---|---|
| 1 % of the **5 h** window | **$5.6** [5.4–6.8] | ≈ 17 Mtok |
| 1 % of the **week** | **$22.1** [21.1–24.6] | ≈ 68 Mtok |
| Fable multiplier | **×1.75** [1.6–2.5]: 1 % week ≈ $13 of Fable 5.1 | ≈ 14 Mtok of Fable¹ |
| full 5 h window | ≈ $560 ≈ 25 % of a week | |
| full week | ≈ $2,200 | ≈ 6.8 Gtok |
| 100 % of every week for a month (×4.35) | ≈ $9,600 (≈ $5,700 if all Fable) | ≈ 30 Gtok |
¹ Agentic Claude Code traffic is 98 % cache reads by tokens. On Opus 5.5 they
are about 61 % of the dollars, cache writes 32 % and output 8 %. The same 1 % of
the week buys ≈ 1.1 M Opus 5.5 output tokens, or 5.5 M fresh input, or 110 M
cache reads.
- **No separate Fable bucket.** The `seven_day_<model>` key never appeared in
the status line's `rate_limits` from 2026-07-17 to 2026-10-12, despite heavy
Fable 5 and Fable 5.1 use. Fable draws on the same 5 h and week windows, about
1.75× faster per list-price dollar than Opus 5.5.
- **The week binds, not the 5 h window.** A full 5 h window is ≈ 25 % of the
week, and a week has 33.6 of them.
- **Earlier eras differ.** In Jul–Aug (Opus 4.8 / Opus 5 / Fable 5), unweighted
fits gave ≈ $30 per 1 % of the week. Limits and prices move with model
launches, so re-run the script after one.
## Plan units: what the limits actually count
Re-fitting the whole history (Jul → Oct 2026) per model version shows the
*token* allowance never moved — only its list-price value did. When every new
model is metered **at its predecessor's price** (Opus 5.5 as Opus 5, Fable 5.1
as Fable 5, Sonnet 5.5 as Sonnet 5, Haiku 5.5 as Haiku 4.5) and Fable/Mythos
tokens weigh **×1.75**, one constant explains July, August and October alike:
≈ 42–52 *plan units* per 1 % of the personal week, ≈ 8–12 per 1 % of 5 h. That
is `pricing.plan_units`. Effort changes how many tokens a turn spends, not the
rate (the per-effort fits all overlap). The work seat meters the same way with
smaller buckets: ≈ 24 units per 1 % of its week (½ the personal), ≈ 4 per 1 %
of its 5 h (⅓).
| version | used | list $ per 1 % week | list $ per 1 % 5h | Mtok per 1 % week |
|---|---|---|---|---|
| Opus 4.8 | Jul | 35 | 6.8 | 46 |
| Opus 5 | Jul–Oct | 42 | 9.1 | 59 |
| Opus 5.5 | Sep → | 22 | 5.6 | 69 |
| Fable 5 | Jul–Aug | 26 | 5.2 | 16 |
| Fable 5.1 | Sep → | 14 | 3.6 | 16 |
| Sonnet 5 | Jul–Oct | 36 | 6.9 | 123 |
## In the app (since 2026-10-12)
- **Exact prices.** `backend/conversations/pricing.py` keeps a per-version price
book (`MODEL_PRICES`, stamped `PRICE_BOOK` on every summary as `priceBook`):
each model's own cache-read rate, 5 m / 1 h cache writes, fast mode ×2, Haiku
5.5's >100K-prompt tier. Unknown ids price as the newest of their family.
- **One usage per API call.** Claude Code writes an assistant message as one
record per content block, each repeating the call's full `usage`; the parser
used to sum them all (tokens ≈ 1.93× too high). It now prices a call once,
keyed on (message id, request id), and re-homes that charge onto the tool the
call drove. With the old family prices on top, the dashboard used to read
≈ 3.7× list price.
- **Per conversation**: `planUnits`, `planPct` (`{week, h5}` — the units over
the account's calibration in force), `byModel` (per model id: turns, tokens,
list $, plan units, usage, and the same per `--effort` level), `meterBins`
(epoch hour → [list $, plan units]), `fastCalls`.
- **The official % is recorded by the backend** (`backend/usage/`): every change
of each account's `rate-limits.json` (mounted read-only from
`~/.claude{,-work}/usage-state`, `USAGE_STATE_DIRS`), plus a 30 min
heartbeat, kept forever in `usage_limits`. History before that came from HA:
`scripts/usage-backfill-ha.py` posts HA's hourly statistics to
`/api/usage/import` (idempotent).
- **Daily recalibration** (`usage/calibrate.py`): `Δ% = units/k + c` over the
last 14 days, hourly; a fit becomes the rate in force only when it is
*trusted* (≥ 24 busy hours, R² ≥ 0.5, within ×3 of the seed rate — usage on an
unpaired machine in the same hours as lab work fits cleanly at a rate 10×
off). A trusted move of more than 20 % pushes a notification (a model launch
or a limits change). `POST /api/usage/calibrate {"backfill": true}` adds the
weekly fits over the whole history (the *$ per 1 % over time* chart).
- **Untracked usage**: per day, the observed rise minus what the archived spend
predicts — claude.ai, the desktop app, a machine that isn't paired.
- **Where it shows**: the conversation card chip (`≈0.4% wk`), the Details
drawer's *Plan usage* card (per-model / per-effort table), the Conversations
dashboard (spend by model version, $ per 1 % over time), the *Plan usage*
dashboard (`/dashboards/plan`: gauges, history, observed vs predicted,
calibration, recalibrate), and every `--final` notification (` · $3.20 ·
≈0.4% wk`).
`scripts/usage-calibration.py` stays as the offline cross-check (pandas/scipy,
per-version fits, bootstrap).