ai-agent/CLAUDE.md

# CLAUDE.md — `ai-agent`

Guidance for Claude Code when working in this repo. It was extracted from the
homelab monorepo (formerly `services/ai-agent/`, full history preserved); the
homelab keeps a deployment shim at `~/homelab/services/ai-agent/` (compose with
`build.context` pointing here, traefik.yml, `deploy.sh`, the live `data/`
volume), and the homelab root `CLAUDE.md` conventions (worktrees, Traefik copy
step, commit/notify) still apply when deploying there.

## North star

This service is headed somewhere bigger than a homelab dashboard: **the
open-source, intuitive, private and secure agent layer of any computer, to
build and host websites and interact with data** — Lovable, but open source
and self-hosted. Zipgo under the hood for hosting, shipped as a **single
Docker container**, with self-update handled **inside the container** (not by
host scripts). See [GOAL.md](GOAL.md) for the full vision and the gap list.
When making changes here, prefer container-internal solutions, generic config
over homelab-hardcoded paths, and zipgo over bespoke hosting.

## What this is

A viewer/editor + analytics dashboard for the homelab's **Claude context**, served
as an installable PWA at `https://ai-agent.lab.gabvdl.xyz` (behind Authelia). It:

- exposes every `CLAUDE.md` and the `.claude/` tree (skills, hooks, settings) as a
  flat, editable file list with **real** Claude token counts + dollar costs, plus
  the repo's root `data/` tree (plans, logs, notes, kanban) surfaced **read-only**;
- browses the archived Claude Code **conversation transcripts** (thread view with
  per-turn usage/cost, skill-usage analytics, spawn/resume/interrupt live runs);
- catalogs **~/projects** (mounted at `/workspace/projects`; gallery + detail) and the repo's **services/** (static
  parse of compose + Traefik configs — never Docker itself);
- surfaces the assistant's **memories** and the scaffolding **templates**;
- streams updates over SSE so the UI refreshes without a manual reload.

It is a **viewer**: it reads the repo context (mostly read-only mounts) and does
static parsing. It never drives Docker or mutates infrastructure. The only writes
are file edits into the mounted `.claude/` + `~/projects` trees and its own SQLite
store / metadata sidecar.

## Layout

- `backend/` — FastAPI app (`main:app`, port 8080). Serves `/api/*` and the built
  PWA from `STATIC_DIR` (SPA catch-all at the end of `main.py`).
- `frontend/` — Vite + React + TS + Tailwind PWA. Built in the Dockerfile's first
  stage into `/app/static`.
- `sidecar/` — a **host-side** FastAPI process (NOT in the container) that launches
  `claude -p` sessions. See below.
- `Dockerfile` — 2 stages: build the React PWA, then a `python:3.12-slim` image
  running `uvicorn`.
- `docker-compose.standalone.yml` — the generic, no-host-mounts shape (see
  "Standalone mode" below). The homelab's compose service (profile `dev`,
  container `ai-agent`, host-only port `127.0.0.1:8096:8080`, network `main`),
  its Traefik router (`ai-agent.lab.gabvdl.xyz`, `auth-chain`) and the
  blue-green `deploy.sh` + `deploy.override.yml` live in the homelab repo's
  `services/ai-agent/` shim.
- `PROJECT_CLAUDE.md` — the **generic, host-agnostic** project working conventions
  (worktree workflow, artefacts, plans, committing, the notify+`DONE` finish
  signal), split out of the homelab root `CLAUDE.md` so the ai-agent can ship it
  as the starting `CLAUDE.md` for a new project. Not this service's own guidance
  (that's this file) — it's a shippable template with `{{…}}` placeholders.
- `data/` — SQLite DB, transcript archive, metadata sidecar, `ui-state.json`
  (bind-mounted, gitignored).

### Backend modules

- `main.py` — FastAPI app, all routes, env/path wiring, startup (indexer + watcher).
- `db.py` — SQLite store (WAL): `files` (content hash + real token count + cost +
  word/char stats, computed lazily only when a file's hash changes), `skills`,
  `transcripts` bookkeeping. Decoded transcript summaries are cached in-process
  (`all_summaries()`/`get_summary()` return **read-only** dicts — copy before
  mutating) and `summaries_version` bumps on change so request-level memoizers
  (e.g. the project cost rollup) know when to recompute.
- `indexer.py` — background daemon: token/cost accounting (via the `count_tokens`
  endpoint, debounced) + skill-usage mining from transcripts. Pricing consts here.
- `conversations.py` — parse `*.jsonl` transcripts. `ParserState` is a **resumable**
  parser (`feed(record)` + `summary()`); `parse_conversation(path, full=…)` is the
  one-shot wrapper (cheap rollup vs. full thread). Real per-turn usage/cost from
  `usage` blocks, plus the per-item metadata the viewer shows on each row:
  `durationMs` (a tool call's run time — call → its result record; on a message,
  the assistant API call's latency), `model`, `toolUseId`, `resultChars`/`resultLines`.
- `fsscan.py` — `os.scandir`-based directory walking. The poll loops re-walk
  thousands of files every couple of seconds; `pathlib.rglob` allocates a `Path`
  per entry and was a large chunk of idle CPU. Use this, not `rglob`, in any loop.
- `githist.py` — cached git history. `BucketedHistory` walks a repo's log **once**
  (`git log --name-only -- <prefix>`) and buckets commits per directory, keyed on
  the repo HEAD; `RepoLogCache` TTL-caches standalone project repos. Before this,
  `/api/services` forked two `git log`s per service **per request** (11 s).
- `events.py` — `Hub` (SSE pub/sub) + `Watcher` (polls meta + live transcripts,
  re-indexes, publishes `meta`/`transcript` events).
- `meta.py` — per-conversation action metadata JSON sidecar (projects, state,
  committed/pushed/merged/deployed/notified timestamps), keyed by session id.
- `cron.py` — cron jobs: scheduled agent sessions (Settings → Cron). A job =
  a 5-field cron schedule (dependency-free matcher, container-local time —
  `TZ` in compose) + a prompt file under `.claude/agents/` whose content
  (frontmatter stripped) is spawned as a session via the normal sidecar path;
  each firing is recorded in the job's `history` (`{timestamp: sessionId}`)
  and the conversation is tagged (`meta.cron` → violet badge linking back to
  the job). **How a job runs is declared in that file's frontmatter** — see
  below. Store is `/data/cron-jobs.json` (seeded with the every-5h
  `goal-keeper` job on first run); the scheduler thread claims each fire
  minute through the store, so the deploy-cutover window where two backends
  share `/data` can't double-spawn. REST under `/api/cron` (+ `/run`,
  `/prompt`); mutations and fires publish a `cron` SSE event.
- `hooks.py` — conversation hooks: follow-up prompts fired **after** a run
  finishes (Settings → Hooks). Cron's twin, with two differences — the trigger
  is an event (`conversation-done`) plus a delay instead of a schedule, and a
  firing **resumes the conversation that just finished** rather than spawning a
  new one. Store is `/data/hooks.json`, holding the hooks *and* a persistent
  `pending` queue so a queued firing survives a restart; seeded with the
  enabled `post-task` hook (delay 10s, prompt written to
  `.claude/agents/hooks/post-task.md` on first run). REST under `/api/hooks`
  (+ `/run`, `/prompt`) plus the read-only `/api/claude-hooks`; mutations and
  fires publish a `hooks` SSE event. See "A hook fires once per conversation"
  below.
- `models.py` — the models a session can run on (`GET /api/models`). Fetched from
  the Anthropic API (`/v1/models`, same key the token counter uses) and cached
  in-process for 6h, then reduced to **the newest of each family** (`opus`,
  `sonnet`, `haiku`, `fable` — older snapshots are dropped, so the tag row stays
  four chips) with the CLI alias attached. A newly released model appears in the
  composer with no code change. Falls back to a small static list when the key is
  missing or the call fails.
- `notify_audio.py` — the sound a notification made, replayable in the browser
  (`POST /api/notifications/audio` → a WAV, behind the ▶ button on an expanded
  notification card). There is **no synthesizer here**: it POSTs to the homelab
  phone service's `/render` (`services/phone/bridge/`), which composes exactly
  what it would have dialled — the per-type jazz jingle, then the spoken line in
  the cloned French voice — so the card and the handset can never drift apart.
  Results are memoized per `(spoken, type)`. `--spoken` only became mandatory
  recently, so a push without one falls back to reading its own title + message,
  minus the screen-only parts (notify.sh's cost footer, URLs, emoji).
- `ui_state.py` — server-side store (`/data/ui-state.json`) for the PWA's small
  client state (Settings config + notification feed seen/new bookkeeping), moved
  off browser localStorage so it follows the user across devices. Dumb key→string
  map: the key is a Zustand `persist` store name, the value its opaque serialized
  blob. Served by `GET/PUT/DELETE /api/ui-state/{key}`; the frontend points its
  `persist` stores at it via `frontend/src/lib/serverStorage.ts`.
- `projects.py` / `svc.py` — projects gallery / services catalog (static parsing).
- `skills.py` — the skills catalog behind `/api/skills` (the `/skills` page): a
  disk scan of every `.claude/skills/<name>/SKILL.md` (repo + each project's)
  joined with the transcript-mined usage. `/api/skills` returns the **union** of
  disk skills and invoked-skill names, so a built-in/plugin skill with no
  `SKILL.md` here still shows up (`sourceKind: "builtin"`, no editor link).
  Per skill: calls, distinct conversations, last-used, context tokens/cost
  (loading its dir once), `estSpend` = context × calls, and the coarser
  `sessionCost` (whole spend of every conversation that invoked it). Usage is
  keyed by bare skill name — the identity `Skill(name)` itself uses — so a
  project-local skill sharing a repo skill's name shares its counts.
- `activity.py` — the activity dashboard (`/api/activity`, the `/dashboards/activity`
  page): **when** the agents ran, as opposed to what they cost. Three rollups in
  one pass over the stored summaries — per local day (the contribution graph),
  per 5-minute slot per model (the day histogram, as positional
  `[slot, modelIndex, agents]` triples), and per model (runs / working time /
  tokens / cost). The per-turn half is already done by the parser: each summary
  carries `activeBins` (model id → the 5-minute bins that run produced a turn
  in), so a rollup never re-reads a transcript. **Bins are epoch-absolute UTC**;
  every entry point takes a `tzOffset` (minutes east of UTC, straight from the
  browser) and buckets local days against it — the one representation that
  survives a DST change. An *agent run* is one transcript, conversation or
  subagent alike; the heatmap's intensity is deliberately conversations only,
  while the histogram and per-model stats count every run.
- `goal.py` — a dir's `GOAL.md` (the north star the `goal-keeper` cron agent pushes
  forward), shared by both catalogs: `goal_summary()` returns the `{done, total}`
  checklist rollup that tags a card, `goal_detail()` adds the markdown for the
  detail page. Counting skips fenced code blocks, so a `- [ ]` inside a snippet
  isn't mistaken for a real task. No GOAL.md ⇒ `goal: null` (a valid state).
- `memories.py` — surfaces the assistant's per-repo memory files (frontmatter).
- `templates.py` — browse `~/projects/templates/` scaffolds.
- `schemas.py` — Pydantic response models mirroring `frontend/src/types.ts`. They
  drive the OpenAPI schema (and thus the generated frontend types) but are
  **documentation-only**: attached to routes via `responses={200: {"model": …}}`
  (helper `_r()` in `main.py`), never `response_model`, so FastAPI serves them in
  `/openapi.json` without validating or filtering the handlers' actual dicts.

### API types are generated (orval)

The frontend's API DTOs are **generated from the backend's OpenAPI spec**, not
hand-maintained. Flow: `backend/schemas.py` → `frontend/openapi.json` (a dumped
spec, committed) → orval → `frontend/src/generated/` (committed) → re-exported by
`frontend/src/types.ts` under their original names. `types.ts` keeps only the
non-API helpers (`TreeNode`, `FileKind`, `ServiceAuth`).

- **After changing a backend response shape**, update the matching model in
  `schemas.py`, re-dump `frontend/openapi.json` from the app's `app.openapi()`,
  then `cd frontend && npm run gen:api` to regenerate, and `npx tsc --noEmit` to
  surface fallout. Generated optionals are `T | null` (Pydantic `Optional`), so
  widen consumers to accept `null` — the app runs unchanged; only `tsc` cares.
- The Docker build doesn't regenerate (no backend at build time) — it relies on
  the committed `src/generated/` tree, so **commit regenerated output**.
- There's no venv on the host, so re-dump the spec with the service's own image:

  ```bash
  docker run --rm --entrypoint python -v "$PWD:/svc" -w /svc/backend \
    -e DB_PATH=/tmp/x.db -e TRANSCRIPTS_DIR=/tmp -e WORKSPACE=/tmp -e REPO_DIR=/tmp \
    homelab-ai-agent -c \
    "import json, os, main; json.dump(main.app.openapi(), open('/svc/frontend/openapi.json','w'), indent=2); os._exit(0)"
  ```

  Both odd-looking bits are load-bearing: **`--entrypoint python`** because the
  image's entrypoint (`docker-entrypoint.sh`) otherwise ignores the command and
  boots the server, and **`os._exit(0)`** because importing `main` starts the
  indexer/watcher threads, so a normal exit hangs waiting on them.

  It runs as root, so delete the `backend/__pycache__` it leaves behind (a
  root-owned dir will otherwise block `git worktree remove`).

### The mock backend (`frontend/src/mock/`) — demo + test without the API

The frontend ships an **in-browser mock backend** that sits behind those same
generated DTOs, so the PWA can be demoed and automatically tested with no
FastAPI, no sidecar and no transcripts on disk. Turn it on with `?mock=1` on any
URL (sticky until `?mock=0` — works against the deployed app too) or build/serve
with `VITE_MOCK=1` (`npm run dev:mock`); a `MOCK API` badge marks the mode.

It swaps `window.fetch` (every `/api/*` route) and `window.EventSource`
(`/api/events`, same `meta`/`transcript` events as `backend/events.py`) for
versions served from an in-memory, localStorage-persisted database. Writes really
mutate it, and **spawn/resume/interrupt actually run**: a scripted turn streams
into the conversation item by item over SSE, then stamps the notification + `DONE`
that mark it finished.

The seed is **combinatorial** — the cross-product of every field the UI branches
on (conversation harness × state × lifecycle × archived, every `ThreadItemKind`
and tool card, every `FileEntryKind`, every service auth, every plan status,
every `FileDiffStatus`/`DiffLineType`) — so every visual state is reachable from
a cold load, and it is deterministic (fixed clock, no RNG). Test hooks live on
`window.__mock` (`db()`, `reset()`, `speed()`, `flush()`).

Two rules keep it useful: install it **before the app tree is imported**
(`main.tsx` → dynamic `./mock`, then dynamic `./root`; the server-backed Zustand
stores fetch at import time), and type the seed rows with the **generated**
models — a backend schema change then breaks the seed at `tsc` time instead of
letting the mock drift. Details in `frontend/src/mock/README.md`.

### Every new piece of conversation data gets a visibility toggle

The conversation page has an eye-icon **visibility popover** (`components/
VisibilityPopover.tsx`, switches + store in `lib/visibility.ts`) — a scrollable,
grouped list of switches for everything the page renders: per-item metadata
(price, tokens, duration, timestamp, model, tool id, result size, diff stat),
content (thinking blocks, tool cards, run-command cards, tool output, rich
widgets, error states) and gizmos (task panel, fast-forward, animations). State
persists server-side via `ui-state` under the key `ai-agent-visibility`, so a
reading setup follows the user across devices.

**When you add anything to a message, a tool card or the conversation chrome,
add its switch too** — a new metadata field, a new inline widget, a new floating
gizmo, a new animation. It is not optional: the thread is dense, and every
addition has to be something the user can turn back off.

To add one: append a `VisItem` to the right group in `VIS_GROUPS`
(`lib/visibility.ts`) — the `VisKey` union, the popover list and the persisted
state all derive from that array — then gate the render on `useVis()` (or
`useVisible(key)` for a single switch). The store records only what's *hidden*,
so a newly added switch starts ON for everyone with no migration.

### Fixed lists are rearrangeable: `HoldEditable` + `useListOrder`

Small lists of links, chips or cards are user-arrangeable by **holding** one for
1.4s: it lifts out of the flow and follows the pointer, the rest of the group
starts jumping (iOS-springboard style), dragging into a neighbour's slot hands
it over, and releasing drops it there and ends edit mode. All of it lives in
`technical/ui/HoldEditable.tsx` — pointer-events only (one code path for mouse
and touch) and a body-level portal for the lifted item so nothing clips it. It's
layout-agnostic: the group's flex/grid classes come from the caller, so the same
component drives a horizontal navbar and a vertical panel (one row or column —
not a wrapping grid).

**The DOM order never changes during a drag.** Slots are measured once at pickup
and the rearrangement is expressed purely as transforms; only the drop commits a
real reorder, once the pointer is gone. This is load-bearing, not tidiness:
reordering live means React moves the pressed node — and re-renders whichever
item now sits in a position-dependent slot, like the navbar's raised centre tab —
and **a browser cancels the touch whose target left the document**, so the drag
died on the first hand-over. For the same reason the held item's (invisible)
in-slot copy keeps rendering with the props it had before the pickup: its node
must not be replaced. Freezing the DOM also fixes the geometry, so hit-testing
needs no re-measuring and no animation lock, and a fast drag can't outrun the
shuffle.

```tsx
const [tabs, setTabs] = useListOrder("nav", TABS, tabKey);   // lib/listOrder.ts
<HoldEditable items={tabs} getKey={tabKey} onReorder={setTabs} className="flex …">
  {(tab, { index, held, editing }) => <Tab …/>}
</HoldEditable>
```

For a list that *is* the data (prompt shortcuts), pass the array and write it
back. For a list hard-coded in the app, `useListOrder(name, items, getKey)`
persists only the order, as ids, under `settings.listOrders[name]`, and
reconciles it against the code list on read — so adding, renaming or removing an
entry in a later release can never strand or duplicate an item. Already wired:
the bottom navbar (`nav`), the dashboards / settings / files drawer panels, and
the prompt-shortcut cards.

A hold ends in a click, so `HoldEditable` swallows the click that closes a drag —
otherwise rearranging the navbar would also navigate. Presses that start inside
an `input`/`textarea`/`select`/`[contenteditable]` (or anything marked
`data-hold-editable-ignore`) never pick up.

Travel during the hold cancels it **only for a finger** (>12px), where it means
"I'm scrolling, not holding". A mouse is deliberately exempt: it can't scroll
with the button down, and a hand resting on one drifts far more than that over
1.4s — cancelling on drift made hold-to-drag impossible on desktop. The pickup
then happens wherever the cursor ended up, not where it went down.

## Performance: what the hot paths cost, and the rules that keep them cheap

This service polls the filesystem continuously and serves lists built from ~900
transcripts, so the naive version of almost anything here is quadratic. Four
rules, each of which was once a real regression:

1. **Never walk the workspace (or the transcript dirs) per request or per tick.**
   The file catalog comes from the `files` table; the discovery walk is cached in
   the indexer (`_exposed_files`, refreshed every `WALK_INTERVAL_SECS`, forced on
   save). The watcher tells the indexer *which* transcripts changed
   (`scan_transcripts(only=…)`), so a live turn re-parses one file instead of
   re-walking ~2 600. Walk with `fsscan.walk_files`, never `rglob`.
2. **Never re-parse a whole transcript to see what was appended.** A live session
   writes every couple of seconds; the indexer keeps a `ParserState` per growing
   transcript and feeds it only the new lines (byte offset in `Indexer._live`).
   Re-parsing from byte 0 each tick is O(n²) over a session.
3. **Never shell out to git per item.** Both catalogs share one HEAD-keyed history
   walk (`githist`).
4. **Never send content you don't need.** `/api/bundle` is metadata-only (content
   loads per-file via `/api/file`); `/api/conversations` is paginated + lite
   (`usage`/`byToolSub` only under `full=1`, which only the analytics dashboard
   asks for); an SSE `transcript` ping patches **one row** via
   `/api/conversation-summary` instead of refetching the list.

Measured on the real archive (898 transcripts / 458 MB): `/api/bundle` 12.5 s →
26 ms, `/api/services` 11.2 s → <1 s cold / 0.5 s warm, `/api/projects` 3.4 s →
0.35 s, the conversation list 1.3 MB → 89 KB, and idle CPU **~58% → ~7%** while a
session streams. If you add a feature here, check it against the four rules — and
re-measure, don't assume.

## Security: the trusted-caller gate

The backend has no auth of its own — Authelia guards it at Traefik. But the
container also sits on the shared `main` docker network, where a `PUT /api/file`
into `.claude/` (hooks!) or a `POST /api/spawn` is **host-level code execution**.
So `main.py` has a middleware that answers `/api/*` only for: Traefik (resolved
from `TRAEFIK_HOST`), the docker gateway (how host-originated connections to the
published `127.0.0.1:8096` port appear), localhost, or a caller presenting
`INTERNAL_API_TOKEN`. Everything else gets 403; `/api/health` stays open for the
deploy probe.

**Read-only API keys** (`READONLY_API_KEY`, comma-separated) are the one
credential that is *checked before* the trusted-IP path: a caller presenting a
matching `X-API-Key` is allowed **safe methods only** (GET/HEAD/OPTIONS) and any
mutating method is 403 — even from an otherwise-trusted source IP. That downgrade
is the point: the desk **phone** reaches the backend over the trusted host
loopback but must only ever *read* (it fetches `GET /api/notifications/unread` to
read notifications aloud), never spawn a session or PUT into `.claude/`. The
unread endpoint is a pure read — it never advances the read ledger
(`notif_read.py` / `/data/notif-read.json`); only the PWA opening/dismissing a
card (or `POST /api/notifications/seen`) marks notifications read. Assets served from repo/user bytes (`/api/data-asset`,
`/api/upload-file`, …) go out `nosniff` and, unless they're an allow-listed
raster image or video, `Content-Disposition: attachment` — an uploaded
`.html`/`.svg` must never execute on the app's origin.

## The runner: same sidecar, two places it can run

`sidecar/sidecar.py` is the only thing that launches `claude -p`. It runs in one
of two places, and the backend doesn't know the difference — it just POSTs to
`SIDECAR_URL`:

- **On the homelab host** (the default, below): a run then has the host user's
  own auth, hooks, skills and CLAUDE.md — i.e. it is exactly a session started
  from a terminal. This is why the homelab keeps it.
- **Inside the container** (`RUNNER_IN_CONTAINER=1`, the standalone default):
  the image bakes in the Claude Code CLI (a self-contained native binary — see
  the Dockerfile) and `docker-entrypoint.sh` starts the *same* sidecar module on
  `127.0.0.1:8790`, mints a `SIDECAR_TOKEN` if none was given, and overrides
  `SIDECAR_URL` to point at it. The CLI's `$HOME` is `RUNNER_HOME`
  (`/data/home`, on the data volume), so its config, credentials and the
  transcripts it writes survive a restart; `RUNNER_TRANSCRIPTS_DIR` (defaulted
  from it) is added to the backend's live `SOURCE_DIRS`, which is what makes an
  in-container session stream into the viewer like any other. Auth is
  `ANTHROPIC_API_KEY`, or an already-logged-in Claude home mounted at
  `RUNNER_HOME` (`-v ~/.claude:/data/home/.claude`). Remote control is off by
  default there (it needs an interactive login).

That's GOAL.md's "single Docker container": a standalone image spawns sessions
with no host process. Keep it that way — a new runner feature belongs in
`sidecar.py`, not in a host script.

### The host sidecar (what the homelab runs)

`sidecar/sidecar.py` runs **natively on the host as the repo owner** and the
backend proxies to it over `host.docker.internal:8790` (Bearer `SIDECAR_TOKEN`):

- `POST /api/spawn` → sidecar `/spawn`: `claude -p <prompt> --session-id <uuid>
  --model <model> --remote-control` (detached). Transcript lands in the watched
  projects dir and the new conversation shows up in the viewer within ~2s.
- `POST /api/resume` / `POST /api/interrupt` → continue / SIGINT an existing run.

`model` is the `claude --model` argument the composer's **model tag** carries — a
family alias (`opus`) for a family's newest model, or a pinned id
(`claude-opus-4-5-20251101`) for an older one. Unset on spawn ⇒ `SIDECAR_MODEL`
(default `opus`); unset on resume ⇒ no `--model` flag, so the session keeps its
own. The sidecar only shape-checks the value (it goes on a command line) — the
pickable set is whatever `/api/models` returned.

### Model tags

Models are a first-class tag on both sides of the app (`frontend/src/lib/models.tsx`):
picking one in a composer (a mutually-exclusive chip per model — RichInput's
`exclusive` key, added in `@gabvdl/ui` ≥0.11) runs that session on it and the
toolbar echoes the choice; the pick is sticky (`spawnModel` in Settings, stored
server-side). Reading back, a conversation is tagged with every model that
actually produced a turn — `conversations.py` collects them (`models`, busiest
first; a session can switch mid-thread), so past conversations are tagged too
(bump `PARSER_VERSION` to force the re-parse that back-fills a new summary field).

### Effort (claude) vs. thinking (pi)

How hard a run works a turn is **one knob shown two ways**, per harness — never
both at once:

- **claude** gets an `EffortSelect` (`--effort low|medium|high|xhigh|max`), the
  CLI's own scale. Sticky as `spawnEffort`; `""` means "send no flag" and let the
  CLI default. The sidecar validates against an allow-list (`EFFORT_LEVELS`),
  unlike `--model`, because the scale is a closed set.
- **pi** keeps the binary `ThinkingToggle` — it has no `--effort`. `_effort_args`
  emits nothing for it, so a stale pick can't leak onto a pi command line.

The composers send only the active harness's control (the other goes out unset).
Reading back, `conversations.py` mines the `effort` field the CLI stamps on each
**assistant record** (it rides on the record, not the message) into `efforts`,
busiest first — same shape as `models`, rendered by `EffortTags` on conversation
rows and in the detail header. Empty for pi runs and for transcripts predating
the flag, which is a valid state, not a blank chip.

### A cron job's run config lives in its prompt file's frontmatter

A scheduled job has no composer to pick a harness/model/effort in, so it says so
itself: the leading YAML block of `.claude/agents/<job>.md` carries `harness`
(`claude`|`pi`), `model` (a family alias or a full id), `effort`
(`low|medium|high|xhigh|max`, or `none` for no flag) and `thinking`
(`true`|`false`). `cron.run_config()` reads them, `_fire_cron_job` passes them
to the same `_send_message` the composer uses, and the resolved config comes
back on the job as `runConfig` so Settings → Cron shows it as chips. Every
homelab job currently declares `claude` + `sonnet` + no effort.

Two deliberate properties: the keys are a **subset of Claude Code's own agent
frontmatter** (`model` means the same thing there), so a job's prompt file stays
a usable `.claude/agents` subagent; and an invalid value is **dropped, not
raised** — a firing on the defaults beats a job that stops firing, and the
missing chip in the UI is the feedback. The store's per-job `model` field is now
only the fallback for a file whose frontmatter sets none.

### A hook fires once per conversation, and that is the whole design

A conversation hook resumes the session that just finished. The follow-up run
ends with `DONE` and exits too — which is exactly the signal that queued it — so
the naive version loops forever. Two things stop that, and neither is optional:

- **The ledger.** Every firing stamps `meta.hooks[hookId] = iso` on the session
  *before* the resume is sent. `MetaStore.update` **merges** that mapping instead
  of replacing it (a replace would let every earlier hook fire again), and both
  the enqueue path and the due-time preconditions refuse a stamped pair. One
  firing per (hook, session), ever. `POST /api/hooks/{id}/run` is the deliberate
  exception — a manual re-fire.
- **The claim.** The pending queue lives in the same JSON as the hooks, and the
  scheduler *removes* an entry (re-reading the file first) before firing it, so
  the deploy-cutover window where two backends share `/data` can't double-resume.

Queuing happens in two places, because a conversation can finish two ways: the
exit watcher (`_mark_run_exited`) and `POST /api/notify` (terminal-launched runs
the sidecar never saw). Both call `_enqueue_hooks(sid)`, which is best-effort —
a hook must never break the finish path.

The preconditions are checked at **due time**, not at enqueue time, because the
conversation may have been resumed, interrupted or archived in the meantime.
They return one of three verdicts: *go*, *postpone* (still live, transcript not
synced — re-queued a minute later, up to `hooks.MAX_RETRIES`) or *skip* (settled:
already fired, cron-spawned, a subagent, interrupted/paused, no `DONE` marker,
or over the hook's context cap). A skip is a normal outcome, not a failure, and
the page's status chip colours it amber rather than red.

**The context cap is a cost control, not a nicety — a resume doesn't hit the
finished run's prompt cache.** `--resume` is a fresh `claude -p` process, and
Claude Code rebuilds that process's environment preamble (cwd, git status,
date, …) from scratch; any byte of drift there (near-guaranteed on a checkout
other agents are also committing to) invalidates every cache breakpoint after
it. Sampled real transcripts confirm it: `cache_read_input_tokens` collapses
to the small cross-session baseline and `cache_creation_input_tokens`
re-creates the *entire* prior conversation, every firing, regardless of the
delay. That cost tracks the original conversation's size, so `maxContextK: 0`
means an unbounded tidy-up bill. See `hooks.py`'s module docstring for the
detail; the seeded `post-task` hook now defaults to `maxContextK: 60`.

A firing passes `model`/`effort`/`thinking` as `None` unless the prompt file's
frontmatter sets them, which `_send_message` reads as "keep whatever the session
already ran with" — a follow-up must not silently switch models mid-conversation.

Native Claude Code hooks (`.claude/settings.json`) are **read-only** here
(`GET /api/claude-hooks`, listed at the bottom of the page): writing that file
from the container would be host-level code execution — see the trusted-caller
gate above.

Install/manage the sidecar with `sidecar/install.sh` (systemd **user** unit
`claude-sidecar`); config comes from the repo `.env` (`SIDECAR_TOKEN`,
`SIDECAR_PORT`). See `sidecar/README.md`.

## Schema drift: the transcript format is not ours

`backend/conversations.py` parses Claude Code's **internal, undocumented** JSONL
transcript format, so a CLI update can silently change what it emits. The
early-warning tool is `scripts/schema-drift.py`:

```bash
scripts/schema-drift.py            # newest transcripts vs committed baseline
scripts/schema-drift.py --update   # accept reviewed drift as the new baseline
```

It does two things: diffs a **structural footprint** (key paths + content-block
types, never values — `scripts/schema-baseline.json` is safe in git) of recent
`~/.claude/projects` transcripts against the baseline, and runs a **parser
canary** (the real `parse_conversation` cross-checked against raw record
counts: assistant turns with usage, token totals, models, timestamps). Exit 1
on drift or a failed invariant. Run it after a Claude Code update or whenever
conversations render oddly; on drift, decide whether the parser needs to handle
the new shape, then `--update`. Every transcript record carries the CC
`version` stamp — a new version alone is INFO, not failure (releases are ~daily
and almost never drift).

## Deploying

**Never** `compose up --force-recreate` this service — use the blue-green script,
which validates a fresh build on a standby container before touching the live one:

```bash
~/homelab/services/ai-agent/deploy.sh   # the homelab shim's copy (runs from anywhere)
```

It builds a new image, brings up a second `ai-agent-deploy` standby on
`homelab_main`, health-gates it on `/api/health`, cuts Traefik over to it
(file-provider hot-reload), recreates the canonical `ai-agent` on the new image,
then restores routing and removes the standby — zero downtime, and if the standby
never goes healthy the live container is left untouched. An `flock` on
`/tmp/ai-agent-deploy.lock` serializes deploys; a second run aborts immediately.
`deploy.override.yml` is only for the standby slot (renames the container, drops
the host port, joins the external `homelab_main` network).

## Standalone mode: a missing catalog is a valid state

The homelab `docker-compose.yml` mounts ~8 host paths (transcripts, memories,
screenshots, services/, plans, …), but the image must also run with **none** of
them — that's the GOAL's "generic first-run setup" / one-liner `docker run`.
`docker-compose.standalone.yml` is that shape: a workspace + a data volume, and
nothing else required.

So **every catalog degrades to an empty list when its dir is absent** — never a
500. (`/api/templates` used to raise; it doesn't now.) A *named* item that isn't
there is still a 404 — it's only the missing-mount case that's benign. When you
add a catalog backed by a mount, follow that rule.

The guard is a smoke test — run it after touching mounts, env wiring or a
catalog route:

```bash
scripts/standalone-smoke.sh            # boot the current image on a bare workspace
scripts/standalone-smoke.sh --build    # build first, then smoke
```

It boots the image with only a throwaway workspace + data dir (and
`RUNNER_IN_CONTAINER=1`), asserts every catalog endpoint and the PWA shell answer
200, that the bundled `claude` CLI is on PATH and its runner answers `/health`,
and greps the log for tracebacks (a background indexer/watcher crash won't fail a
request). Exit 1 = regressed.

Standalone **spawns** sessions now (the in-image runner, above). The one thing it
still can't do is **build** off-homelab: `frontend/.npmrc` pulls `@gabvdl/ui` from
the host-bound verdaccio, so a standalone build needs that registry — a prebuilt
image runs anywhere. Tracked in [GOAL.md](GOAL.md).

## Local development

There is no lint/test pipeline. Frontend and backend are built into the image;
to iterate:

- **Frontend only**: `cd frontend && npm install && npm run dev` (Vite on `:5180`).
  It needs the backend API — point it at a running container or backend.
- **Full stack**: rebuild + redeploy via the homelab shim's `deploy.sh`, or hit the running
  container directly on the host at `http://127.0.0.1:8096` (plain HTTP, bypasses
  Authelia — handy since new `*.lab` domains can't get an LE cert here).
- **Health**: `curl -s http://127.0.0.1:8096/api/health` → `{"ok":true}`.

After changing the homelab shim's `traefik.yml`, run `make link-traefik-configs`
from the homelab repo root (the file provider watches `config/dynamic/`, not the
shim's file).

## Gotchas

- **Read-only mounts**: most of the repo context is mounted `:ro` (services/, .git,
  transcripts, memories, screenshots, and the root `data/` tree). Only `.claude/`,
  `~/projects` (host-side; `/workspace/projects` in-container), `CLAUDE.md` and the service's own `data/` (SQLite store) are
  writable. The container runs as `1000:1000` so edits keep repo ownership.
- **Root `data/` tree**: mounted `${PWD}/data:/workspace/data:ro` and surfaced in
  the file tab for visibility only. The indexer exposes just text files (see
  `DATA_TEXT_EXTS` in `main.py` — md/json/py/log/…; binaries like the PNG
  thumbnails are skipped) and marks them non-editable. `data/tmp/` is in
  `IGNORE_DIRS`, so scratch files and secrets (e.g. `*-creds.env`) never appear.
- **Token counts cost API calls**: they hit the `count_tokens` endpoint
  (`ANTHROPIC_API_KEY` from `.env`) and are recomputed only when a file's content
  hash changes, after a debounce — don't add code that recounts on every request.
- **Conversation "finished" state**: three signals, strongest first. (1) A
  `notify-done` notification **plus** a last message ending with the literal
  `DONE` marker (see the root CLAUDE.md "When work is done"). (2) The **exit
  watcher**: a background thread in `main.py` (`_watch_sidecar_runs`) polls the
  sidecar's `/sessions` every ~2s and stamps `finished` (+`exitedAt`) the moment
  a sidecar-launched run's process dies — so runs whose prompt never said DONE
  don't sit on "running". The stamp is distrusted if the transcript later grows
  past `exitedAt` (a terminal/remote-control continuation, `_outlived_exit`),
  and in-flight resume/interrupt swaps are shielded via `_sidecar_busy`. (3)
  The recency fallback (`RUNNING_WINDOW_SECS`, 600s) for anything the watcher
  never saw live (terminal sessions, runs that died while the backend was down).
- **Conversation "paused" state**: a run whose last act was parking itself on a
  trigger — a trailing `ScheduleWakeup` (non-`stop`) or a pending `Monitor` — is
  shown `paused` instead of `running` (`waiting_trailing` in `conversations.py`,
  resolved in `main.py:_resolve_state`). The flag clears on any later tool call,
  real user turn, or interrupt — not on the same turn's closing text.
- **Two containers briefly share `data/`** during a deploy cutover (SQLite); this is
  a short window and intentional — don't "fix" it by locking the DB.
- **A resume must never answer 200 without a live run behind it.** The composer
  echoes the typed turn optimistically and only drops the echo once the
  transcript grows, so any resume that reports success but starts nothing leaves
  the UI on "sending…" forever. Two guards keep that honest and must stay:
  `sidecar.py` watches a freshly launched `claude` for `LAUNCH_PROBE_S` and 502s
  with its log tail if it dies on the spot (a `--resume` whose transcript isn't
  found exits 1 in ~2.5s), and `/resume` only reports `reused` for a process that
  is *genuinely still running* — a finished-but-unreaped child is a zombie, and a
  zombie answers `os.kill(pid, 0)` (hence `_reap()` + the `Z` check in `_alive`).
  `reused` means the turn was **dropped**, so the backend returns 409, not 200.
  The backend always resumes with `force: true`: a genuinely-live run is stopped
  first (`_stop_run` — SIGINT the group, wait up to `STOP_WAIT_S`, escalate to
  SIGKILL) and the resume then launches, so sending a message into a running or
  paused conversation restarts it on the new turn instead of 409ing. `reused`
  can now only come from a sidecar predating the flag.