ai-agent/CLAUDE.md

# CLAUDE.md — `ai-agent`

Guidance for Claude Code when working in this repo. It was extracted from the
homelab monorepo (formerly `services/ai-agent/`, full history preserved); the
homelab keeps a deployment shim at `~/homelab/services/ai-agent/` (compose with
`build.context` pointing here, traefik.yml, `deploy.sh`, the live `data/`
volume), and the homelab root `CLAUDE.md` conventions (worktrees, Traefik copy
step, commit/notify) still apply when deploying there.

## North star

This service is headed somewhere bigger than a homelab dashboard: **the
open-source, intuitive, private and secure agent layer of any computer, to
build and host websites and interact with data** — Lovable, but open source
and self-hosted. Zipgo under the hood for hosting, shipped as a **single
Docker container**, with self-update handled **inside the container** (not by
host scripts). See [GOAL.md](GOAL.md) for the full vision and the gap list.
When making changes here, prefer container-internal solutions, generic config
over homelab-hardcoded paths, and zipgo over bespoke hosting.

## What this is

A viewer/editor + analytics dashboard for the homelab's **Claude context**, served
as an installable PWA at `https://ai-agent.lab.gabvdl.xyz` (behind Authelia). It:

- exposes every `CLAUDE.md` and the `.claude/` tree (skills, hooks, settings) as a
  flat, editable file list with **real** Claude token counts + dollar costs, plus
  the repo's root `data/` tree (plans, logs, notes, kanban) surfaced **read-only**;
- browses the archived Claude Code **conversation transcripts** (thread view with
  per-turn usage/cost, skill-usage analytics, spawn/resume/interrupt live runs);
- catalogs **~/projects** (mounted at `/workspace/projects`; gallery + detail) and the repo's **services/** (static
  parse of compose + Traefik configs — never Docker itself);
- surfaces the assistant's **memories** and the scaffolding **templates**;
- streams updates over SSE so the UI refreshes without a manual reload.

It is a **viewer**: it reads the repo context (mostly read-only mounts) and does
static parsing. It never drives Docker or mutates infrastructure. The only writes
are file edits into the mounted `.claude/` + `~/projects` trees and its own SQLite
store / metadata sidecar.

## Layout

- `backend/` — FastAPI app (`main:app`, port 8080). Serves `/api/*` and the built
  PWA from `STATIC_DIR` (SPA catch-all at the end of `main.py`).
- `frontend/` — Vite + React + TS + Tailwind PWA. Built in the Dockerfile's first
  stage into `/app/static`.
- `sidecar/` — a **host-side** FastAPI process (NOT in the container) that launches
  `claude -p` sessions. See below.
- `Dockerfile` — 2 stages: build the React PWA, then a `python:3.12-slim` image
  running `uvicorn`.
- `docker-compose.standalone.yml` — the generic, no-host-mounts shape (see
  "Standalone mode" below). The homelab's compose service (profile `dev`,
  container `ai-agent`, host-only port `127.0.0.1:8096:8080`, network `main`),
  its Traefik router (`ai-agent.lab.gabvdl.xyz`, `auth-chain`) and the
  blue-green `deploy.sh` + `deploy.override.yml` live in the homelab repo's
  `services/ai-agent/` shim.
- `PROJECT_CLAUDE.md` — the **generic, host-agnostic** project working conventions
  (worktree workflow, artefacts, plans, committing, the notify+`DONE` finish
  signal), split out of the homelab root `CLAUDE.md` so the ai-agent can ship it
  as the starting `CLAUDE.md` for a new project. Not this service's own guidance
  (that's this file) — it's a shippable template with `{{…}}` placeholders.
- `data/` — SQLite DB, transcript archive, metadata sidecar, `ui-state.json`
  (bind-mounted, gitignored).

### Backend modules

- `main.py` — FastAPI app, all routes, env/path wiring, startup (indexer + watcher).
- `db.py` — SQLite store (WAL): `files` (content hash + real token count + cost +
  word/char stats, computed lazily only when a file's hash changes), `skills`,
  `transcripts` bookkeeping. Decoded transcript summaries are cached in-process
  (`all_summaries()`/`get_summary()` return **read-only** dicts — copy before
  mutating) and `summaries_version` bumps on change so request-level memoizers
  (e.g. the project cost rollup) know when to recompute.
- `indexer.py` — background daemon: token/cost accounting (via the `count_tokens`
  endpoint, debounced) + skill-usage mining from transcripts. Pricing consts here.
- `conversations.py` — parse `*.jsonl` transcripts. `ParserState` is a **resumable**
  parser (`feed(record)` + `summary()`); `parse_conversation(path, full=…)` is the
  one-shot wrapper (cheap rollup vs. full thread). Real per-turn usage/cost from
  `usage` blocks, plus the per-item metadata the viewer shows on each row:
  `durationMs` (a tool call's run time — call → its result record; on a message,
  the assistant API call's latency), `model`, `toolUseId`, `resultChars`/`resultLines`.
- `fsscan.py` — `os.scandir`-based directory walking. The poll loops re-walk
  thousands of files every couple of seconds; `pathlib.rglob` allocates a `Path`
  per entry and was a large chunk of idle CPU. Use this, not `rglob`, in any loop.
- `githist.py` — cached git history. `BucketedHistory` walks a repo's log **once**
  (`git log --name-only -- <prefix>`) and buckets commits per directory, keyed on
  the repo HEAD; `RepoLogCache` TTL-caches standalone project repos. Before this,
  `/api/services` forked two `git log`s per service **per request** (11 s).
- `events.py` — `Hub` (SSE pub/sub) + `Watcher` (polls meta + live transcripts,
  re-indexes, publishes `meta`/`transcript` events).
- `meta.py` — per-conversation action metadata JSON sidecar (projects, state,
  committed/pushed/merged/deployed/notified timestamps), keyed by session id.
- `cron.py` — cron jobs: scheduled agent sessions (Settings → Cron). A job =
  a 5-field cron schedule (dependency-free matcher, container-local time —
  `TZ` in compose) + a prompt file under `.claude/agents/` whose content
  (frontmatter stripped) is spawned as a session via the normal sidecar path;
  each firing is recorded in the job's `history` (`{timestamp: sessionId}`)
  and the conversation is tagged (`meta.cron` → violet badge linking back to
  the job). **How a job runs is declared in that file's frontmatter** — see
  below. Store is `/data/cron-jobs.json` (seeded with the every-5h
  `goal-keeper` job on first run); the scheduler thread claims each fire
  minute through the store, so the deploy-cutover window where two backends
  share `/data` can't double-spawn. REST under `/api/cron` (+ `/run`,
  `/prompt`); mutations and fires publish a `cron` SSE event.
- `ctas.py` — conversation CTAs: the prompt buttons a finished conversation
  offers (Settings → CTAs). Cron's other twin — a prompt file plus a button,
  with **no scheduler at all**: the viewer renders one per CTA under the last
  message of a finished run, a click loads its prompt into the resume box and a
  long-press fires it at the session in the background. Store is
  `/data/ctas.json`; seeded with the enabled `complete` CTA (prompt written to
  `.claude/agents/cta/complete.md` on first run). REST under `/api/ctas`
  (+ `/run`, `/prompt`, `/reorder`) plus the read-only `/api/claude-hooks`;
  mutations and runs publish a `ctas` SSE event. See "CTAs replaced the
  conversation hooks" below.
- `gitdiff.py` / `worktreediff.py` / `editsdiff.py` — the **diff view** (`/diff/*`),
  in three layers. `gitdiff` does committed history: recent commits per repo
  token (`project:<slug>` | `service:<slug>` | `super:_`), one commit's hunk
  model, the net diff across a conversation's commits, and the discovery of
  *which* commits a conversation produced (its active time window, plus
  `main..<worktree branch>`; cached in the DB by a fingerprint of the repo
  HEADs). The other two answer "what isn't committed yet":
  - `worktreediff` — a **real** `git diff HEAD` (+ untracked files) read from
    the conversation's still-live worktree. A worktree's `.git` file points at a
    *host* path, so it maps the host repo root onto the mounted one and drives
    git with an explicit `--git-dir`/`--work-tree` pair. Needs the worktrees root
    mounted (`WORKTREES_DIR`, `/worktrees` in the homelab compose); without it
    every lookup misses and the replay below takes over.
  - `editsdiff` — the fallback for a torn-down worktree: the same view *replayed*
    from the transcript's `structuredPatch`/`originalFile` payloads, hence
    `approx: true`. Paths from the homelab checkout, a `~/projects/<slug>` repo
    and either flavour of worktree all normalise onto the repo-relative paths
    that repo token's git commands use.
- `scaffold.py` — the completion scaffold (`GET /api/conversations/{id}/scaffold`):
  a conversation reduced to markdown — metadata, its prompts, merged
  bash/tool/subagent summaries, and one squashed diff across every commit it
  produced (via `gitdiff`) — with `## Summary` and `## Analysis` left blank for
  the model. `PUT …/summary` publishes the filled version onto `meta.summary`.
  The CLI wrapper is `scripts/conv-scaffold.sh`.
- `models.py` — the models a session can run on (`GET /api/models`). Fetched from
  the Anthropic API (`/v1/models`, same key the token counter uses) and cached
  in-process for 6h, then reduced to **the newest of each family** (`opus`,
  `sonnet`, `haiku`, `fable` — older snapshots are dropped, so the tag row stays
  four chips) with the CLI alias attached. A newly released model appears in the
  composer with no code change. Falls back to a small static list when the key is
  missing or the call fails.
- `notify_audio.py` — the sound a notification made, replayable in the browser
  (`POST /api/notifications/audio` → a WAV, behind the ▶ button on an expanded
  notification card). There is **no synthesizer here**: it POSTs to the homelab
  phone service's `/render` (`services/phone/bridge/`), which composes exactly
  what it would have dialled — the per-type jazz jingle, then the spoken line in
  the cloned French voice — so the card and the handset can never drift apart.
  Results are memoized per `(spoken, type)`. `--spoken` only became mandatory
  recently, so a push without one falls back to reading its own title + message,
  minus the screen-only parts (notify.sh's cost footer, URLs, emoji).
- `ui_state.py` — server-side store (`/data/ui-state.json`) for the PWA's small
  client state (Settings config + notification feed seen/new bookkeeping), moved
  off browser localStorage so it follows the user across devices. Dumb key→string
  map: the key is a Zustand `persist` store name, the value its opaque serialized
  blob. Served by `GET/PUT/DELETE /api/ui-state/{key}`; the frontend points its
  `persist` stores at it via `frontend/src/lib/serverStorage.ts`.
- `projects.py` / `svc.py` — projects gallery / services catalog (static parsing).
- `skills.py` — the skills catalog behind `/api/skills` (the `/skills` page): a
  disk scan of every `.claude/skills/<name>/SKILL.md` (repo + each project's)
  joined with the transcript-mined usage. `/api/skills` returns the **union** of
  disk skills and invoked-skill names, so a built-in/plugin skill with no
  `SKILL.md` here still shows up (`sourceKind: "builtin"`, no editor link).
  Per skill: calls, distinct conversations, last-used, context tokens/cost
  (loading its dir once), `estSpend` = context × calls, and the coarser
  `sessionCost` (whole spend of every conversation that invoked it). Usage is
  keyed by bare skill name — the identity `Skill(name)` itself uses — so a
  project-local skill sharing a repo skill's name shares its counts.
- `agents.py` — the **agents** catalog behind `/api/agents` (the `/agents` page),
  the same page shape as skills for `.claude/agents/**/*.md`: a disk scan
  (frontmatter `name`/`description`/`tools` + the `harness`/`model`/`effort`
  run-config keys) unioned with the agent types the transcripts saw run, so a
  built-in type (`Explore`, `general-purpose`) shows up as `sourceKind:
  "builtin"`. `.claude/agents/cta/` is skipped — those are CTA prompts
  (`ctas.py`), not agent definitions.
  **The money here is measured, not estimated**: an agent run is a whole
  conversation, so `main._build_agent_runs` joins two origins into one run list
  — a **subagent** run (a sidechain transcript whose `agent-*.meta.json` names
  the `agentType`, credited to its parent conversation) and a **session** run
  (`cron.py` fires a definition as a top-level `claude -p`; the agent page's
  "Run now" does the same and stamps `agentRun` into the metadata sidecar).
  `POST /api/agents/{name}/run` is that CTA; scheduling reuses `POST /api/cron`
  with the definition as the prompt file.
- `activity.py` — the activity dashboard (`/api/activity`, the `/dashboards/activity`
  page): **when** the agents ran, as opposed to what they cost. Three rollups in
  one pass over the stored summaries — per local day (the contribution graph),
  per 5-minute slot per model (the day histogram, as positional
  `[slot, modelIndex, agents]` triples), and per model (runs / working time /
  tokens / cost). The per-turn half is already done by the parser: each summary
  carries `activeBins` (model id → the 5-minute bins that run produced a turn
  in), so a rollup never re-reads a transcript. **Bins are epoch-absolute UTC**;
  every entry point takes a `tzOffset` (minutes east of UTC, straight from the
  browser) and buckets local days against it — the one representation that
  survives a DST change. An *agent run* is one transcript, conversation or
  subagent alike; the heatmap's intensity is deliberately conversations only,
  while the histogram and per-model stats count every run.
- `goal.py` — a dir's `GOAL.md` (the north star the `goal-keeper` cron agent pushes
  forward), shared by both catalogs: `goal_summary()` returns the `{done, total}`
  checklist rollup that tags a card, `goal_detail()` adds the markdown for the
  detail page. Counting skips fenced code blocks, so a `- [ ]` inside a snippet
  isn't mistaken for a real task. No GOAL.md ⇒ `goal: null` (a valid state).
- `memories.py` — surfaces the assistant's per-repo memory files (frontmatter).
- `templates.py` — browse `~/projects/templates/` scaffolds.
- `schemas.py` — Pydantic response models mirroring `frontend/src/types.ts`. They
  drive the OpenAPI schema (and thus the generated frontend types) but are
  **documentation-only**: attached to routes via `responses={200: {"model": …}}`
  (helper `_r()` in `main.py`), never `response_model`, so FastAPI serves them in
  `/openapi.json` without validating or filtering the handlers' actual dicts.

### API types are generated (orval)

The frontend's API DTOs are **generated from the backend's OpenAPI spec**, not
hand-maintained. Flow: `backend/schemas.py` → `frontend/openapi.json` (a dumped
spec, committed) → orval → `frontend/src/generated/` (committed) → re-exported by
`frontend/src/types.ts` under their original names. `types.ts` keeps only the
non-API helpers (`TreeNode`, `FileKind`, `ServiceAuth`).

- **After changing a backend response shape**, update the matching model in
  `schemas.py`, re-dump `frontend/openapi.json` from the app's `app.openapi()`,
  then `cd frontend && npm run gen:api` to regenerate, and `npx tsc --noEmit` to
  surface fallout. Generated optionals are `T | null` (Pydantic `Optional`), so
  widen consumers to accept `null` — the app runs unchanged; only `tsc` cares.
- The Docker build doesn't regenerate (no backend at build time) — it relies on
  the committed `src/generated/` tree, so **commit regenerated output**.
- There's no venv on the host, so re-dump the spec with the service's own image:

  ```bash
  docker run --rm --entrypoint python -v "$PWD:/svc" -w /svc/backend \
    -e DB_PATH=/tmp/x.db -e TRANSCRIPTS_DIR=/tmp -e WORKSPACE=/tmp -e REPO_DIR=/tmp \
    homelab-ai-agent -c \
    "import json, os, main; json.dump(main.app.openapi(), open('/svc/frontend/openapi.json','w'), indent=2); os._exit(0)"
  ```

  Both odd-looking bits are load-bearing: **`--entrypoint python`** because the
  image's entrypoint (`docker-entrypoint.sh`) otherwise ignores the command and
  boots the server, and **`os._exit(0)`** because importing `main` starts the
  indexer/watcher threads, so a normal exit hangs waiting on them.

  It runs as root, so delete the `backend/__pycache__` it leaves behind (a
  root-owned dir will otherwise block `git worktree remove`).

### The mock backend (`frontend/src/mock/`) — demo + test without the API

The frontend ships an **in-browser mock backend** that sits behind those same
generated DTOs, so the PWA can be demoed and automatically tested with no
FastAPI, no sidecar and no transcripts on disk. Turn it on with `?mock=1` on any
URL (sticky until `?mock=0` — works against the deployed app too) or build/serve
with `VITE_MOCK=1` (`npm run dev:mock`); a `MOCK API` badge marks the mode.

It swaps `window.fetch` (every `/api/*` route) and `window.EventSource`
(`/api/events`, same `meta`/`transcript` events as `backend/events.py`) for
versions served from an in-memory, localStorage-persisted database. Writes really
mutate it, and **spawn/resume/interrupt actually run**: a scripted turn streams
into the conversation item by item over SSE, then stamps the notification + `DONE`
that mark it finished.

The seed is **combinatorial** — the cross-product of every field the UI branches
on (conversation harness × state × lifecycle × archived, every `ThreadItemKind`
and tool card, every `FileEntryKind`, every service auth, every plan status,
every `FileDiffStatus`/`DiffLineType`) — so every visual state is reachable from
a cold load, and it is deterministic (fixed clock, no RNG). Test hooks live on
`window.__mock` (`db()`, `reset()`, `speed()`, `flush()`).

Two rules keep it useful: install it **before the app tree is imported**
(`main.tsx` → dynamic `./mock`, then dynamic `./root`; the server-backed Zustand
stores fetch at import time), and type the seed rows with the **generated**
models — a backend schema change then breaks the seed at `tsc` time instead of
letting the mock drift. Details in `frontend/src/mock/README.md`.

### Every new piece of conversation data gets a visibility toggle

The conversation page has an eye-icon **visibility popover** (`components/
VisibilityPopover.tsx`, switches + store in `lib/visibility.ts`) — a scrollable,
grouped list of switches for everything the page renders: per-item metadata
(price, tokens, duration, timestamp, model, tool id, result size, diff stat),
content (thinking blocks, tool cards, run-command cards, tool output, rich
widgets, error states) and gizmos (task panel, fast-forward, animations). State
persists server-side via `ui-state` under the key `ai-agent-visibility`, so a
reading setup follows the user across devices.

**When you add anything to a message, a tool card or the conversation chrome,
add its switch too** — a new metadata field, a new inline widget, a new floating
gizmo, a new animation. It is not optional: the thread is dense, and every
addition has to be something the user can turn back off.

To add one: append a `VisItem` to the right group in `VIS_GROUPS`
(`lib/visibility.ts`) — the `VisKey` union, the popover list and the persisted
state all derive from that array — then gate the render on `useVis()` (or
`useVisible(key)` for a single switch). The store records only what's *hidden*,
so a newly added switch starts ON for everyone with no migration.

### "Waiting for feedback" — the home page's answer queue

The home page opens with the one list that is waiting on *the user*: every
**pending form** (`forms.py`, the `ask-form` skill) and every **pending ask**
(`notify.py`, the `notify-done` skill's tappable choice), above the Running rail
and the notification feed. `lib/waiting.ts` merges the two sources and joins
each question to its conversation by `sessionId`; the section
(`conversations/components/WaitingForFeedback.tsx`) renders it **amber with a
spinner** while that session is still running and **red** once it isn't, and
reports the conversation ids it showed so the Running rail below skips them.

Three things make that work, and each is easy to break:

- **Both skills must stamp `sessionId`.** It is the only link back to a
  conversation — a question with no session still renders, but as an orphan
  card. `ask.sh` resolves it the same way `notify.sh` and `ask-form.sh` do
  (`$CLAUDE_SESSION_ID`, else conv-meta's `resolve-session.sh`).
- **A question can be older than the loaded page.** The home list is the first
  ~50 conversations, so `GET /api/conversations?sessions=a,b,c` resolves exactly
  the ids a first pass missed (archived included, no paging). Never reach for
  `limit=0` here — the full lite list is ~1.5 MB for one title.
- **Both kinds are answerable in place.** A form flips `?form=<id>` and the
  page's own `<FormModal/>` opens it full-screen (no navigation); an ask POSTs
  to `/api/ask/{id}/answer` — the very endpoint the phone's tapped button
  uses, so the agent's long-poll wakes either way.

An ask the backend still calls `pending` past its `expiresAt` is filtered out
client-side: `notify.py` only flips a lapsed ask to `expired` when *that* ask is
read, so the list would otherwise keep showing dead questions.

### Fixed lists are rearrangeable: `HoldEditable` + `useListOrder`

Small lists of links, chips or cards are user-arrangeable by **holding** one for
1.4s: it lifts out of the flow and follows the pointer, the rest of the group
starts jumping (iOS-springboard style), dragging into a neighbour's slot hands
it over, and releasing drops it there and ends edit mode. All of it lives in
`technical/ui/HoldEditable.tsx` — pointer-events only (one code path for mouse
and touch) and a body-level portal for the lifted item so nothing clips it. It's
layout-agnostic: the group's flex/grid classes come from the caller, so the same
component drives a horizontal navbar and a vertical panel (one row or column —
not a wrapping grid).

**The DOM order never changes during a drag.** Slots are measured once at pickup
and the rearrangement is expressed purely as transforms; only the drop commits a
real reorder, once the pointer is gone. This is load-bearing, not tidiness:
reordering live means React moves the pressed node — and re-renders whichever
item now sits in a position-dependent slot, like the navbar's raised centre tab —
and **a browser cancels the touch whose target left the document**, so the drag
died on the first hand-over. For the same reason the held item's (invisible)
in-slot copy keeps rendering with the props it had before the pickup: its node
must not be replaced. Freezing the DOM also fixes the geometry, so hit-testing
needs no re-measuring and no animation lock, and a fast drag can't outrun the
shuffle.

```tsx
const [tabs, setTabs] = useListOrder("nav", TABS, tabKey);   // lib/listOrder.ts
<HoldEditable items={tabs} getKey={tabKey} onReorder={setTabs} className="flex …">
  {(tab, { index, held, editing }) => <Tab …/>}
</HoldEditable>
```

For a list that *is* the data (prompt shortcuts), pass the array and write it
back. For a list hard-coded in the app, `useListOrder(name, items, getKey)`
persists only the order, as ids, under `settings.listOrders[name]`, and
reconciles it against the code list on read — so adding, renaming or removing an
entry in a later release can never strand or duplicate an item. Already wired:
the bottom navbar (`nav`), the dashboards / settings / files drawer panels, and
the prompt-shortcut cards.

A hold ends in a click, so `HoldEditable` swallows the click that closes a drag —
otherwise rearranging the navbar would also navigate. Presses that start inside
an `input`/`textarea`/`select`/`[contenteditable]` (or anything marked
`data-hold-editable-ignore`) never pick up.

Travel during the hold cancels it **only for a finger** (>12px), where it means
"I'm scrolling, not holding". A mouse is deliberately exempt: it can't scroll
with the button down, and a hand resting on one drifts far more than that over
1.4s — cancelling on drift made hold-to-drag impossible on desktop. The pickup
then happens wherever the cursor ended up, not where it went down.

## Performance: what the hot paths cost, and the rules that keep them cheap

This service polls the filesystem continuously and serves lists built from ~900
transcripts, so the naive version of almost anything here is quadratic. Four
rules, each of which was once a real regression:

1. **Never walk the workspace (or the transcript dirs) per request or per tick.**
   The file catalog comes from the `files` table; the discovery walk is cached in
   the indexer (`_exposed_files`, refreshed every `WALK_INTERVAL_SECS`, forced on
   save). The watcher tells the indexer *which* transcripts changed
   (`scan_transcripts(only=…)`), so a live turn re-parses one file instead of
   re-walking ~2 600. Walk with `fsscan.walk_files`, never `rglob`.
2. **Never re-parse a whole transcript to see what was appended.** A live session
   writes every couple of seconds; the indexer keeps a `ParserState` per growing
   transcript and feeds it only the new lines (byte offset in `Indexer._live`).
   Re-parsing from byte 0 each tick is O(n²) over a session.
3. **Never shell out to git per item.** Both catalogs share one HEAD-keyed history
   walk (`githist`).
4. **Never send content you don't need.** `/api/bundle` is metadata-only (content
   loads per-file via `/api/file`); `/api/conversations` is paginated + lite
   (`usage`/`byToolSub` only under `full=1`, which only the analytics dashboard
   asks for); an SSE `transcript` ping patches **one row** via
   `/api/conversation-summary` instead of refetching the list.

Measured on the real archive (898 transcripts / 458 MB): `/api/bundle` 12.5 s →
26 ms, `/api/services` 11.2 s → <1 s cold / 0.5 s warm, `/api/projects` 3.4 s →
0.35 s, the conversation list 1.3 MB → 89 KB, and idle CPU **~58% → ~7%** while a
session streams. If you add a feature here, check it against the four rules — and
re-measure, don't assume.

## Security: the trusted-caller gate

The backend has no auth of its own — Authelia guards it at Traefik. But the
container also sits on the shared `main` docker network, where a `PUT /api/file`
into `.claude/` (hooks!) or a `POST /api/spawn` is **host-level code execution**.
So `main.py` has a middleware that answers `/api/*` only for: Traefik (resolved
from `TRAEFIK_HOST`), the docker gateway (how host-originated connections to the
published `127.0.0.1:8096` port appear), localhost, or a caller presenting
`INTERNAL_API_TOKEN`. Everything else gets 403; `/api/health` stays open for the
deploy probe.

**Read-only API keys** (`READONLY_API_KEY`, comma-separated) are the one
credential that is *checked before* the trusted-IP path: a caller presenting a
matching `X-API-Key` is allowed **safe methods only** (GET/HEAD/OPTIONS) and any
mutating method is 403 — even from an otherwise-trusted source IP. That downgrade
is the point: the desk **phone** reaches the backend over the trusted host
loopback but must only ever *read* (it fetches `GET /api/notifications/unread` to
read notifications aloud), never spawn a session or PUT into `.claude/`. The
unread endpoint is a pure read — it never advances the read ledger
(`notif_read.py` / `/data/notif-read.json`); only the PWA opening/dismissing a
card (or `POST /api/notifications/seen`) marks notifications read. Assets served from repo/user bytes (`/api/data-asset`,
`/api/upload-file`, …) go out `nosniff` and, unless they're an allow-listed
raster image or video, `Content-Disposition: attachment` — an uploaded
`.html`/`.svg` must never execute on the app's origin.

## The runner: same sidecar, two places it can run

`sidecar/sidecar.py` is the only thing that launches `claude -p`. It runs in one
of two places, and the backend doesn't know the difference — it just POSTs to
`SIDECAR_URL`:

- **On the homelab host** (the default, below): a run then has the host user's
  own auth, hooks, skills and CLAUDE.md — i.e. it is exactly a session started
  from a terminal. This is why the homelab keeps it.
- **Inside the container** (`RUNNER_IN_CONTAINER=1`, the standalone default):
  the image bakes in the Claude Code CLI (a self-contained native binary — see
  the Dockerfile) and `docker-entrypoint.sh` starts the *same* sidecar module on
  `127.0.0.1:8790`, mints a `SIDECAR_TOKEN` if none was given, and overrides
  `SIDECAR_URL` to point at it. The CLI's `$HOME` is `RUNNER_HOME`
  (`/data/home`, on the data volume), so its config, credentials and the
  transcripts it writes survive a restart; `RUNNER_TRANSCRIPTS_DIR` (defaulted
  from it) is added to the backend's live `SOURCE_DIRS`, which is what makes an
  in-container session stream into the viewer like any other. Auth is
  `ANTHROPIC_API_KEY`, or an already-logged-in Claude home mounted at
  `RUNNER_HOME` (`-v ~/.claude:/data/home/.claude`). Remote control is off by
  default there (it needs an interactive login).

That's GOAL.md's "single Docker container": a standalone image spawns sessions
with no host process. Keep it that way — a new runner feature belongs in
`sidecar.py`, not in a host script.

### The host sidecar (what the homelab runs)

`sidecar/sidecar.py` runs **natively on the host as the repo owner** and the
backend proxies to it over `host.docker.internal:8790` (Bearer `SIDECAR_TOKEN`):

- `POST /api/spawn` → sidecar `/spawn`: `claude -p <prompt> --session-id <uuid>
  --model <model> --remote-control` (detached). Transcript lands in the watched
  projects dir and the new conversation shows up in the viewer within ~2s.
- `POST /api/resume` / `POST /api/interrupt` → continue / SIGINT an existing run.

`model` is the `claude --model` argument the composer's **model tag** carries — a
family alias (`opus`) for a family's newest model, or a pinned id
(`claude-opus-4-5-20251101`) for an older one. Unset on spawn ⇒ `SIDECAR_MODEL`
(default `opus`); unset on resume ⇒ no `--model` flag, so the session keeps its
own. The sidecar only shape-checks the value (it goes on a command line) — the
pickable set is whatever `/api/models` returned.

### Model tags

Models are a first-class tag on both sides of the app (`frontend/src/lib/models.tsx`):
picking one in a composer (a mutually-exclusive chip per model — RichInput's
`exclusive` key, added in `@gabvdl/ui` ≥0.11) runs that session on it and the
toolbar echoes the choice; the pick is sticky (`spawnModel` in Settings, stored
server-side). Reading back, a conversation is tagged with every model that
actually produced a turn — `conversations.py` collects them (`models`, busiest
first; a session can switch mid-thread), so past conversations are tagged too
(bump `PARSER_VERSION` to force the re-parse that back-fills a new summary field).

### The tag registry (Settings → Tags)

Tags are **derived, never authored** (`frontend/src/lib/tags.ts`): `useTags()`
aggregates the projects/services on disk, the stack + `package.json` keywords a
project declares, and every mention carried by a conversation, memory, plan or
project-local skill into one list of `<kind>:<slug>` tags with per-source counts.
Nothing creates, renames or deletes a tag — rename a directory and the tag
follows; a tag that is referenced but has no directory is flagged `exists:
false` rather than hidden, since that is usually a rename something still points
at.

What a human *does* own is a tag's presentation, stored per id in the settings
store's `tagMeta` bag (server-side, like the rest of Settings): **colour** (a key
into `TAG_COLORS`, `lib/tagStyles.ts`), **icon** (a key into the curated
`TAG_ICONS` set, `lib/tagIcons.tsx`) and **description**. Unset fields fall back
to the tag kind's shipped look — projects primary/`Tag`, services sky/`Server`,
stacks violet/`Blocks` — which is why an untouched homelab looks exactly as it
did before the page existed.

`useTagPresentation()` is the queryless resolver every surface uses, so a tag
looks the same on a conversation card (`ConvMetaBits`), in a memory/plan list,
and as a composer chip (`lib/richComposer.tsx`). The description matters most in
the composer: it's the text that goes out in the prompt's `Work in the following
homelab location(s):` bullet, so editing it here changes what a spawned agent
actually reads — the Tags editor previews that exact line.

The icon set is curated rather than lucide's full `icons` barrel on purpose: tag
glyphs render synchronously all over the app, so they can't come from a lazy
chunk, and pulling ~1500 icons into the main bundle to serve one settings page
is a bad trade.

### Effort (claude) vs. thinking (pi)

How hard a run works a turn is **one knob shown two ways**, per harness — never
both at once:

- **claude** gets an `EffortSelect` (`--effort low|medium|high|xhigh|max`), the
  CLI's own scale. Sticky as `spawnEffort`; `""` means "send no flag" and let the
  CLI default. The sidecar validates against an allow-list (`EFFORT_LEVELS`),
  unlike `--model`, because the scale is a closed set.
- **pi** keeps the binary `ThinkingToggle` — it has no `--effort`. `_effort_args`
  emits nothing for it, so a stale pick can't leak onto a pi command line.

The composers send only the active harness's control (the other goes out unset).
Reading back, `conversations.py` mines the `effort` field the CLI stamps on each
**assistant record** (it rides on the record, not the message) into `efforts`,
busiest first — same shape as `models`, rendered by `EffortTags` on conversation
rows and in the detail header. Empty for pi runs and for transcripts predating
the flag, which is a valid state, not a blank chip.

### A cron job's run config lives in its prompt file's frontmatter

A scheduled job has no composer to pick a harness/model/effort in, so it says so
itself: the leading YAML block of `.claude/agents/<job>.md` carries `harness`
(`claude`|`pi`), `model` (a family alias or a full id), `effort`
(`low|medium|high|xhigh|max`, or `none` for no flag) and `thinking`
(`true`|`false`). `cron.run_config()` reads them, `_fire_cron_job` passes them
to the same `_send_message` the composer uses, and the resolved config comes
back on the job as `runConfig` so Settings → Cron shows it as chips. Every
homelab job currently declares `claude` + `sonnet` + no effort.

Two deliberate properties: the keys are a **subset of Claude Code's own agent
frontmatter** (`model` means the same thing there), so a job's prompt file stays
a usable `.claude/agents` subagent; and an invalid value is **dropped, not
raised** — a firing on the defaults beats a job that stops firing, and the
missing chip in the UI is the feedback. The store's per-job `model` field is now
only the fallback for a file whose frontmatter sets none.

### CTAs replaced the conversation hooks

There used to be a **hook** scheduler here: ten seconds after any run ended, it
resumed that same conversation with a follow-up prompt. It worked, and it was
the wrong shape. Two reasons, both structural:

- **It cost a full transcript per firing.** `--resume` is a fresh `claude -p`
  process, and Claude Code rebuilds that process's environment preamble (cwd,
  git status, date, …) from scratch; any byte of drift there — near-guaranteed
  on a checkout other agents are also committing to — invalidates every cache
  breakpoint after it. Sampled real transcripts confirmed it:
  `cache_read_input_tokens` collapsed to the cross-session baseline and
  `cache_creation_input_tokens` re-created the *entire* prior conversation,
  every firing, regardless of the delay. A tidy-up pass routinely cost as much
  as the feature work it followed. The `maxContextK` cap was a bandage.
- **Half the work didn't need a model.** "Summarise what happened" is mostly
  aggregation — which prompts were given, which tools ran, what net change
  landed in git — and the backend already parses transcripts and buckets
  commits. Asking a model to re-read a transcript to derive that was paying
  premium rates for a `GROUP BY`.

So the prompts stayed and the scheduler went. A CTA is a prompt file plus a
button under the last message of a finished conversation; **nothing runs until
a gesture says so**, and the gesture picks the mode:

- **click** → the prompt lands in that conversation's resume box, editable
  before it is sent (`Cta.prompt` ships with the list, so there is no second
  round-trip);
- **long-press** (or right-click) → `POST /api/ctas/{id}/run` sends it at the
  session in the background, no composer involved.

The same prompts are selectable as `cta:<id>` tags in every composer, where they
fold their whole prompt file into the typed message rather than a bullet.

With no scheduler there is **no loop guard and no once-per-session ledger** —
a CTA runs because it was pressed, so re-running one is a feature, not a bug the
design has to prevent. `meta.ctas[ctaId]` still stamps each background run (the
map is *merged*, newest wins), which is what the CTA pill and the COMPLETED
stamp read. A CTA passes `model`/`effort`/`thinking` as `None` unless its prompt
file's frontmatter sets them, which `_send_message` reads as "keep whatever the
session already ran with".

The seeded `complete` CTA is the old post-task pass, done better: it runs
`conv-scaffold` (see `scaffold.py`) for the algorithmic half, hands that
scaffold to **one subagent** to fill in and publish — the scaffold is already a
compressed view, so the analysis never needs the parent's context — and tidies
up (memories, docs, artefacts, worktrees, changelog) in parallel.

Native Claude Code hooks (`.claude/settings.json`) are **read-only** here
(`GET /api/claude-hooks`, listed at the bottom of the CTAs page): writing that
file from the container would be host-level code execution — see the
trusted-caller gate above. They are now the only hooks the viewer knows about.

Install/manage the sidecar with `sidecar/install.sh` (systemd **user** unit
`claude-sidecar`); config comes from the repo `.env` (`SIDECAR_TOKEN`,
`SIDECAR_PORT`). See `sidecar/README.md`.

## Schema drift: the transcript format is not ours

`backend/conversations.py` parses Claude Code's **internal, undocumented** JSONL
transcript format, so a CLI update can silently change what it emits. The
early-warning tool is `scripts/schema-drift.py`:

```bash
scripts/schema-drift.py            # newest transcripts vs committed baseline
scripts/schema-drift.py --update   # accept reviewed drift as the new baseline
```

It does two things: diffs a **structural footprint** (key paths + content-block
types, never values — `scripts/schema-baseline.json` is safe in git) of recent
`~/.claude/projects` transcripts against the baseline, and runs a **parser
canary** (the real `parse_conversation` cross-checked against raw record
counts: assistant turns with usage, token totals, models, timestamps). Exit 1
on drift or a failed invariant. Run it after a Claude Code update or whenever
conversations render oddly; on drift, decide whether the parser needs to handle
the new shape, then `--update`. Every transcript record carries the CC
`version` stamp — a new version alone is INFO, not failure (releases are ~daily
and almost never drift).

## Deploying

**Never** `compose up --force-recreate` this service — use the blue-green script,
which validates a fresh build on a standby container before touching the live one:

```bash
~/homelab/services/ai-agent/deploy.sh   # the homelab shim's copy (runs from anywhere)
```

It builds a new image, brings up a second `ai-agent-deploy` standby on
`homelab_main`, health-gates it on `/api/health`, cuts Traefik over to it
(file-provider hot-reload), recreates the canonical `ai-agent` on the new image,
then restores routing and removes the standby — zero downtime, and if the standby
never goes healthy the live container is left untouched. An `flock` on
`/tmp/ai-agent-deploy.lock` serializes deploys; a second run aborts immediately.
`deploy.override.yml` is only for the standby slot (renames the container, drops
the host port, joins the external `homelab_main` network).

## Standalone mode: a missing catalog is a valid state

The homelab `docker-compose.yml` mounts ~8 host paths (transcripts, memories,
screenshots, services/, plans, …), but the image must also run with **none** of
them — that's the GOAL's "generic first-run setup" / one-liner `docker run`.
`docker-compose.standalone.yml` is that shape: a workspace + a data volume, and
nothing else required.

So **every catalog degrades to an empty list when its dir is absent** — never a
500. (`/api/templates` used to raise; it doesn't now.) A *named* item that isn't
there is still a 404 — it's only the missing-mount case that's benign. When you
add a catalog backed by a mount, follow that rule.

The guard is a smoke test — run it after touching mounts, env wiring or a
catalog route:

```bash
scripts/standalone-smoke.sh            # boot the current image on a bare workspace
scripts/standalone-smoke.sh --build    # build first, then smoke
```

It boots the image with only a throwaway workspace + data dir (and
`RUNNER_IN_CONTAINER=1`), asserts every catalog endpoint and the PWA shell answer
200, that the bundled `claude` CLI is on PATH and its runner answers `/health`,
and greps the log for tracebacks (a background indexer/watcher crash won't fail a
request). Exit 1 = regressed.

Standalone **spawns** sessions now (the in-image runner, above). The one thing it
still can't do is **build** off-homelab: `frontend/.npmrc` pulls `@gabvdl/ui` from
the host-bound verdaccio, so a standalone build needs that registry — a prebuilt
image runs anywhere. Tracked in [GOAL.md](GOAL.md).

## Local development

There is no lint/test pipeline. Frontend and backend are built into the image;
to iterate:

- **Frontend only**: `cd frontend && npm install && npm run dev` (Vite on `:5180`).
  It needs the backend API — point it at a running container or backend.
- **Full stack**: rebuild + redeploy via the homelab shim's `deploy.sh`, or hit the running
  container directly on the host at `http://127.0.0.1:8096` (plain HTTP, bypasses
  Authelia — handy since new `*.lab` domains can't get an LE cert here).
- **Health**: `curl -s http://127.0.0.1:8096/api/health` → `{"ok":true}`.

After changing the homelab shim's `traefik.yml`, run `make link-traefik-configs`
from the homelab repo root (the file provider watches `config/dynamic/`, not the
shim's file).

## Gotchas

- **Read-only mounts**: most of the repo context is mounted `:ro` (services/, .git,
  transcripts, memories, screenshots, and the root `data/` tree). Only `.claude/`,
  `~/projects` (host-side; `/workspace/projects` in-container), `CLAUDE.md` and the service's own `data/` (SQLite store) are
  writable. The container runs as `1000:1000` so edits keep repo ownership.
- **Root `data/` tree**: mounted `${PWD}/data:/workspace/data:ro` and surfaced in
  the file tab for visibility only. The indexer exposes just text files (see
  `DATA_TEXT_EXTS` in `main.py` — md/json/py/log/…; binaries like the PNG
  thumbnails are skipped) and marks them non-editable. `data/tmp/` is in
  `IGNORE_DIRS`, so scratch files and secrets (e.g. `*-creds.env`) never appear.
- **Token counts cost API calls**: they hit the `count_tokens` endpoint
  (`ANTHROPIC_API_KEY` from `.env`) and are recomputed only when a file's content
  hash changes, after a debounce — don't add code that recounts on every request.
- **Conversation "finished" state**: three signals, strongest first. (1) A
  `notify-done` notification **plus** a last message ending with the literal
  `DONE` marker (see the root CLAUDE.md "When work is done"). (2) The **exit
  watcher**: a background thread in `main.py` (`_watch_sidecar_runs`) polls the
  sidecar's `/sessions` every ~2s and stamps `finished` (+`exitedAt`) the moment
  a sidecar-launched run's process dies — so runs whose prompt never said DONE
  don't sit on "running". The stamp is distrusted if the transcript later grows
  past `exitedAt` (a terminal/remote-control continuation, `_outlived_exit`),
  and in-flight resume/interrupt swaps are shielded via `_sidecar_busy`. (3)
  The recency fallback (`RUNNING_WINDOW_SECS`, 600s) for anything the watcher
  never saw live (terminal sessions, runs that died while the backend was down).
- **Conversation "paused" state**: a run whose last act was parking itself on a
  trigger — a trailing `ScheduleWakeup` (non-`stop`) or a pending `Monitor` — is
  shown `paused` instead of `running` (`waiting_trailing` in `conversations.py`,
  resolved in `main.py:_resolve_state`). The flag clears on any later tool call,
  real user turn, or interrupt — not on the same turn's closing text.
- **Two containers briefly share `data/`** during a deploy cutover (SQLite); this is
  a short window and intentional — don't "fix" it by locking the DB.
- **A resume must never answer 200 without a live run behind it.** The composer
  echoes the typed turn optimistically and only drops the echo once the
  transcript grows, so any resume that reports success but starts nothing leaves
  the UI on "sending…" forever. Two guards keep that honest and must stay:
  `sidecar.py` watches a freshly launched `claude` for `LAUNCH_PROBE_S` and 502s
  with its log tail if it dies on the spot (a `--resume` whose transcript isn't
  found exits 1 in ~2.5s), and `/resume` only reports `reused` for a process that
  is *genuinely still running* — a finished-but-unreaped child is a zombie, and a
  zombie answers `os.kill(pid, 0)` (hence `_reap()` + the `Z` check in `_alive`).
  `reused` means the turn was **dropped**, so the backend returns 409, not 200.
  The backend always resumes with `force: true`: a genuinely-live run is stopped
  first (`_stop_run` — SIGINT the group, wait up to `STOP_WAIT_S`, escalate to
  SIGKILL) and the resume then launches, so sending a message into a running or
  paused conversation restarts it on the new turn instead of 409ing. `reused`
  can now only come from a sidecar predating the flag.