Gabriel Vidal
ebae75df94
Idle CPU sat at ~58-110% and the catalogs took seconds, because the hot paths all did O(everything) work on a timer or per request: - the indexer os.walk'd the whole workspace every 10s to find ~280 files, and the watcher rglob'd ~2600 transcripts every 2s (pathlib allocates a Path per entry) -> fsscan.walk_files (os.scandir) + a cached discovery walk, refreshed every WALK_INTERVAL_SECS and forced on save; - a live session's growing transcript was re-parsed from byte 0 on every tick (O(n^2) over a session) -> conversations.ParserState is now resumable, and the indexer feeds it only the appended lines from a stored byte offset; - the watcher already knew which transcripts changed but threw it away -> scan_transcripts(only=…) imports + re-parses just those, instead of re-walking both source dirs and the whole archive; - /api/services forked two `git log`s per service per request (11s for 34) -> githist.BucketedHistory walks each repo's log once, buckets commits per dir and caches on the repo HEAD; standalone project repos get a TTL cache; - every list request json.loads'd ~900 summary blobs and deep-copied the meta sidecar per card -> decoded summaries are cached in the store (read-only) with a version counter, and MetaStore.peek() skips the copy on hot paths. Also: SQLite in WAL, file word/char stats stored on change, and the skills rollup no longer SELECT *'s every summary blob to read one column. Verified: parse output is byte-identical to the old parser across all 905 archived transcripts (full and incremental), and idle CPU drops ~58% -> ~7% measured against the live container on the same workload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
36 lines
1.2 KiB
Python
36 lines
1.2 KiB
Python
"""
|
||
Fast filesystem walking.
|
||
|
||
``pathlib.Path.rglob`` builds a ``Path`` object per entry, and the transcript
|
||
dirs (~2 600 files) get re-walked by the watcher every couple of seconds — that
|
||
alone was a double-digit chunk of this service's idle CPU. ``os.scandir``
|
||
returns dirents with a cached type and one ``stat`` per file, which is ~2×
|
||
faster and allocates nothing, so the poll loops use these instead.
|
||
"""
|
||
|
||
import os
|
||
|
||
|
||
def walk_files(root, suffix: str):
|
||
"""Yield ``(abs_path, mtime, size)`` for every file under ``root`` whose
|
||
name ends with ``suffix``. Symlinked dirs are not followed; unreadable dirs
|
||
and vanished entries are skipped."""
|
||
stack = [str(root)]
|
||
while stack:
|
||
try:
|
||
it = os.scandir(stack.pop())
|
||
except OSError:
|
||
continue
|
||
with it:
|
||
for e in it:
|
||
try:
|
||
if e.is_dir(follow_symlinks=False):
|
||
stack.append(e.path)
|
||
continue
|
||
if not e.name.endswith(suffix):
|
||
continue
|
||
st = e.stat()
|
||
except OSError:
|
||
continue
|
||
yield e.path, st.st_mtime, st.st_size
|