docgen

docgen — per-directory READMEs for Claude agents

A small CLI that walks the repo, picks a directory whose contents have changed since the last analysis (or were never analyzed), feeds the directory's files to claude -p, and writes the response as <dir>/README.md. Every generated README starts with an HTML comment marker so it's distinguishable from hand-written ones.

The point is not to document the code for humans. It's to give the NEXT Claude agent a one-paragraph index per directory so it can answer "what does this code do" without re-grepping the whole tree.

Usage

tools/docgen/docgen                  # analyze one stale/uncovered dir, exit
tools/docgen/docgen --until-done     # loop until every dir has fresh docs
tools/docgen/docgen status           # print coverage report (no analysis)
tools/docgen/docgen --dir <path>     # force-analyze a specific directory
tools/docgen/docgen --dry-run        # print what would be analyzed, don't call claude

Optional flags:

• --model <name> — override the Claude model (default: CLI default)
• --timeout-ms <n> — per-call timeout (default 5 min)

How it decides what to analyze

State lives in .docgen/state.json at the repo root. Each entry records the directory path, the file-name → mtime fingerprints from the last successful analysis, and the timestamp.

On each invocation, docgen walks the repo, picks the first directory that's either:

• never-analyzed — no entry in state, OR
• file-set-changed — files added or removed, OR
• file-modified — any file's mtime differs from the recorded value.

Then it feeds that directory's files to Claude with the prompt in prompt.md, writes the response to <dir>/README.md, and updates state. With --until-done it loops until nothing remains.

What it skips

Directory names anywhere in the tree: node_modules, .git, dist, build, .tooling, .docgen, .next, .cache, coverage, __pycache__, .venv, venv, .pytest_cache, .turbo, .parcel-cache. Hidden directories other than .github are skipped.

File extensions: binaries (.png, .pdf, .zip, .wasm, …), lock files (.lock, .lockb), media (.mp4, …), fonts (.ttf, …).

Per-directory caps:

• files larger than 1 MB skipped individually
• directories with more than 60 files skipped entirely (asset dumps)
• files truncated to 8 KB each when assembled into the prompt

How it calls Claude

Same pattern as the evolve loop's bots/scripts/backtest/factory/evolve/ claude.ts: spawn('claude', ['-p', '--output-format', 'text']), prompt + context piped on stdin. No shell interpolation anywhere. The runner is injected for tests so the test suite never actually shells out to the CLI.

Prompt design

prompt.md instructs Claude to produce a tight, agent-oriented README: Purpose (1–2 sentences), Files (one-bullet-each summary), Dependencies / collaborators, optional Gotchas. The hand-tuned constraints are inline in prompt.md — read that file for the actual contract.

Programmatic API

docgen.mjs exports its building blocks for tests and future use:

• loadState(root) / saveState(root, state) — JSON state IO
• walkDirs(root) — generator over content-bearing directories
• selectFiles(dir) — files we'd actually feed to the LLM
• needsAnalysis(root, dir, state) — staleness reason or null
• assembleContext(root, dir, files) — the string sent to Claude
• analyzeOne(root, { runner, promptText, ... }) — one-dir pipeline
• computeStatus(root) — coverage report

Inject runner in tests so the suite is hermetic.

Getting Claude to actually use the READMEs

A README in a sub-directory is shelfware unless something puts it in front of the agent — Claude Code only auto-loads CLAUDE.md at the project root, not per-directory READMEs. The inject-readme-context.mjs PreToolUse hook closes that gap.

When the agent calls Read / Edit / Write / Glob / Grep, the hook walks from the target path up the ancestor chain to the git root, collects each directory's README.md (hand-written or generated, both useful), and emits the contents on stdout. Claude Code feeds hook stdout into the next LLM turn as additional context. The agent therefore "sees" the pyramid (root → subsystem → directory) before it touches the file, without ever needing to explicitly Read the READMEs.

Registered in .claude/settings.json so it's per-repo, committed, and fires automatically once you pull this branch. The hook:

• never blocks — it always exits 0; any failure inside silently emits nothing so a broken hook can't break the agent
• dedups within a session via /tmp/docgen-injected-<session_id>.json so the same READMEs aren't re-injected on every Read in the same directory
• caps each README at 6 KB and walks at most 6 ancestors to bound token cost on deep trees
• skips when the target IS a README so reading a README doesn't re-inject its ancestors

Verified empirically (see inject-readme-context.test.mjs + a real claude -p test): the hook fires, walks up properly, dedups, and the LLM's response quotes README-only phrasing — concrete proof the injected content shaped the answer rather than just the source file comments.

Disable the trace log by setting DOCGEN_HOOK_TRACE=0 if /tmp/docgen-hook-trace.log becomes noisy. The trace exists so a future debugging session can confirm the hook is firing without re-running the empirical test.

Auto-refresh on every push (optional git pre-push hook)

Per-repo, opt-in. After install, every git push from this clone spawns a detached background process that runs docgen, commits any README changes, and pushes them back to the same branch. If the branch was merged + deleted server-side between push and refresh, the hook opens a follow-up PR via gh pr create instead.

Install / uninstall

./tools/docgen/install-push-hook.sh            install
./tools/docgen/install-push-hook.sh uninstall  remove
./tools/docgen/install-push-hook.sh status     show install state + recent log

Coexists with tools/secret-scrub: if you have core.hooksPath set (secret-scrub does that), the installer drops pre-push alongside the existing pre-commit there. Refuses to overwrite a non-docgen pre-push hook — back it up first if you want to convert.

How it works

1. git push fires the pre-push hook.
2. Hook spawns a detached background process via setsid -f / nohup & and exits 0 immediately — your push isn't blocked.
3. Background process sleeps 5s (lets your foreground push settle), then runs tools/docgen/docgen --until-done --parallel 4.
4. If READMEs changed, commits them and pushes back to the same branch with DOCGEN_HOOK_SKIP=1 set (the hook checks that env var at the top and noops if set — that's the loop guard).
5. If the push fails because the branch is gone (PR merged + branch deleted), opens a fresh PR titled docs: refresh docgen READMEs (auto) against main.

Following along

tail -F /tmp/docgen-push-hook.log

The background work logs everything there. The user-facing git push shows nothing extra — the refresh is fully detached.

Failure modes

• docgen run fails → logged, no commit, no push. Foreground push was already accepted.
• commit fails (no diff, hook failure) → logged, exit.
• push fails AND no gh CLI → doc commit stays local; logged.
• gh pr create fails → logged. User can gh pr create manually from the branch the hook left behind.

The hook is never load-bearing for correctness — if it does nothing, the only consequence is staler READMEs until the next manual docgen run.

Future work (intentionally not v1)

• Per-touched-files mode so the refresh only re-analyses dirs containing files that just changed in the push, not the whole tree — cheaper for incremental edits.
• .docgenignore pattern file for repo-specific excludes beyond the built-in skip lists.
• Cost log (mirror the evolve loop's cost-log.jsonl) so we can see $/run as the corpus grows.
• Cross-directory linking — let Claude's response reference sibling directories' READMEs so the index becomes navigable end-to-end.