The first automated run of Command’s new wiki distilled 20 of my Claude sessions with a Haiku subagent, and Haiku recorded a confident, wrong verdict: a “structural failure” judgment against DemandForge, one of my ventures, when the session it was summarizing had proposed that reading and then retracted it. It also wrote a brainstormed pivot as if I’d committed to it, and manufactured a contradiction flag out of two truncated summaries [measured, PRs #117 and #120]. The whole point of the wiki is that other sessions trust it, so a wrong verdict there doesn’t just sit quietly; it propagates. That run got reverted and re-distilled on a capable model, and the failure reshaped the pipeline’s design.
The wiki is the memory layer of Command, the operator I run my local projects through. It’s 134 interlinked markdown pages that answer “what is this venture, what’s decided, what’s open, what happened to that dead project” without re-mining raw chat logs, modeled on Andrej Karpathy’s LLM wiki pattern. Claude and I built it over four days (PRs #92 through #127 in Command’s private repo, July 3 to 6). This post covers what it is and the design decisions that let me trust what it says.
The pattern, and what Command already had
Karpathy’s setup is three layers: immutable raw sources at the bottom, an LLM-maintained wiki of synthesis in the middle, and a schema document on top that defines page types and workflows. Three operations keep it alive: ingest (one new source updates the pages it touches), query (answers cite pages, and good answers become pages), and lint (find orphans, stale claims, contradictions). His thesis is that humans abandon wikis because maintenance outgrows value, and LLMs don’t tire of cross-referencing.
Command already had two of the three layers. The bottom existed in volume: session handoffs, research studies, a durable JSON store, git history, and the chat transcripts themselves (which Claude Code deletes after 30 days by default, so check your settings). A proto-wiki existed too, as one-fact memory files with links between them. What was missing was the middle: nothing connected sources into stable, citable pages. The build was mostly about that middle layer, and about the honesty machinery the generic pattern doesn’t specify.
Every page carries frontmatter (type, maintenance tier, visibility, status, sources), a one-line summary, dated “State as of” headers, and provenance tags on nontrivial claims: [measured: source], [derived: source], [assumed]. Contradictions between sources get flagged in place, never silently resolved. There are 12 page types in the schema, from ventures and decisions to internal concepts and retro-mined patterns [measured, my count of the schema].
Wire the guaranteed reader before writing pages
The first shipped PR contained the schema, 57 seed pages, a deterministic lint script, and one thing that mattered more than any of them: command-start, the skill that opens every Command session, was wired to read the wiki from day one. It now reads the current-focus snapshot, skims the index, and (a few PRs later) runs a diff of raw sources against what’s been ingested, surfacing a one-line nudge only when the wiki is behind.
The failure mode this defeats is the write-only wiki. A knowledge base nobody is forced to read rots invisibly, because nothing downstream breaks when it’s wrong. Giving the wiki a guaranteed reader on session one means staleness and errors surface as friction in real work, not in an audit nobody schedules.
Treat fabrication as the default failure mode
The founding safety rule came from a measurement, not a principle. The same day the wiki was designed, a retrospective over Command’s sessions caught 2 of 4 reader agents fabricating their extraction quotes wholesale, inventing them rather than pulling them from source; an ad-hoc grep gate was the only thing that caught it [measured, the 2026-07-03 retro study]. So the wiki inherited that gate as law: every quote and specific claim gets grep-verified against its cited source before a page saves.
The backfill ran under it. All 48 accumulated sources (41 handoffs, 7 studies) were synthesized oldest to newest so later sources correct earlier ones, and all 221 anchor quotes were verified against source before anything was applied, with zero fabrications recorded [measured, PR #95].
The graveyard takes the same idea to its cheapest extreme. 59 of the 61 project pages cover dormant repos, and those pages were generated by a deterministic scaffold that transcribes only mechanically harvested git facts plus each README’s first line. The PR states it plainly: no LLM synthesis beyond harvested facts, so fabrication is structurally impossible [measured, PR #100]. Where synthesis adds nothing, the honest move is to not let the model write at all.
A cheap distiller launders; the fix was structural, not just a bigger model
The self-updating pipeline is the part that makes the wiki compound without me. A deterministic scanner (zero tokens, no model) enumerates the day’s genuine sessions past a watermark, which is a durable cursor marking how far the scan has read. It drops automated and trivial sessions, and a subagent distills each survivor into a committed digest; a batched ingest then folds digests into pages and opens exactly one PR. The watermark only advances after a re-scan confirms every eligible session actually got a digest, so a partial run stalls visibly instead of silently skipping sessions forever.
Then came the run described at the top. Distillation runs through the Agent tool at flat rate, so model choice looked free, and the first run used Haiku. The result was confident misreading of exactly the sessions that matter most, the strategic ones. The correction landed in two parts [measured, PRs #117 and #120]:
- A capable-model floor. The distiller now runs Sonnet at minimum and Opus for substantive or strategic sessions, chosen by a cheap classifier that can escalate but never downgrade. The reverted batch was re-distilled as 15 Opus and 5 Sonnet digests, and the three misreads corrected.
- A quarantine tier that doesn’t depend on the model. Any strategic claim (a verdict flip, a pivot, a kill-or-revive call) whose only source is a machine digest cannot become wiki prose. It has to be written as a visible
⚠ PROPOSED (cold-log)block that I ratify or reject, and a deterministic lint check backstops the rule. Before a digest claim becomes a page assertion at all, the ingest opens the underlying transcript and confirms the claim survives the actual turns.
The lesson I’d carry to any synthesis layer: a cheap summarizer doesn’t fail loudly, it launders. Confident-but-wrong output enters a store that everything downstream treats as ground truth. The durable fix was provenance that quarantines a machine-only claim regardless of which model wrote it; upgrading the model only lowers how often that quarantine has to fire.
Private first; publishing is a flip behind a wall
Every page has carried visibility: private frontmatter since the seed, so a future public export is a flip, not a redesign. Two categories (people, finances) are hard-excluded from export in code, and the gate refuses them even when a page is hand-marked as a candidate. From the browser I can stage a page as public-candidate, and a nightly review batches the staged pages, scans each for leaks (internal source references, links into private pages), checks the prose against this site’s voice bar, and opens one PR. The merge is the publish wall; nothing self-publishes.
Building the gate as code got tested by its own review passes. Two latent leaks were caught before anything was ever published: an emit mode that shipped every private page’s metadata into inline JSON and filtered it client-side, and a public page’s link list that exposed private page paths [measured, PRs #107 and #125]. Allowlist-by-construction held as a design, but it took multiple passes to actually hold as an implementation.
A browser that finds pages isn’t one that reads them
The first visual browser shipped the same day as the seed: a zero-dependency script that serves the live wiki as searchable cards plus a link graph. A structured usability review of it returned a verdict I’m quoting because it’s the whole redesign in one line: “finding a page is fast and pleasant, but reading it is impossible” [measured, the review study]. Cards showed only summary lines, clicking led nowhere, and graph labels rendered six pixels tall.
The redesign made it a reader. Clicking any card or graph node renders the full page body in place with working internal links, incoming and outgoing link chips, and a focused connections map. The graph got a deterministic force layout over real cross-links (catalog links toggle separately so hub edges don’t drown the structure), full-text search matches page bodies with highlighted snippets, and an isolated filter surfaces pages with zero cross-links (44 of 134 at the redesign), which is coverage debt made visible [measured, PR #127]. A per-page flag control writes a review note that the next lint treats as a priority input. I kept the custom viewer instead of re-platforming onto Obsidian or Quartz; the requirement was a fast way to use the thing, not a knowledge product.
One defect from this arc is worth stealing as a testing rule. The first browser rendered completely blank in a real browser: two regex literals inside a template string lost their backslashes when the template evaluated, and the resulting syntax error killed the entire inline script. It had only ever been curl-tested, which checks the HTML string, not whether the JavaScript executes. The two redesign PRs since were verified by driving real Chrome with Playwright (the first passed 40 of 40 DOM-grounded checks), because for a rendered UI, the browser is the only ground truth [measured, PRs #114 and #125].
What’s live, and what isn’t yet
The nightly digest pipeline is live: it runs its cycle end to end, distilling the day’s sessions, folding them in, and opening one gated PR for me to merge. What isn’t live is the public export. The route is decided (a section of this site, rendered through the same allowlist gate), but the sanitizing body renderer isn’t built yet, so today’s export strips page bodies entirely rather than risk shipping one unvetted paragraph. Until that renderer clears the export’s defense-in-depth sanitization floor, a public slice of the wiki stays a plan, not a result.
Every mechanism above is the same observation applied at a different layer: an agent will sometimes write things that aren’t in the source, and it will do it with full confidence. So quotes get grep-verified at ingest, dormant projects get deterministic scaffolds instead of synthesis, machine-only strategic claims sit in quarantine until I ratify them, and everything that publishes waits for a human merge. Four days of building kept confirming the premise: the fabricating readers, the Haiku run, the two latent export leaks. Design for the failure you’ve already measured.