Skip to content

00 What claude-toolbox caught while AI was building capy

"I took the shortcut."
— your LLM

Three /kk:review-code runs on the same capy diff. Each time, the agent read the code first and got to the review checklists late or not at all. In the third run it was told to "follow the review process exactly". It loaded 28 files and still read the diff first. The skill itself said to start with the diff, so the fix was to change the order, not to write a sterner prompt.

capy is a local MCP server, written in Go, that keeps large outputs out of a coding agent's context window. It was built from start to finish with claude-toolbox: skills, sub-agents and config for Claude Code and Codex, run as /kk: commands from design through review, tests and docs. Below is what they caught along the way, quoted from capy's own records. Nothing is staged.

Left behind All 16 skills now load their method before they touch your code. From review-code/SKILL.md: "The standard workflow below is strictly sequential."

From the same record: "Enforcement is convention, not mechanism."

$ /kk:review-code # three runs, same diff

docs/adr/0004-skill-workflow-ordering.md

  1. 15 - **Run 1.** … When pressed, the agent admitted: "I had the full diff visible, recognized the changes as a clean source-kind refactor, and decided I could pattern-match a review from the diff alone. I took the shortcut."
  2. 17 - **Run 3.** User explicitly asked the agent to "follow the review process exactly." Agent loaded 28 files but still diffed first, then circled back to read checklists after findings were already forming.
  3. 21 This is a workflow-ordering bug, not an agent-discipline bug.

# where the three runs happened

docs/feat/done/kubernetes-support/design.md

  1. 483 **Flagged by:** Task 7 P0 verification dry-run (2026-04-19) — user-led `/kk:review-code` runs in a sibling Go project (`capy`) reproduced the same process-bypass behavior across three consecutive invocations.

01/05 Design

One reader handled a new line type. The other two silently dropped it.

capy's session vault read Claude transcripts with three separate readers: one for search, one for show, one for the terminal viewer. capy's knowledge base had a recurring bug on record: a new line type handled in one reader was silently lost from the other two. The plan to add Codex sessions cited that record and didn't add three more readers. It replaced them with one decoder per format.

Before any of the old reader code changed, Task 1 recorded what those readers produced for every session file on the machine, 700 of them. Every later task had to match that output byte for byte. The final /kk:test run compared 698 and found 0 mismatches.

Left behind ADR-031: "A new line type is handled once, in the decoder; the "three parsers must stay in sync" convention is retired." And TestGolden_CoversEveryLineType, which "enforces the one-case-per-arm rule mechanically".

$ /kk:design # command inferred from the record

docs/feat/done/codex-vault-sessions/design.md

  1. 19 There is a second, pre-existing cost this feature exposes: the three Claude readers duplicate the same pass-1 logic, and the project's knowledge base records the recurring bug shape — a new line type handled in one reader is silently lost from the other two. Adding a Codex trio the same way would double that surface.

# Task 1, before any reader code changed

docs/feat/done/codex-vault-sessions/tasks.md

  1. 26 - [x] 1.6 Produce the baseline on `master` and re-run unchanged: 0 mismatches (700 files baselined at `$REPO/bench-results/vault-parity-baseline.tsv`, gitignored; compare run: 699 compared, 0 mismatches, 1 skipped as a live session)

$ /kk:test # final verification, 2026-09-16

docs/feat/done/codex-vault-sessions/tasks.md

  1. 211 - [x] 14.1 Run `/kk:test` skill to verify all tasks — … parity canary 698 compared / **0 mismatches** / 2 skipped as live sessions / 31 new since baseline

02/05 Design review

This upgrade would have stopped every existing vault from opening.

The same plan then went through four separate /kk:review-design runs before any code existed. Two of them found that a new index ran before the migration that creates its column. On any existing vault, that fails at open. The same pass found that merging an old vault would silently drop about 1 in 11 sessions.

Every claim was re-checked against the code and the real session files on the machine before the plan changed. The design was revised on 2026-09-13; the code landed two days later.

Left behind An invariant in the implementation plan: "schemaSQL may therefore reference only columns that exist in every historical schema." And TestMigrate0006_LegacyVaultOpensThroughStore, which opens a pre-migration vault through the store.

$ /kk:review-design # four independent runs, R1–R4

docs/feat/done/codex-vault-sessions/design-triage.md

  1. 3 **Date:** 2026-09-13 · **Inputs:** four independent `kk:review-design` runs (R1–R4) · … Nothing below is taken from a review on trust.
  2. 11 | A | `idx_sessions_parent` in `schemaSQL` breaks legacy vault open (R3, R4) | **Confirmed** | P0 | Fix |
  3. 44 On a pre-0006 vault `CREATE TABLE IF NOT EXISTS` is a no-op, so `CREATE INDEX IF NOT EXISTS idx_sessions_parent ON vault_sessions(parent_uuid)` in `schemaSQL` (…) fails with "no such column" and every existing vault refuses to open.
  4. 60 … **41/464 (8.8 %)** would return an error. … so merging any pre-0006 vault silently drops ~1 in 11 sessions.

04/05 Code review

The tests could chmod a developer's real key file.

A capy change made setup repair the permissions on database key files. The full test suite passed. Then a fresh reviewer agent, one that hadn't written the code, noticed that the existing setup tests read the developer's global config. After this change, running them could change the permissions on the developer's real key.

The external model reviewing alongside it reported no issues and claimed protections the code didn't have. The record says so and doesn't adopt them.

Left behind The tests now isolate global config in TestMain. A probe with a deliberately loose 0644 key file "left that file at 0644". The non-atomic repair is recorded as a known limit, with the hardening steps written down.

$ /kk:review-code:isolated

docs/feat/reviews/issue-124-key-file-permissions.md

  1. 26 `kk:review-code:isolated` used a fresh `code-reviewer` agent and two-step PAL review.
  2. 28 The code reviewer found **P2: existing setup tests inherited global configuration and could chmod a developer's real absolute key file**.
  3. 30 PAL reported no issues. Its broad claims of TOCTOU protection are not adopted: opened-descriptor validation protects the read admission check, but pathname-based setup repair is not atomic.

05/05 Spec review

The live tests passed. None of them exercised the default wait.

capy_execute waits up to 25 seconds, then hands back a "still running" receipt so the agent isn't stuck. The live acceptance runs on real Claude Code and Codex hosts all passed. But they were short commands with the wait raised to 30 seconds, so none of them ever hit the default.

The closing spec review caught it. Both hosts were rerun with a 27-second command at the default wait. Both answered "running" before it finished, and the full output was recovered afterwards.

Left behind Live evidence from both hosts, retained with exact inputs: 27,011 ms on Codex and 27,012 ms on Claude.

$ /kk:review-spec:isolated # command inferred from the record

docs/feat/done/routine-command-capture/task17-spec-review.md

  1. 40 The independent closeout review initially identified a P2 acceptance gap: short commands specifying `wait_ms=30000` did not exercise the default 25-second attached wait.

# the rerun

docs/feat/done/routine-command-capture/task17-verification.md

  1. 110 `wait_ms=30000` calls completed normally and zero-wait recovery worked.
  2. 113 The final review identified that those short calls did not measure default-wait viability. Both trusted hosts were therefore resumed for one additional 27-second command with `wait_ms=25000` and `timeout=40000`. Both returned a healthy `running`/`recording` response before command completion, without a host timeout.
  3. 122 | Codex | `70ec2b9a-37e3-4dcc-9651-c34fe0c764a5` | 27,011 ms | 44 |
  4. 123 | Claude | `34426298-5d84-42cf-9194-43c9bfe057a4` | 27,012 ms | 46 |

Each catch left something behind.

A rule, an invariant, a test, a recorded limit. Every excerpt above is the record's own text, and every line number links to it.

673
commits since 2026-03-18
37
ADRs
27
finished feature records

capy, as of 2026-11-14