I Built docproof: a CLI That Tests Your Documentation by Running It

Last month I was setting up a library I hadn't touched in a year. I followed my own README, step by step, and the third command failed. The package had moved on, the docs hadn't. I fixed the README, felt a little embarrassed, and then realized something: I had tests for every function in that project, but nothing tested the one file every visitor reads first.

That gap bothered me enough to build a tool for it. It's called docproof, and it does one thing: it runs the code blocks in your markdown files and tells you which ones fail.

npx @ipseeta/docproof

docproof executes your documentation with your user's permissions in a temp directory. Run it on docs you trust (your own), and mark anything destructive as skipped. More on that below.

The Problem: Docs Rot Silently

Code has tests, CI, and reviewers. Documentation has none of that. When you rename a flag, bump a major version, or change a config format, nothing fails. The README just quietly becomes fiction, and you find out from a GitHub issue titled "Getting Started doesn't work".

I wanted the same feedback loop for docs that I have for code: run something, get a red or green answer, wire it into CI.

What docproof Does

Point it at a markdown file (or let it scan the repo), and it goes through four steps:

1. Extract all fenced bash, js, and ts code blocks, with their line numbers

2. Run each block in a throwaway temp workspace

3. Report pass/fail with stderr and exit codes

4. Suggest a fix for failing blocks using AI (optional, off by default)

Here it is running against a small demo README where the last step references a CLI that was renamed:

Four blocks pass, one fails with command not found, and the process exits non-zero. That last part matters: a failing doc can now fail a build.

How It Works Under the Hood

The interesting engineering decisions were smaller than I expected. A few worth sharing.

1. Blocks in the same file share a workspace

Documentation is sequential. A "create a config" block is usually followed by a "use the config" block. So docproof creates one temp directory per markdown file and runs the blocks in order inside it:

const workdir = await mkdtemp(join(tmpdir(), "docproof-"));

for (const block of blocks) {
  const res = await runBlock(block, workdir, opts);
  // ...classify pass/fail...
}

await rm(workdir, { recursive: true, force: true });

This means a README where one block creates a config:

echo '{"port": 3000}' > config.json

and a later block reads it:

const config = require("./config.json");
console.log("starting on port", config.port);

just works, because both run in the same directory, in order.

The key things here:

• The workspace is per file, not global, so two markdown files can't contaminate each other

• Everything is deleted afterwards, even if a block fails

• bash blocks run under bash -e, so the first failing command fails the block instead of errors scrolling by silently

2. Not every block should run

Real docs are full of blocks that are illustrative by design: install commands, snippets that need credentials, destructive examples. Failing on those would make the tool useless, so docproof supports two skip directives.

A token on the fence:

```bash skip
npm install -g some-tool
```

Or a comment above it:

<!-- docproof:skip -->

```js
await client.chargeCreditCard();
```

Skipped blocks show up in the report as skipped, never as failed. docproof's own README is checked by docproof, and this is what that looks like:

Languages docproof can't execute (json, text, output samples) are counted as "not runnable" and don't affect the result either.

3. AI suggestions, but only where they add value

I wrote before about adding AI features without overengineering them, and I followed my own rule here: use AI only where it adds value. Extraction, execution, and reporting are plain deterministic code. The one place AI genuinely helps is explaining why a block failed and proposing a corrected version.

With --suggest, each failing block is sent to Claude along with the error output:

const response = await client.beta.messages.create({
  model: "claude-opus-5",
  max_tokens: 16000,
  system:
    "You fix broken code samples found in project documentation. " +
    "Reply with the corrected code block only, followed by at most " +
    "two sentences explaining what was wrong. If the sample cannot " +
    "run standalone by design, say so and suggest a skip directive.",
  messages: [{ role: "user", content: failureContext }],
});

The prompt does two jobs. It asks for a corrected block, but it also gives the model an exit: if the sample needs credentials or a running service, the right answer is "mark this as skip", not a rewritten block. Without that instruction, the model would confidently "fix" things that were never meant to run standalone.

This is entirely optional. No API key, no AI. The core tool works offline.

4. Exit codes are the API

The report is colored and human-friendly, but the contract is the exit code: 0 when everything passes, 1 when anything fails. Add --json and you get a machine-readable report. That's all you need to put docs in CI:

docproof README.md docs/*.md || exit 1

No plugin system, no config file. I've come to appreciate tools that do this.

What I Learned Building It

• Markdown is messier than it looks. Fences can be backticks or tildes, three or more characters, indented inside lists, and nested inside four-backtick fences. The extractor tests were the first thing I wrote, and I'm glad I did.

• Dogfooding keeps you honest. docproof's README is checked by docproof, which forced the design question early: what should happen when the doc being checked contains its own npm install -g command? Without a skip mechanism, the tool would install software while checking the docs that describe it. That's why the skip directives exist.

• Sequential beats clever. My first instinct was to run blocks in parallel. But docs are tutorials, and tutorials have order. Sequential execution with a shared workspace matched how people actually write documentation.

Try It

npx @ipseeta/docproof

The tool scans your markdown, runs what it can, and tells you the truth about your docs. There's more coming: a GitHub Action so doc checks run on every PR, and a --fix mode that applies AI suggestions in place.

If you run it on your README and it finds something broken, I'd honestly love to hear about it. That first red ✗ on my own docs is what convinced me this was worth building.

Final Thoughts

We treat documentation as content, but for a developer tool, documentation is code that hasn't been executed yet. Running it is the only honest test.

docproof lives at docproof.ipseeta.dev and on npm as @ipseeta/docproof. It's a small tool, but it has already made my own READMEs more trustworthy. Maybe it will catch something in yours too.