Measure Documentation Coverage for AI Agents With This Scorecard

The argument that AI makes documentation obsolete confuses two different activities: producing prose and preserving operational knowledge.

An agent can generate prose cheaply. It cannot recover a missing deployment condition, a forgotten modifier-click behavior, or a test nobody recorded. When evidence is absent, fluent text can make the gap harder to see.

Ben Halpern's current DEV discussion frames this as the myth of a “post-documentation era.” I want to make the product decision measurable: does the work item contain enough documented evidence for an agent to act and for a human to review it?

Five coverage dimensions

Use a 100-point scorecard:

Each dimension is documented (100% of its weight), partial (50%), or missing (0%). This is not a universal quality score. It is a coverage alarm for a declared unit of work.

Apply it to one bounded MonkeyCode case

I applied the scorecard only to MonkeyCode issue #824, pull request #859, and the relevant public code at commit c58bcd4. This is not a score for the project or its documentation as a whole.

The issue identifies a /workspace/... Markdown link returning the app home page. The PR distinguishes a normal click, which should open task file preview, from new-tab and copy-link actions, which should preserve a file-manager deep link. It also reports lint, online build, and manual Markdown-link checks.

My evidence file rates that narrow case:

{

  "name": "reproduction",

  "weight": 20,

  "status": "partial",

  "evidence": "Path and click are described; browser and deployment are absent"

}

The result is 80/100, with reproduction and limitations marked partial. The number does not declare the fix good or bad. It tells a product team where another question or test record would reduce agent guesswork.

Make scoring auditable

The companion zero-dependency script validates that weights total 100, statuses use the defined scale, every dimension cites evidence, and the revision is pinned.

node score-docs.mjs doc-coverage.json

node test-score.mjs

Expected output:

coverage=80/100 gaps=reproduction,limitations

PASS score=80; rejected invalid weights and missing evidence

The test deliberately breaks the weights and removes evidence. A spreadsheet can calculate the same number, but validation prevents a polished dashboard from hiding an invalid rubric.

Use the score as a workflow gate

Different work deserves different thresholds:

• exploration: allow lower coverage, but display gaps to the agent;


• routine implementation: require problem, expected behavior, and verification;


• security or irreversible changes: require every dimension and human approval;


• incident response: ship the mitigation when necessary, then make missing evidence explicit debt with an owner.

The useful product behavior is not “block anything below 90.” It is to change what the agent may do. Low coverage can permit investigation and test creation while forbidding an automatic merge.

Measure whether the scorecard helps

Run it as a product experiment, not a doctrine. Track:

1. clarification rounds before implementation;


2. reviewer-found scope mismatches;


3. reopened issues caused by missing secondary behavior;


4. time spent documenting versus time saved in rework;


5. disagreement between scorers.

If teams game the status labels, require evidence URLs and sample scored packets in calibration sessions. If the score does not predict rework or review quality, change the dimensions.

The broader point is simple: generated documentation can lower the cost of formatting knowledge, but it does not create missing observations. A coverage scorecard keeps that distinction visible to the agent, the reviewer, and the product manager deciding how much autonomy to grant.

Disclosure: I contribute to the MonkeyCode project. The score above is a limited analysis of the linked issue, PR, and pinned code—not a project-wide documentation rating. The scoring script and fixtures were tested locally.