The demo videos always end at the same moment. A figma frame turns into a passing test in twelve minutes. Someone in the room says the word “productivity.” The recording stops.
The parts that come after that moment are the parts I actually get paged about. Who owns the ticket the agent opened at 3:14 a.m.? Which model call produced the assertion in test case 47? What closes the 17 draft tickets a stuck run left behind before the next sprint planning notices them? None of that shows up in the demo. All of it shows up on the on-call rotation. After 20 years of leading test automation across consumer-scale platforms, I have a strong bias about which slide in the deck predicts whether a pipeline ships or stalls. It is never the architecture slide. It is the runbook.
This piece is about the runbook. I built an unattended agentic test pipeline over the pattern is the classical treatment — but the LLM literature mostly still treats it as a prompt problem. It isn’t. You cannot fix it with a better system prompt. You fix it with a typed handoff contract that both agents agree to before they run, plus a validator between them that fails loud when the shape drifts.
The heuristic I now use: Every agent-to-agent boundary gets a schema and a validator. Every agent gets a golden-input regression suite that catches contract drift before it reaches a downstream agent. Neither of those is glamorous. Both are the difference between a pipeline that produces work and a pipeline that produces a very expensive game of telephone.
If you want to know whether a team’s agentic pipeline is going to survive the first quarter, ask them what the contract is between agent one and agent two. If the answer is “the model figures it out,” budget for the cleanup.
Provenance, or the audit trail nobody wrote
The second thing I insist on now is that every artifact the pipeline produces has to answer three questions without a human doing archaeology: which agent produced it, which model call produced it and which upstream inputs the agent was looking at when it did.
This sounds like a nice-to-have. It is not.
Somewhere around week three of running the pipeline I hit a subtle case: the requirements agent was quoting a Confluence page back to itself. The MCP server for Confluence had returned an empty result on a first call, the agent had written a placeholder requirement, the second call had succeeded, the agent had retrieved its own placeholder and by the third pass it was citing that placeholder as source truth. The ticket said, entirely in earnest, that the requirement came from the design owner. It hadn’t come from anywhere. It had come from itself, four minutes ago.
You cannot debug that class of failure with logs of what the model said. You need logs of what the model was looking at. That means capturing the tool-call ID for every MCP call, stamping every artifact with the set of tool-call IDs it derived from and refusing to accept any retrieved fact into a downstream stage that cannot be traced to a real external source. I call it the prove-the-source rule. It’s boring. It’s a two-line requirement in the runbook. It is also the single guardrail that has saved me the most on-call time.
There is a nice side effect. When editors, reviewers or auditors ask where a decision came from, the pipeline can show them. That matters if you work anywhere near a regulated stack, and increasingly it matters everywhere else — the
SOCIAL SHARE CARD GENERATOR