I ran an experiment with the same project through two AI LLM model scenarios — once with a standard prompt, once with spec driven workflow. The results weren't what I expected.
The headline isn't about tokens or the best performing LLM model. It's about measuring what the agents thought they delivered versus what they actually delivered.
Repo: — an open source spec-driven development tool that externalises behaviour into structured markdown specs and generates pytest scaffolding, with traceable links between them.
The idea is simple: define what correct looks like before the agent starts coding, then verify against it. The workflow looks like BDD, but quacks like TDD.
There are many spec driven dev tools out there (sorry, yes this is another one) - but they are generally for the AI assisted dev workflows, so still need human dev to drive. SpecLeft tries a different approach – it is agent-native, meaning it's optimised towards AI agent adoption with an agent contract to verify safety. This adds trust to building software without too much intervention or technical review.
To summarise the goal — can we trust AI agents to develop software that actually behaves as it should, while keeping the code readable, maintainable while fulfilling the intent?
The Experiment
The application: A document approval workflow API — documents move through draft → review → approved/rejected, with multi-reviewer approval, time-bound delegation, automatic escalation, and a handful of edge cases.
We don't want a basic CRUD system for a nice vibe-coding showcase. This system scope has state machine, concurrent decision handling, time-based logic, and business rules that interact with each other. Complex enough that an agent can't just wing it.
The only difference was whether SpecLeft was involved.
💻 Repos and Session playbacks have been attached to each test run.
🎥 Session has to be downloaded and played with
Code:
Bugs Discovered Post-Analysis:
Missing Auto-Escalation Feature: Despite PRD requiring automatic escalation after timeouts, only manual escalation is implemented. The
check_and_escalatefunction exists but performs no escalation, violating core business requirements.Potential Timezone Brittleness: Delegation expiry checks assume naive datetimes are UTC, which could fail if assumptions are incorrect.
Concurrency Risks: No explicit locking for concurrent reviewer decisions, potentially leading to race conditions.
GPT Codex 5.2 — Baseline
Codex moved faster and more aggressively. Implementation came out in parallel batches — models, schemas, routes, services written simultaneously. But it backtracked more. Tests failed 4 times before going green. Server failed to start on the first attempt. Behaviour verification required 4 patches to services.py.
Total time: ~18 minutes. Total tokens: 53,000.
The retrospective was vague: logic gaps were "caught early," timezone handling was a known issue. No specific bugs named.
Session Playback (asciinema cast): . But this time SpecLeft is installed as a dependency, and the prompt tells the agent to externalise behaviour before writing code:
You are an autonomous agent guided by a planning-first workflow.
Build a document approval API using FastAPI and SQLAlchemy.
The project has had the initial setup already.
Follow ../prd.md for product requirements.
Follow ../SKILLS.md for instructions.
Initialize SpecLeft and use its commands to externalize behaviour before implementation.
I have installed v0.2.2.
Only if required, use doc: https://github.com/SpecLeft/specleft/blob/main/AI_AGENTS.md for more context.
Do not write implementation code until behaviour is explicit.
Go with your own recommendations for system behaviour instead of verifying with me.
Then I walked away again.
Note: The AI_AGENTS.md is to help the agent know how to use SpecLeft tool better.
Claude Opus 4.6 — With SpecLeft
Opus externalised all 5 features into SpecLeft specs before writing a line of implementation code. It updated scenario priorities to match feature priorities — a decision it made on its own. Then it generated test skeletons with specleft test skeleton, giving it 27 decorated test stubs mapped directly to scenarios.
Code:
GPT Codex 5.2 — With SpecLeft
This was the surprise. Codex consumed the SpecLeft specs and test skeletons, and then did something I didn't engineer: it wrote functional test logic before implementation code. Genuine TDD, driven by the structure of the skeletons. The scaffolding naturally guided the agent into writing assertions first, then building the code to satisfy them. Sweet!
Code :
SpecLeft Results
| Metric | Codex 5.2 | Opus 4.6 |
|---|---|---|
| Total tokens | 146,499 | ~147,000 |
| Total tests passed | 27 (100%) | 27 (100%) |
| Failed test runs | 2 | 1 |
| Issues found in retro | 0 | 3 |
| Time to completion | ~38m | 21m 1s |
| Tokens to externalise specs | 49,000 | 45,000 |
| Tokens before implementation | 89,000 | 63,000 |
Side-by-Side Comparison
Opus without specs generated 53 tests, nearly double the SpecLeft run's 27 — but quantity isn't coverage. The 53 tests were whatever the agent decided mattered, with no traceability to product requirements — which is shown with the missing auto-escalate requirement. The 27 SpecLeft tests each map to a specific scenario in the PRD
| Metric | Codex Baseline | Codex + SpecLeft | Opus Baseline | Opus + SpecLeft |
|---|---|---|---|---|
| Total tokens | 53,000 | 146,000 | 83,243 | ~147,000 |
| Total tests passed | 19 | 27 | 53 | 27 |
| Failed test runs | 4 | 2 | 2 | 1 |
| Bugs found during retro | 0 | 0 | 0 | 3 |
| Missing Requirements | 0 | 0 | 1 | 0 |
Missing Requirements: count of unimplemented PRD features.
Which Stack Stayed on Track the Best?
Having a look at the code and testing the API manually - both spec driven runs are strong so it's pretty even. Codex had a much cleaner data model and modern sqlalchemy implementation; while Opus was more flat in its design. With that in mind - I'd feel better about picking up the Codex SpecLeft project in a realistic situation. That being said the code wasn't mind blowing either - especially the lack of exception handling around database queries in the service layer.
I've also prompted a few neutral agents (Gemini-3, Kimi K2.5, Grok) to evaluate the codebases on quality, maintainability, and correctness.
Full analysis found in the
Docs: if you utilise Python and AI agents in your dev workflow and want to be involved.
Drop a comment — I'm keen to hear your thoughts.
SOCIAL SHARE CARD GENERATOR