When the Free Executor Cost More: 40 Trials on Opus + Local Qwen Ended Up the Most Expensive Cloud Arm

/ repository at commit b210c0e (v0.26.8, MIT license). Each trial starts with git checkout -- . && git clean -fd to restore the base state.

• 
  T1 — Breakage recovery: 25 errors injected via AST (10 mypy + 10 ruff + 5 pytest collection failures). The agent has to return the harness to fully green.


• 
  T2 — Refactor: Move get_params_from_function from typer/utils.py to a new module typer/_param_extractor.py. Update every import site. All tests still passing.


• 
  T3 — Feature-add: Implement get_version_banner(prefix, uppercase) -> str, re-export from typer/__init__.py, and pass a SHA-256-fingerprinted test file.

The judge

mypy + ruff check + pytest — exit code 0 = success, anything else = failure. Per-task verifiers (verify-T2.sh / verify-T3.sh) add structural checks (function actually moved to the new module, fingerprinted test unmodified, etc.).

No LLM is ever asked "is this OK?". The judgment is deterministic and reproducible.

Results (success-only medians, n=3 per cell)

Bold = per-column winner. Total Anthropic spend across 40 trials: $35.98 — cheap for a paper.

The row worth staring at is arm B. On all three tasks, its cost ($) is the cloud-arm worst ($2.27 / $1.38 / $0.42). Qwen's tokens cost zero. Opus + Qwen is more expensive than Opus alone anyway.

Why T3 Blew Up to 5.3×

The most extreme case is T3, the smallest task — about 6 iterations.

Same mechanism, different ratio. The base context (system + tools + initial prompt) is cache_write-ed once on the first turn and cache_read cheaply thereafter. On long tasks (T1, T2), that base is a small fraction of the cumulative input. On a short task, it's a big fraction. So "base re-read every turn + executor summary re-read every turn" overhead dominates everything else, and T3's B/A ratio spikes to 5.3×.

Conversely, arm C (Opus + Haiku) has a smaller cache_read footprint than arm A on T1 and T2 (0.79× and 0.70× of A). Haiku does substantive work that Opus would have otherwise had to do itself, and the substance translates into useful summaries instead of dead weight. Which is the opposite end of the Qwen-summary-bloat story.

When Orchestration Would Win (cases I deliberately excluded)

The "strong orchestrator + cheap executor" recipe falters in iterative tool-loops because, over dozens of turns, the orchestrator's cache_read becomes the dominant cost line. One-shot routing has no such problem.

The experiment was, in that sense, designed against arm B:

• 
  Executor returns are free-form (Qwen stdout summary up to 4000 chars). If you constrain returns to "one structured diff and nothing else," the orchestrator's accumulated context shrinks.


• 
  Tasks are sequential (T1/T2/T3 cannot be parallelized within a single trial). Tasks where the orchestrator can dispatch "go edit these three places at once" might pay for orchestration overhead.

Re-running arm B with tightly-bounded executor returns is the next experiment on my list. I expect T3 to invert. T1 is harder to call.

Practical Takeaways

Sitting with these numbers, here is how my own agentic coding setup changed:

1. 
   For tasks that finish in a handful of iterations, Opus solo is the cheapest cloud option. T3: $0.17, 69 seconds, 6 iterations — cloud-best. "Opus is expensive" is a one-shot framing. Across an iterative loop, Opus's per-iteration efficiency pays.


2. 
   For tasks that need dozens of iterations, the model with the lowest per-iteration cost wins on dollars. T1: Haiku solo at $0.30 is 5.5× cheaper than the cheapest cloud arm. But it fails 25% of the time, so retry-adjusted expected cost narrows the gap to 4.2×.


3. 
   For cloud-only setups, Opus + Haiku is the most balanced. Ties Opus solo on T1, wins T2 on cost, narrowly loses to Opus solo on T3. The safe pick if you don't want Haiku-solo's failure rate.


4. 
   If you're going to use a local Qwen "for free," constrain the executor return size structurally. Free-form stdout returns just shift the cost to the orchestrator's cache_read line.

"Strong + cheap" composition has a narrower design surface than it seems. Unless you also specify what and how much the executor is allowed to return, you regenerate the "orchestrator-becomes-expensive" pattern. I regenerated it three times suspecting measurement error before finally accepting it.

Limitations

The honesty section:

• 
  n=3 per cell. Mann-Whitney U p-values use a normal approximation where 0.050 is the small-sample floor — it means "as different as this sample size can show." Trust the Cliff's delta effect sizes; don't over-read p-value differences.


• 
  All three tasks are on the typer repo. Generalization needs other codebases. The harness, breakage injector, runner, and analysis are all MIT-licensed in the repo, so reproducing this on your own codebase is cheap.


• 
  The orchestrator system prompt is asymmetric. It instructs "do not edit directly, delegate instead." This mirrors a real deployment shape but is a real confounder in the results.

Reproducing

Tested on Ubuntu 22.04 with Python 3.10+, uv 0.4+, and anthropic Python SDK 0.83+. Arm B uses Ollama 0.4+ running qwen3.5:9b. If you skip arm B, Ollama is not needed.

git clone https://github.com/kenimo49/free-executor-paradox

cd free-executor-paradox

# run arm A on T3, one trial

python scripts/runners/runner.py --arm A --task T3 --trial 1

The repo README and paper PDF have the full reproducibility setup — harness, breakage injection, runner, and analysis scripts are all included.

Closing

The cost debate around agentic coding tends to fixate on what the executor costs per token. The dominant term is actually what the orchestrator re-reads, and how often. Qwen here is just one instance of the pattern — every "free local executor" that comes next will hit the same issue. Free executor tokens don't make orchestrator cache_read free.

I wrote it up as a paper because numbers are harder to argue with than vibes. The most satisfying outcome would be someone replying "got the same thing on my codebase" or "actually got the opposite, here's why."

• 
  Paper (Zenodo):
  


• 
  GitHub Release: v1.0.0