The Stack Under the Model: oMLX, llama.cpp, Hermes, and Why There Are So Many

It doesn't reinvent Retort — it composes the same design generator, ANOVA, and

Pareto engine — but it adds three new factors on top of the usual language/model:

harness_config — the orchestration strategy (the headline factor)

scaffold — the reasoning structure wrapped around each attempt

none · plan-and-solve (plan first, then execute) · reflexion (attempt,

self-critique, retry).

model — the raw model, spanning cheap→frontier

deepseek-v4-pro · glm-5.2 · opus-4.8 · gpt-5.2, reached through

OpenRouter (a unified API that routes one request format to many providers),

chosen to spread from cheap to frontier so the analysis can separate a model

effect from a harness effect.

What it would actually test

Because a fractional-factorial design crosses all of these at once (rather than

changing one thing at a time), the ANOVA can attribute the variance in a metric to

model vs harness vs scaffold vs language + their interactions, and report which

effects are cleanly estimated versus confounded. Concretely, it answers questions

the model-only grid structurally can't:

• Of the lift from +agenticow-memory, how much is the memory branching versus
  just the underlying model being good?


• Does routed actually hold reliability while cutting cost — i.e., does it sit
  on the accuracy-vs-\$ Pareto front?


• Is self-consistency-5 worth 5× the tokens, or a rounding error on a task the
  model already nails?


• 
  Is a reasoning scaffold a real lever or a ritual — and does that depend on
  model strength? (Retort already found the sibling result that the prompt is a
  lever only in proportion to how weak the model is; the metaharness generalizes
  that from prompt to full orchestration.)

The honest caveats: it's cloud-only (OpenRouter, metered — the

self-consistency × frontier × replicates corner gets expensive) and the real

orchestration logic lives in an external solver the adapter shells out to, so

without that solver only a \$0 stub runs. That's why it's a documented

side-branch rather than a headline result — it stays one until a first screening

run shows the harness variance is real enough to promote. The staged plan lives in

docs/future-experiments.md.

Takeaway

There are "so many" harnesses and engines because at least six independent forces

— hardware, format, new architectures, ease-vs-control, vendor-vs-open, and

cost-vs-privacy — each keep their own alternatives alive, and no layer has a

winner that retires the others. The practical consequence is the thesis of this

whole repo: don't benchmark the model, benchmark the stack. The model is one

band in a tall column, and the quiet layers under and around it — the serving

engine, the context threshold, the sampling defaults, and the orchestration

strategy — routinely decide the result. The metaharness is Retort turning the last

of those, the orchestration layer, from an assumption into a measurement.