LLM-as-judge has become the dominant pattern for evaluating language model outputs. Tools like Promptfoo, Braintrust, LangSmith all converge on the same architecture: send your prompt to your model, send the output to a different model with a rubric, take the second model's score as the quality signal.
This works. It's also expensive (judge tokens cost real money), slow (extra API roundtrip), variance-prone (the same eval gets different scores across runs), and architecturally a bit circular (using an LLM to evaluate an LLM trained on overlapping data distributions). The single signal becomes a bottleneck for trust.
So I built an eval module that has two independent signals instead of one.
What the tool does
Side-by-side blind comparison. Two agents answer the same prompt. One runs raw, the other can optionally have a cognitive harness wired in as a tool call. A separate blind judge model scores both responses, sees only A and B labels with no knowledge of which is which. Standard setup so far.
But alongside the judge, four cognitive posture heat maps run on each response. These are not LLM-based. Deterministic text analysis that visualizes HOW the model wrote, not just whether it agreed with you.
When the heat maps agree with the judge's verdict, you have confidence. When they disagree, you have a question worth investigating. Two independent signals beat one signal that wraps itself.
How the heat maps work
Each response is split into 100 word-chunks arranged on a 10x10 grid. Two grids per agent.
Top grid: confidence posture. Per chunk, count hedge words (maybe, might, possibly, seems, could) and assertive words (definitely, must, always, never, clearly). Compute net (asserts - hedges) / (asserts + hedges). Add punctuation cadence as a secondary signal: periods are positive (definite statements end with them), commas are negative (qualifications stack with them). Normalize to [-1, 1]. Color the chunk diverging from blue (hedged) through gray (neutral) to red (assertive).
Bottom grid: reasoning density. Per chunk, count explicit reasoning connectives (because, therefore, since, if/then, due to, as a result, this means). The denser the reasoning markers, the brighter the cell. Sequential palette from dark to hot.
A 2D Gaussian blur runs over both grids so sparse markers spread into spatial blobs instead of isolated cells. Empirically this matters: a single "because" in a 100-chunk response forms a small heat radius on the reasoning grid even when neighboring chunks have nothing. The blob shapes are easier to scan at a glance than scattered pixels.
The whole computation runs client-side in plain JavaScript. No API call, no model inference. Pure word counting plus a smoothing pass. Free to compute, deterministic, fast.
Try it yourself
bash
git clone https://github.com/ejentum/agent-teams.git
cd agent-teams/agent_evaluation_module_xp95
python serve.py
SOCIAL SHARE CARD GENERATOR