BenchmarkDotNet Measures Performance—But Who Enforces the Budget?

You add [MemoryDiagnoser] and [Benchmark] to a class, run dotnet run -c

Release, and BenchmarkDotNet hands you back a beautifully formatted table:

mean, median, standard deviation, allocated bytes per operation. It's one of

the best measurement tools in any ecosystem.

And then what?

You read the numbers. You compare them, in your head, to what you remember

from last week. Maybe you screenshot the table and paste it into a Slack

message. Maybe you don't, because nothing regressed today, so there's

nothing to say.

That's the gap. BenchmarkDotNet is a measurement tool. It has no opinion

about whether 4.5ms is acceptable or a regression. It doesn't remember what

the number was yesterday. It doesn't fail your build. It doesn't stop a pull

request. Every one of those decisions is left as an exercise for the reader

— which in practice means it doesn't happen, until someone notices

production got slower and goes spelunking through git blame.

A budget is not a measurement

The distinction I kept running into while working on

CedarRecon, a reconciliation

engine where I'm doing a lot of comparative benchmarking (dictionary-based

vs. indexed classification strategies, that kind of thing): a measurement

tells you what happened. A budget is a policy — a committed, reviewable

statement that says "this operation must not take more than X, and here's

what we do if it does."

Those are different artifacts with different lifecycles. A measurement is

disposable — you take it, you look at it, you move on. A budget needs to be:

• 
  Committed — visible in a pull request diff, not a number someone
  remembers


• 
  Reviewable — a change to the budget is itself a decision someone signs
  off on, the same way you'd review a change to a config file


• 
  Enforced — a CI failure, not a Slack message someone might read

BenchmarkDotNet gives you excellent measurements. It was never trying to be

the second thing.

Why not just eyeball it in CI logs?

The honest answer is: you can, for a while, on a small project with one or

two people watching closely. It stops working the moment either of those

conditions changes.

I found this out concretely, not hypothetically. While dogfooding an early

build of a tool I've been writing specifically to close this gap (more on

that below), I ran a real benchmark suite from CedarRecon, deliberately

introduced a regression into one method to confirm the tool would catch it

— and it did, correctly, twice. But on the same run, an unrelated method I

hadn't touched showed a 28% regression at the largest input size.

If I'd been eyeballing a CI log, that's the kind of number that either gets

ignored ("probably just noise, I'm busy") or gets chased for an hour before

someone realizes the machine was probably running warm. What actually

happened: I reran just that benchmark in isolation, and it came back

clean — comfortably within a few percent of baseline, no regression at all.

It was noise. A shared CI runner, or a laptop that hadn't idled down between

runs, produced one bad sample.

That's not a knock on BenchmarkDotNet — its statistics are good. It's a

demonstration of why "look at the number and decide" doesn't scale as a

process, even when the underlying measurement tool is solid. You need

something that applies a consistent rule every time, and — just as

importantly — something that treats a single suspicious result as a

hypothesis to confirm, not a verdict to act on immediately.

What a performance contract actually needs

Turning "here are some numbers" into "here is an enforced budget" means a

few concrete things have to exist that BenchmarkDotNet doesn't try to

provide:

A committed baseline. Not a number in someone's head — a file in the

repo. JSON, reviewable, diffable. When a baseline changes, that's a

deliberate act someone did on purpose, visible in the PR.

A threshold policy. How much regression is acceptable before it's a

failure? 5%? 15%? Does a tiny absolute change (1ns → 1.2ns) count, even

though it's "20%"? These are decisions, and they should live in

configuration, not in someone's judgment call at 4pm on a Friday.

A pass/fail exit code. The single most useful thing a CI step can

produce. Not a report someone might read — a build that goes red.

No SaaS, no hosted database. This one's a deliberate design choice more

than a hard requirement, but it matters to me: baselines and policies as

plain repo-local JSON mean no external account, no network dependency to

evaluate a benchmark, and — critically — the same command produces the same

decision locally and in CI. Nothing hidden behind a dashboard only CI can

reach.

Where this is going

That's the shape of the tool I've been building —

Cedar.BenchmarkGate,

a local-first performance regression gate designed specifically for

BenchmarkDotNet output. It doesn't run your benchmarks. It doesn't replace

BenchmarkDotNet's statistics. It reads the JSON BenchmarkDotNet already

produces, compares it against a committed baseline under a policy you

control, and returns an exit code CI can act on — plus a Markdown summary

that explains why, not just that.

v0.1.0-alpha.1 is live now, built and dogfooded against CedarRecon's real

benchmark suite — including that 28% false alarm, caught and correctly

dismissed by rerunning rather than reacting.

Next up: the architecture — normalized benchmark identities, why baselines

and policies are deliberately separate documents, and the exit-code

contract that makes this usable as an actual CI gate and not just a fancier

report.