Scarab Diagnostic Field Test #033 - Prometheus Remote-Write Label Order Boundary

Target:

Pull request:

This field test was about a small but important remote-write boundary in Prometheus:

labels are required to be sorted lexicographically, but incoming remote-write labels could pass through a conversion path before that requirement was enforced.

That sounds narrow because it is narrow.

But this is exactly the kind of case Scarab is meant to make visible.

Not "the remote-write subsystem is broken."

Not "rewrite ingestion."

Not "let an agent go hunting."

The question was smaller:

where does repo truth say invalid input should stop, and where was the actual boundary allowing that truth to blur?

Field Lab record

The public Field Lab record for this case is here:

The changed files are public:

• storage/remote/write_handler.go


• storage/remote/write_handler_test.go

The patch does not redesign remote-write ingestion.

It does not change remote-read.

It does not introduce a new validation framework.

It keeps the repair at the specific boundary identified by the field test.

Why this was not a broad rewrite

This is one of the recurring lessons from Scarab field testing:

when the boundary is clear, the repair should get smaller, not larger.

A code agent can produce a large patch very quickly.

That is not the hard part anymore.

The hard part is deciding what the patch is allowed to claim.

For this case, the claim is not:

"remote-write validation is now perfect."

The claim is:

"incoming remote-write labels that are out of lexicographic order are checked before they can be normalized into the internal label representation."

That is a much narrower statement.

It is also a more reviewable one.

Why the diagnostic result mattered

The practical bug is about label ordering.

The larger field-test value is about method.

Modern code agents are very good at moving through a repository and producing edits. That can be useful, but it can also make patches feel suspicious when the intent is unclear.

Scarab's position is different:

first find the truth boundary, then patch only that boundary.

In this case, the repo truth was that labels must be sorted.

The boundary failure was that remote-write input could cross into a representation where the original invalid ordering was no longer visible.

The patch follows from that.

Not because the agent guessed a fix.

Because the public issue, the code surface, and the validation boundary pointed to a narrow repair.

That is the shift I am testing in public:

code-agent work should not begin with "what can we generate?"

It should begin with "where did the repo stop enforcing what it already says is true?"

Validation

The repair was validated in a Linux arm64 container with the Prometheus test workflow used for the public PR.

Validation recorded in the Field Lab:

• 
  make test passed


• 
  make lint passed


• 
  go test ./storage/remote -count=1 passed

At the time this draft was prepared on June 19, 2026, the public pull request was open and ready for review.

The public status is:

• PR open


• DCO passing


• Netlify deploy-preview successful or informational


• upstream review required

That status matters too.

This field report does not claim upstream acceptance.

It claims a public diagnostic record, a narrow repair, and a submitted PR.

Maintainers decide whether the patch belongs upstream.

Field test result

Result:

diagnostic proof and repair submitted.

The field test produced:

• a public issue-to-boundary record


• a narrow repair patch


• regression coverage for the repaired behavior


• a validation summary


• a public upstream PR

The patch is intentionally boring.

That is the point.

The interesting thing is not that a code change exists.

The interesting thing is that the repair boundary is explainable without exposing proprietary diagnostic logic or asking maintainers to trust a black box.

Public claim

This field test supports a narrow public claim:

SDS identified a remote-write label-order boundary in Prometheus where invalid input needed to be checked before conversion could normalize it, and a human-submitted repair was prepared for that boundary.

It does not claim:

• that Prometheus accepted the patch


• that all remote-write validation issues are resolved


• that Scarab repairs projects by itself


• that SDS source or product details are public


• that maintainers endorsed Scarab or the Field Lab

The Field Lab exists to keep those claims separate.

Disclosure

This field report was prepared with AI-assisted editing from public field-test notes, public issue and PR records, and the public Field Lab record. The diagnostic claim, repair boundary, and final wording were human reviewed.

Scarab Diagnostic Suite is proprietary. The Field Lab publishes public case records, issue links, validation summaries, and claim boundaries only.

SDS finds evidence. People make claims. Maintainers decide.