This is the second field test in my Scarab Diagnostic Suite series.
The first test was guided. A known Open WebUI provider/config issue was turned into a bounded repair lane.
This second test was more important.
This was the first field test where SDS moved closer to the thing I actually wanted to prove:
Can Scarab Diagnostic Suite identify the right failure surface before being told the exact bug?
The target was Open WebUI issue #25038.
The user-facing symptom looked like a web search problem.
Open WebUI successfully queried SearXNG.
The UI showed valid search results.
The retrieved snippets already contained useful information.
Then the app entered the “Querying” phase.
Then it showed “No sources found.”
Then the model answered as if live web search was unavailable.
That is a deceptive class of failure.
Search worked.
Retrieval worked.
The UI knew results existed.
But the retrieved information did not survive into usable model context.
The system appeared to succeed and fail at the same time.
The field-test setup
I ran SDS against a fresh Open WebUI v0.9.5 target.
The important constraint:
SDS did not receive the GitHub issue text.
No custom diagnostic script was aimed at the known failure.
No hand-written detector was inserted for this issue.
The goal was to see whether SDS could identify a suspicious subsystem without being handed the answer.
That matters because it is easy to make an AI system look impressive if it is fed the bug.
I wanted to know whether SDS could surface the correct class of instability before the repair agent started changing code.
What SDS found
SDS selected the Memory/RAG surface and identified a retrieval truth boundary problem.
The findings were under:
text retrieval_truth_boundary
with the contract class:
text source_context_contract_consistency
The flagged paths involved variables like:
text res results queries_response
This was the first important signal.
SDS did not say:
“This is Open WebUI issue #25038.”
That would have been suspicious.
Instead, it said something more useful:
“This retrieval/source-context boundary is noisy. Response and result shapes crossing this boundary are not visibly stable.”
That matched the real failure class.
The visible issue was “No sources found.”
The deeper issue was that retrieved truth was not safely surviving the trip into model context.
That is the kind of boundary Scarab Diagnostic Suite is meant to detect.
Why I call it a retrieval truth boundary
In AI applications, retrieval is not enough.
A search provider can return good results.
The UI can show those results.
A source-processing path can begin.
A document loader or vector pipeline can start.
But if the final model context does not receive the retrieved truth in a usable shape, the system still fails.
That is what made this bug interesting.
The user saw:
text Searching the web Searched 5 sites valid URLs valid snippets Querying No sources found
Then the model responded as if it never had web access.
That is not simply “web search is broken.”
That is:
retrieved truth existed,
but the boundary that should carry it into model context collapsed.
That is a retrieval truth boundary failure.
The bounded repair
After SDS selected the hotspot, Codex performed one bounded stabilization pass.
SDS did not become the repair tool.
SDS identified the diagnostic boundary.
Codex performed the repair.
Then SDS reran to verify whether the targeted surface quieted.
The initial SDS finding count for the Memory/RAG retrieval boundary was 3.
After the bounded repair, the targeted findings dropped to 0.
The selected surface moved from noisy/contradictory to quiet/organized.
That proved one internal thing:
The repair addressed the exact surface SDS had selected.
But I still needed functional proof.
A diagnostic finding clearing is useful, but the actual bug still has to go red to green.
Functional proof
Only after SDS selected the hotspot did I use the issue’s concrete failure shape as functional proof.
The repro reproduced:
text TypeError: 'JSONResponse' object is not subscriptable
The failure happened because chat_web_search_handler could receive a JSONResponse object but then consume it as if it were an OpenAI-style completion dictionary with a choices payload.
That is a response-shape contract failure.
The repair added a response-shape boundary.
Instead of directly subscripting the response, the code checks whether the generated-query response has the expected completion shape.
If not, the web-search path falls back safely to the original user query instead of crashing during source/context processing.
The focused regression passed after repair.
SDS reran afterward and the retrieval truth boundary remained quiet.
That gave the full loop:
SDS detected the surface.
Stepwise selected the hotspot.
Codex made a bounded repair.
The real failure went red to green.
SDS verified the selected boundary cleared.
What made this different from Field Test #001
Field Test #001 was useful, but guided.
The issue was known, and SDS helped turn it into a constrained repair lane.
Field Test #002 was stronger because SDS found the relevant subsystem before being given the issue text.
It did not know the GitHub issue.
It did not magically know the whole bug.
But it identified the right class of instability:
retrieval/source-context response-shape inconsistency.
That is the difference between a tool that merely follows instructions and a diagnostic suite that can surface a real failure boundary.
Internal classification
Field Test #002
Project: Open WebUI
Issue: #25038
Surface: Memory/RAG retrieval truth boundary
Contract class: source/context response-shape consistency
Mode: blind-ish diagnostic surface detection followed by functional proof
Result: local repair successful; targeted SDS findings cleared; focused regression passed
Primary lesson: SDS can identify the correct failure surface before issue text is used as functional proof
Why this matters
A lot of AI application bugs live between systems.
Search works, but context injection fails.
A tool returns data, but the model never receives it.
A provider response exists, but the handler expects a different shape.
The UI says one thing, the runtime does another.
These are truth-boundary failures.
Scarab Diagnostic Suite is being built to find those boundaries and keep AI coding agents from making broad, noisy repairs.
This field test showed that pattern working.
Not perfectly.
Not magically.
But usefully.
The suite found the unstable surface.
The agent repaired one slice.
The test proved the real failure.
The diagnostic rerun confirmed the boundary quieted.
That is the repair loop I want:
not bigger AI,
not more autonomous wandering,
but bounded diagnostic repair.
A system that can look at a noisy repo and say:
The problem is probably here.
This is the boundary.
Repair only this lane.
Now prove it.
SOCIAL SHARE CARD GENERATOR