Building FailureDNA: an agent memory that knows when not to trust itself

The validity gate is deliberately boring and deterministic:

No model decides whether a memory is trustworthy. And critically, avoid is enforced, not advised: an action with a symptom-matching prior failure is removed from the candidate list before the model sees it. The agent cannot repeat a known failure even if it wanted to — which matters, because a live LLM handed the same memories as a "hint" will sometimes ignore the hint. The creative part (which action, given the evidence) goes to Qwen; the part that must never hallucinate (is this memory valid? did this action succeed?) stays in deterministic code.

Why Qwen Cloud fit

I used Qwen Cloud through its OpenAI-compatible DashScope endpoint, which made two things nearly free:

1. 
   Embeddings for recall. text-embedding-v3 turns incident symptoms into 1024-d vectors for pgvector search. Hybrid retrieval fuses semantic similarity (weight 0.70) with keyword overlap (0.30), so it catches both paraphrased and exact-token symptoms.


2. 
   Validated action selection. A Qwen chat model receives the symptoms, the environment fingerprint, the (gated) candidate actions, and the memories, and returns a single JSON decision at temperature=0 with thinking disabled — fast, deterministic-ish output that I validate before anything executes.

Because it's OpenAI-compatible, the whole client is a thin, well-typed wrapper with explicit timeouts and one retry — no exotic SDK to fight.

Proving it helps (without fooling myself)

A demo where the new thing wins is easy to fake, so FailureDNA ships a benchmark designed to be hard on itself: three modes (no_memory, naive, failuredna) on identical seeded history, hidden simulator outcomes, evaluator-only safe/unsafe labels, isolated memory per mode, and static shortcut baselines (always_inspect_downstream, …) to check it isn't just rediscovering that one action is usually right.

FailureDNA lifts first-action resolution well above the naive agent, cuts unsafe first actions sharply, resolves in fewer actions, and repeats zero historical failures and zero stale successes. The honest caveat I left in the open: in this small scenario set, a static always-inspect policy also scores well — which is exactly why the shortcut audit exists. FailureDNA's value isn't a magic action; it's that it never repeats a known failure and never blindly reuses a stale fix as environments change — the behavior that generalizes beyond a fixed benchmark.

Running it on Alibaba Cloud (and the sharp edges)

The backend runs as a custom container on Alibaba Cloud Function Compute (FastAPI, port 9000), memory persists in ApsaraDB RDS for PostgreSQL + pgvector (HNSW), and the image lives in ACR Personal Edition. A few things bit, and are worth writing down for the next person:

• 
  ACR tiers matter. Enterprise Economy can't back an FC custom-container function (no image processing). Personal Edition (free) works.


• 
  The default *.fcapp.run domain forces downloads. It adds Content-Disposition: attachment to HTML and JSON responses, so a browser downloads your dashboard or health JSON instead of rendering it. I serve the UI from GitHub Pages and added a small ·
  
  
  

  Built with Qwen Cloud + Alibaba Cloud Function Compute and RDS pgvector.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen