Hi, I'm I covered AI reviewing AI PRs -- the auto-review pipeline that defends quality at the PR stage.
This post is the other side: defending quality in production, via Self-Healing. A production alert fires, an AI investigates it, opens a fix PR, the PR goes through the same auto-review pipeline from Part 3, gets auto-merged and auto-redeployed. And the same fix PR is required to add a new Guide -- whether that's a lint rule, CI guard, type constraint, or guideline update -- so the same anti-pattern gets auto-rejected from then on. The guardrails grow every time.
"Incidents get fixed automatically" is catchy on its own, but on its own it's probably not enough in the long run. You have to close the recurrence class while you fix the incident -- self-healing plus self-strengthening -- before the quality gates start to compound over time.
Start with last month's numbers
115 Self-Healing PRs merged in the past 30 days.
Effectively all of them merged and deployed without human involvement.
Humans only step in when the AI judges "this is not something code can fix."
That's the current state of "incident response" at cortex.
About half (54) are Deploy Failed-style alerts -- CI / Pulumi deploy step caught a failure, the AI absorbed it before it shipped to production. Recently the [Recurrence] loop (covered later) has been piling up countermeasures here, so this bucket is trending down anecdotally
The remaining 61 are production-runtime alerts (Service Error Log Detected / Pipeline Failure / Generator Failure etc.) -- the service is running in production, but an error-log threshold or consecutive-failure threshold tripped. The AI absorbed them before they propagated to user impact
So it's less "incident response" than "production anomalies that monitoring caught, fixed 115 times by AI before anyone woke up." The number of incidents humans actually have to acknowledge is in the low single digits per month.
There's also a clear pattern of the same service firing repeatedly (e.g. gcs-transformer is 25 of the 61) -- which is exactly what the [Recurrence] loop covered later is supposed to eliminate by turning into lint or type gates. That's the back half of this post.
One more honest note: the recent month's number is slightly inflated. The codebase had a fair number of "silent catch" patterns -- catch blocks that swallow exceptions without logging anything. We added the no-silent-catch lint rule and swept the existing silent catches in batches, which exposed previously hidden production errors as alerts. So part of the spike is "monitoring caught up to reality." Once the [Recurrence] loop converts these into lint over time, the number should converge. "Things we couldn't see, we can see now" is a quality improvement -- what we're seeing is the catch-up phase.
One more thing worth saying: doing this by hand is utterly unsustainable. Running 115 manual cycles of "ack alert -> read logs -> context switch -> understand the code -> fix -> open PR -> review -> deploy" would bankrupt any team's engineering bandwidth. The system absorbs them without anyone noticing, and converts the fix into a new Guide (lint / CI guard / type constraint / guideline) at the same time -- that's the actual subject of this post.
The moment an alert fires, the AI starts an investigation, traces Loki / Product Graph / git blame to root cause, opens a fix PR, runs it through the auto-review from
Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same pattern auto-rejected from then on
This article ← you are here
5
Scaling the harness from cortex to toC services
Non-engineer contributions in practice + scaling cortex's harness to the whole product org
Coming soon
6
Series wrap-up
The underlying philosophy (what was given up, what was kept, why this design) plus a retrospective on the failures and lessons
Coming soon
Big picture -- the three layers: Observation, Repair, Strengthening
For Self-Healing to work, you need an Observation layer in front and a Strengthening layer (recurrence prevention) behind it. Self-Healing itself is the middle Repair layer. The "self-healing + self-strengthening" loop only spins up when all three are in place.
Prerequisites: The three layers only stand up on top of two prior pieces: cpg (the unified code / docs / DB / infra knowledge graph from
Layer
Role
Key components
Observation
Real-time detection of production anomalies
OTel SDK / Loki / Mimir / Tempo / Faro / Grafana / Pino logs with trace_id
Repair
AI receives the alert, investigates root cause, opens a fix PR, auto-review, auto-merge, auto-redeploy
The fix PR is required to add a new Guide (lint / CI guard / type constraint / guideline). The same anti-pattern can't reach production again
@cortex/eslint-plugin-graph (26 rules), scripts/check-*.ts (13 guards), , Observability is one of the "supporting foundations" beneath the flywheel.)
Repair -- the Self-Healing flow
MODE=self-healing runs the same webhook-server script as the auto-review setup from
The textual flow looks like:
CODE
[Grafana Alert Rule firing]
↓ POST /webhook/grafana
[Event Relay (in-house)] -- persisted in Firestore
↓ SSE push (event: grafana-alert)
[self-healing mode script]
↓ throttle check (same fingerprint skipped for 4h)
↓ 👀 reaction in Slack to signal "I'm on it"
↓ git worktree add -b hotfix/auto-alert-{service}-{ts} origin/main
↓ run claude -p inside the worktree
- search related code via Product Graph MCP
- pull error logs from Loki via Grafana MCP
- identify root cause and fix
- update tests as needed
- conventional commit
↓ git push + gh pr create
[fix PR]
↓ auto-review (the Part 3 pipeline)
↓ APPROVE -> auto-merge -> auto-redeploy
[recovered]
↓ ✅ in the Slack thread
What happens when the AI judges "this is not fixable in code"
Not every alert is fixable by code. The implementation has a rule: "if you judge it unfixable, exit without changing anything." In that case Slack gets a notification of the form "This alert cannot be addressed in code. Investigation: ..." -- including what the AI investigated.
Worth clarifying on the numbers side: the headline 115 is "Self-Healing runs that reached PR-created -> merged -> deployed." This "unfixable, exit clean" case is a separate bucket, happening several times a month (external transient outages, infra / config issues that aren't code, cases too complex for the AI to judge confidently). The "humans show up" bucket is this separate one -- it isn't a "some of the 115 failed" failure rate.
Here's what a real Slack message looks like (a styling-pattern-watch-transformer case where the GitHub PAT had expired):
ℹ️ This alert cannot be addressed in code. Investigation:
Investigation summary
Checked the error logs in Loki for the past hour. The single error source for styling-pattern-watch-transformer right now is expiration / revocation of the GitHub PAT (Personal Access Token).
What I found
The BQ snapshot pipeline is healthy: 7 consecutive [BQ] Snapshot rows inserted successes. The earlier autodetect type-drift problem was resolved in an earlier PR that put SAFE_CAST(... AS INT64) in place.
The single ongoing error (multiple occurrences in the past hour): GithubUnauthorizedError: GitHub API /commits/master unauthorized (401): PAT likely expired/revoked -- the /check-drift handler returns 401 every time Cloud Scheduler invokes it.
Root cause
The GitHub PAT held in Secret Manager for this service has expired or been revoked. The code already classifies the 401 with a dedicated GithubUnauthorizedError and is already sending the "PAT needs rotation" notification to Slack. The detection / notification design on the code side is working as intended.
Resolution: exit without changes
This is an external-credential operational issue (PAT expiry), and what needs to happen is a human rotates the PAT in Secret Manager. Not the kind of incident code can resolve. Therefore: no code change, no commit, no PR.
Operator actions:
Add a new version of the target secret in GCP Console / Secret Manager with a valid PAT (repo scope, read access to the target repository)
No Cloud Run revision redeploy needed (secretKeyRef version:latest is referenced)
At this level of detail, "what was investigated, why code can't fix it, what the human should do" all come out in one Slack message. Open the thread and the operator can act immediately. The productivity gap vs. "alerts just forwarded blindly" is significant.
Deduplication
A throttle ensures the same fingerprint (Grafana's unique alert identifier) is not re-processed for 4 hours. Without this, alerts that fire again before the fix PR has merged would spawn another worktree, another fix PR, and so on -- an easy infinite loop.
We also permanently skip any alertname containing credential. Credential incidents carry leakage risk if the AI touches them, so they're explicitly escalated to humans.
Self-Healing and Part 3 auto-review -- "the fixer AI" and "the reviewer AI" are independent
This is the most consequential design choice of the agent setup, so calling it out explicitly.
PRs opened by Self-Healing are not special PRs, just fix PRs. They go through the Part 3 auto-review pipeline under exactly the same conditions -- the 9 lenses (Graph / Architecture / Security / Test / Doc / Impact / Observability / AI-Antipattern / Recurrence) get checked in order. Critical / Major findings -> REQUEST_CHANGES; Nit-only / no findings + CI green -> APPROVE -> auto-merge.
The important bit: this is not a monolithic "AI fixing AI" loop. The fixer-side AI and the reviewer-side AI are fully independent:
Different process, different session: the self-healing-mode AI and the reviewer-mode AI are launched as separate claude -p processes. They do not share context
Different input sources: the fixer builds the problem from Grafana alert + Loki + cpg. The reviewer judges from the PR diff + cpg + review guidelines
Different objectives: the fixer is optimizing for "stop the incident." The reviewer is judging "does this violate the 9 lenses or the severity contract?" A deliberate separation of concerns where the two roles' incentives are intentionally misaligned
As a result, PRs the fixer dashed off get blocked by the reviewer (REQUEST_CHANGES -> back to the fixer). The AI does not approve its own output. "Just-make-it-work" fixes don't get through.
This is the often-debated review-independence problem in LLM-agent operation, solved here in the obvious way: split the work across separate agents.
A concrete example: meet subscription's 409 ALREADY_EXISTS
Take the alert from the Google Meet recording auto-fetch service I covered in ) executed autonomously by the AI.
This is the layer that keeps Self-Healing from being just "auto-repair."
In Fowler's Guides / Sensors terms from , , etc. -- used as decision criteria by auto-review
The 9 lenses, severity contract, and no-downgrade rules from to see impact scope across the codebase
cpg is what lets the AI ask "where else does this trap exist." Self-Healing and auto-review (= the Sensors side) share cpg as a substrate, and each run thickens Guides by one notch.
"Add the guard while you fix the bug" runs as a self-sustaining loop driven by Self-Healing.
"We'll do it later" and "introduce as warn" are banned
A couple of important contract clauses from the guidelines:
"Plan to lint later," "lint when we refactor," "another PR will handle this" -- all banned. If it can be addressed in this PR, it must be
"Existing violations remain, so introduce as warn and promote to error later" -- not accepted. This is deferral in disguise. The responsibility for the warn->error promotion goes nowhere and the rule rots
If you add a lint rule, fix all existing violations in the same PR and ship at error
These extend the no-downgrade rules from , caught by auto-review) -- forbids logger.error(err.message) style logs that drop the stack and keep only the message string. Forces the err field to hold serializeError(error) so name / message / stack are preserved as structured fields. Observability is everything here, so logs that drop stack info are treated as inherently broken
cortex-quality/require-fetch-timeout (oxlint -- a Rust-implemented JS/TS lint that runs ESLint-compatible rule sets, dozens of times faster than ESLint due to the Rust impl. cortex uses oxlint for the standard ruleset and ESLint for custom rules that need AST-level work) -- mandates signal: AbortSignal.timeout(...) on external fetch calls. Born from a case where a no-timeout fetch hung indefinitely and triggered a Cloud Tasks redelivery storm
graph/no-bq-string-timestamp-param (ESLint) -- from a case where passing TIMESTAMP as a string to a BigQuery query parameter NULLed the value out through a serializer bug and silently failed every INSERT
graph/require-firestore-ignore-undefined (ESLint) -- forces ignoreUndefinedProperties: true on new Firestore(). From a case where a single NULL row caused a 100% failure rate in a sync batch
check-otel-env-injection (CI guard) -- the recurrence prevention for the Cloud Run OTel env injection case below
TypeScript type tightening (type level) -- tighter function signatures, branded types for ID disambiguation, exhaustive discriminated unions, etc. Patterns that can't be lint-caught but are catchable at the type level get closed from the type side
These aren't textbook-learnable rules -- they're "stepped on once, then mechanized." The number of traps the organization has stepped on translates directly into the number of Guides piled up (across ESLint / oxlint / CI guard / types).
How does the AI write a lint rule without breaking it?
Three structural things keep this sane:
Existing rules are the template: packages/eslint-plugin-graph/src/rules/ already holds 26 custom rules, each as .ts + .test.ts pairs. New rules follow the same shape, so the AI never has to write the AST-walking boilerplate from scratch
Tests first: violation / pass fixtures go into .test.ts first, implementation fills in TDD-style. Coverage threshold (90% statements + branches) is gated by the groups lint / type constraint / CI guard together as the "lint-required" row, and leaves the choice within that bucket (write it as a lint? express it at the type level? add a separate CI guard?) to the AI based on how much AST work is involved and whether runtime semantics matter. Traps that need AST inspection but actually hinge on runtime behavior usually end up as a type constraint (branded type / discriminated union / signature tightening) rather than a custom lint
So "AI writes a lint rule" is supported by existing rule corpus + the test harness + the mechanize-bucket selection criteria -- three together. The path where the AI hand-rolls raw ESLint API and bricks something is structurally closed.
A concrete example: Cloud Run OTel env injection -> promoted to CI guard
Multiple services hit this trap: when a Cloud Run Service / Job is defined in Pulumi, forgetting to inject OTEL_EXPORTER_OTLP_ENDPOINT and GRAFANA_CLOUD_API_KEY via secretKeyRef causes OTel init to be skipped in production, no trace/log reaches Grafana, and incidents become silently invisible.
The normal response would be "we'll be more careful next time." At cortex:
Incident surfaces -> Self-Healing opens a fix PR (adds the env injection to the affected service)
Auto-review's [Recurrence] decides "same trap stepped on -> lint required"
The same PR adds scripts/check-otel-env-injection.ts (CI guard) -- mechanically asserts OTel env injection across all Cloud Run resource definitions under infra/
All other existing services get their env injection added in the same PR
Merge -> deploy -> any future write of the same kind gets rejected by CI
That's what "the guardrails grow every time Self-Healing runs" looks like in practice. The trap is "stepped on -> mechanically checked from then on."
Where Guides stand right now (in numbers)
Snapshot of cortex's Guide inventory:
Category
Count
Notes
Custom ESLint rules (@cortex/eslint-plugin-graph)
26
no-silent-catch / require-firestore-ignore-undefined / no-bq-string-timestamp-param etc.
CI guards (scripts/check-*.ts)
13
check-otel-env-injection / check-cloudscheduler-oidctoken-audience etc.
Standard oxlint rules (set to error)
183
Base config ships everything at error
TypeScript strict gates (baseline)
9
strict / noImplicitAny / strictNullChecks / noUncheckedIndexedAccess etc.
TypeScript type tightening (per-recurrence)
grows over time
branded type / discriminated union / function-signature tightening etc. Patterns that can't be lint-caught but can be type-caught are closed from the type side
Test coverage thresholds
statements + branches 90%
Uniform across all packages
Prettier
1 config
Format auto-fix
Guidelines
the entire review-guidelines repo
Used as the decision basis by auto-review
The first two categories plus the type-tightening row -- Custom ESLint, CI guard, type tightening -- are the part that compounds over time through the [Recurrence] lens every time Self-Healing or auto-review runs. The guardrails grow with time. That's the substance of the Strengthening layer.
The whole loop, from the top
When you compose the three layers:
CODE
[production anomaly] -> Observation layer (OTel/Loki/Grafana) -> Alert firing
↓
Event Relay -> SSE
↓
[Self-Healing mode script]
- claude -p in worktree
- root cause via cpg + Loki + git blame
- commit fix
- (if applicable) add new lint / type gate too
- gh pr create
↓
[Auto-review (Part 3)] -- 9 lenses in order, especially [Recurrence] forces
recurrence-prevention action (lint / horizontal expansion / guideline entry)
↓
APPROVE + CI green
↓
[auto-merge -> Turborepo build -> Pulumi parallel deploy]
↓
[production recovered + same anti-pattern mechanically rejected from now on]
The loop completes without human intervention. Not just repair, but the quality gates that grow with every repair -- that's the "auto-recovery + auto-strengthening" substance at cortex.
That said, as the front of the article spelled out, the loop is only viable because cpg and Observability exist. cpg makes horizontal expansion possible; Observability turns production anomalies into structured data. With those two in place at the foundation, AI can stand on the side that does Repair and Strengthening. Self-Healing is not a standalone mechanism. It's a Sensor riding on top of cortex's Guides (cpg + Observability + lint + guidelines). That's the single most important framing in this post.
Self-Healing by the numbers
Breaking the headline down further.
Main firing categories
What kicked off Self-Healing in the past 30 days (with the mapping back to the front-of-post 2 buckets):
Category
Bucket
Service Error Log Detected (most frequent)
Production-runtime (61 side)
Pipeline Failure -- data pipeline failing a configured number of times in a row
Production-runtime (61 side)
Generator Failure -- AI generation jobs (embedding / annotation etc.) failing
Production-runtime (61 side)
Deploy Failed -- deploy step failures (Pulumi up / Cloud Run revision failed)
Deploy step (54 side)
Alert-firing to production-recovery time
Median 30 minutes to 1 hour. Roughly:
Alert firing -> AI investigation start: under 1 minute (Event Relay + SSE)
AI investigation + fix + PR open: 3-8 minutes
Auto-review (including the : the cortex big picture and harness-engineering framing
: auto-review -- defending quality at the PR stage
Part 4 (this post): Self-Healing + Observability + auto-added guardrails -- defending quality in production while growing the quality gates themselves
The engineering role has shifted, over the last half-year, from "write, review, fix, merge, deploy, incident-respond" -- all of that -- toward looking at the whole system from above and tuning it. human-on-the-loop, working at the Policy layer.
That said, this is a pattern that solidified inside cortex, the internal AI platform. Carrying the same pattern into real consumer-facing toC services (multiple services, multiple stacks, multiple teams) requires changes and additions.
Part 5 will cover scaling cortex's harness to the whole product organization -- the roadmap and the thinking. The first half is the actual operation of "non-engineers opening PRs into cortex" with its limits; the second half is the elements needed to extend the pattern to toC services (service-specialized review rules, the human understanding-of-AI-design process, IaC for test environments, etc.).
"cortex built the pattern, toC services run that pattern at an order-of-magnitude-larger scale" -- that's the Part 5 positioning.
The actual series wrap-up is Part 6. The center of it is the underlying philosophy -- why I picked this design, what I gave up, what I kept. Alongside that, since the series so far has been mostly "what's working," I want to look back at the failures and dead ends behind that surface, and the gap between the philosophy and the implementation. A retrospective for myself, and -- hopefully -- a reference for anyone starting down a similar path.
Vollständiger Original-Artikel
Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.
SOCIAL SHARE CARD GENERATOR