🔧 AI Nachrichten The Next Terrorist Attack Is Predictable(10.09.2026 um 23:41 Uhr)
🔧 AI Nachrichten Could A.I. Really Kill All Humans?(10.09.2026 um 23:53 Uhr)
🔧 AI Nachrichten Amazon Prime Video Uses A.I. for Lip-Synced Translations(11.09.2026 um 01:56 Uhr)
🔧 AI Nachrichten McClatchy Makes Deep Job Cuts to Newspapers Around the Country(11.09.2026 um 04:25 Uhr)
🔧 AI Nachrichten Law schools tell students to put AI away(07.09.2026 um 16:37 Uhr)
🔧 AI Nachrichten Can Huawei build China’s answer to ASML?(08.09.2026 um 04:57 Uhr)
🔧 AI Nachrichten AI is ushering in an era of mass toe-treading at work(08.09.2026 um 06:00 Uhr)
🔧 AI Nachrichten The Next Terrorist Attack Is Predictable(10.09.2026 um 23:41 Uhr)
🔧 AI Nachrichten Could A.I. Really Kill All Humans?(10.09.2026 um 23:53 Uhr)
🔧 AI Nachrichten Amazon Prime Video Uses A.I. for Lip-Synced Translations(11.09.2026 um 01:56 Uhr)
🔧 AI Nachrichten McClatchy Makes Deep Job Cuts to Newspapers Around the Country(11.09.2026 um 04:25 Uhr)
🔧 AI Nachrichten Law schools tell students to put AI away(07.09.2026 um 16:37 Uhr)
🔧 AI Nachrichten Can Huawei build China’s answer to ASML?(08.09.2026 um 04:57 Uhr)
🔧 AI Nachrichten AI is ushering in an era of mass toe-treading at work(08.09.2026 um 06:00 Uhr)

🔧 Programmierung 🕛 vor 2 Monaten 12 Min Lesezeit
0

Tombstone v1.3-v1.4: Resilience Was the Easy Layer

↗ Quelle (dev.to)
🗣️ Stimme:
📑 Inhaltsübersicht

I removed four || true statements from a GitHub Actions workflow on July 9th and watched CI go red in four different ways within the same run. Not one failure. Four — a Python pytest install that had never actually finished, a ruff lint violation nobody had looked at, a Ruby require path that resolved to nothing, and a Java Gradle wrapper that didn't exist in the repo. All four had been "passing" for who knows how long, because the test steps were configured to succeed no matter what came back.





v1.3.0 closes that gap: deployment-evaluator.yaml, deployment-intelligence.yaml, and deployment-marketplace.yaml now ship in the chart. evaluator gets an optional HPA behind evaluator.autoscaling.*. intelligence exposes IS_PRIMARY_REGION straight from values.yaml, which matters for the multi-region setup where secondary regions run in read-only relay mode.



Here's the footgun worth flagging, because it isn't obvious until it bites you: every one of those templates uses tombstone.selectorLabels in spec.selector.matchLabels, not the separate tombstone.labels helper, which includes a version label. It's tempting to use the one "labels" helper everywhere for consistency. Don't. Kubernetes Deployment selectors are immutable once the object exists. If your selector helper includes a label that changes on every release — like a chart version — the second helm upgrade you ever run will fail outright, because the new selector no longer matches the old one. tombstone.selectorLabels is a narrower, stable subset — name and component, nothing that changes across releases — specifically so matchLabels never drifts. One line in a template, and it's the difference between a chart that upgrades cleanly forever and one that works exactly once.






SDK parity is a correctness bug, not a feature request





The previous article's two worst bugs — the Slack kill switch sending environment as a query param when the handler read it from the JSON body, and the four-eyes approval routes that were built, reviewed, and merged but never registered in flag-api/cmd/main.go — share a root cause that has nothing to do with the code itself. Nobody had an easy way to look at what flag-api actually exposed versus what the proto files said it should expose. The OpenAPI spec existed; nobody was looking at it, because there was nowhere convenient to look.



v1.3.0 adds a Redoc explorer at GET /api/v1/docs, embedded via go-redoc rather than pulled from a CDN — it reads the existing grpc-gateway OpenAPI spec at /api/v1/openapi.json, so there's no second source of truth to keep in sync. The implementation detail that made this take longer than expected: the plan referenced a chi adapter for go-redoc that doesn't exist in any published version — the library only ships gin/fiber/echo adapters. The fix was to call goredoc.Redoc{SpecPath: specURL}.Body() directly for pre-rendered HTML and wrap it in a plain http.HandlerFunc. Small feature, but it's the direct answer to how those two bugs happened in the first place: the API surface wasn't something anyone could casually glance at.






GitOps: ordering discipline, then a second controller on purpose



v1.4.1 brought Flux CD v2.3+ into gitops/, structured as three layered Kustomizations — infrastructure, apps, flags — each declaring dependsOn against the one before it. The infrastructure layer also sets healthChecks against the tombstone-operator HelmRelease, which matters more than it sounds: dependsOn alone only blocks applying a Kustomization until the upstream has been applied — it does not wait for the upstream to actually be healthy. Without healthChecks, Flux considers the operator's CRDs "done" the moment the HelmRelease manifest hits the API server, before the CRDs are actually established, and the apps layer — which deploys FeatureFlag CRs via the flagmind chart — could start reconciling against CRDs that technically exist but aren't ready yet. Pairing dependsOn with healthChecks on the upstream is what actually guarantees ordering — this is exactly the kind of GitOps bug that works fine in every test until the day the operator pod is slow to come up.



The more interesting decision is v1.4.2's addition of Argo CD v2.11 — not as a Flux replacement, as a second controller with a deliberately split job. Flux keeps infrastructure: the operator's CRDs, image update automation across the tombstone container images. Argo CD takes over the flagmind chart plus the FeatureFlag and FlagPolicy custom resources themselves. The reason to split: FeatureFlag CRs have a rolloutPct field that the intelligence service's ML rollout recommendations mutate live in the cluster, independent of Git. A naive GitOps setup would see that mutation as drift and revert it on the next reconcile — the platform's own ML-driven rollout logic fought and undone by its own deployment tooling every sync interval.



The fix lives in gitops/clusters/production/argocd/apps.yaml: an ignoreDifferences block excluding /spec/environments/production/rolloutPct and /spec/environments/staging/rolloutPct from diff detection, paired with RespectIgnoreDifferences=true in syncOptions. Both are required — ignoreDifferences alone only suppresses the OutOfSync display; without the sync-options flag, Argo CD still overwrites the field back to the Git value on every sync. Miss either half and the ML rollout percentage gets silently stomped on a fixed interval, a nasty class of bug to chase down because nothing in the intelligence service's logs would look wrong.



Argo CD also needed Lua health checks for Tombstone's own CRDs — FeatureFlag (Pending to Progressing, Synced to Healthy, Error to Degraded) and FlagPolicy (Compliant to Healthy, Violation to Degraded) — because its generic health rollup doesn't understand custom CRD status fields out of the box. Without them, the root Application would just show every FeatureFlag as permanently "Unknown" rather than reflecting real reconciliation state.






Blast radius, but as a deployment gate now





Alongside the CI honesty pass, all remaining GitHub Actions workflow files got pinned to immutable commit SHAs instead of mutable tags like @v4 — nine files in one commit (bfb167f), closing out the supply-chain hardening that earlier workflows (flux-bootstrap.yml) had already established as the convention. Two other release blockers landed in the same commit: an ImageUpdateAutomation resource still on the wrong beta API version, and a structural fix to the production rollouts kustomization that was silently missing a resource reference.



Everything above — the layered Flux Kustomizations, the Argo CD split, the blast-radius AnalysisTemplate — is validated against a local k3d test cluster: every kustomize build passes, Flux bootstraps cleanly, Argo CD installs and reconciles. What it is not yet validated against is the actual production target, Oracle Cloud Kubernetes, blocked on an Oracle Cloud account signup that hasn't happened yet. The operator Helm chart is already published to ghcr.io/sairam0424/charts/tombstone-operator at v0.1.0, ready for the day the cluster exists. Better to say that plainly than let "GitOps shipped" imply more than k3d has actually proven.



The throughline across v1.2 through v1.4 is the same lesson at three different altitudes. v1.2 was runtime resilience — the system surviving its own dependencies failing. v1.3 was correctness at the API and SDK boundary — two clients agreeing on what a flag evaluates to. v1.4 is deployment and CI honesty — the pipeline that ships the system telling the truth about its own state, in the right order, without silently eating failures. None of these layers were broken in an obvious way. They were all quietly returning something other than the truth, and the only way to find that out was to stop letting them.

Vollständiger Original-Bericht
Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
↗ Original-Artikel auf dev.to lesen
Wie bewertest du diesen Beitrag?
1 Klick Feedback
Teilen mit Netzwerk & Team:

Community-Analysen & Experten-Meinungen 0

Verfasse deine eigene Analyse, teile Workarounds oder diskutiere diesen Vorfall im Blog.
Noch keine Community-Analyse verfasst. Markiere einen Textabschnitt oder klicke oben auf Eigene Analyse verfassen“!
Community Pulse: Relevanz-Einschätzung
1 Klick Experten-Votum
🔴 Akute Relevanz 0%
🟡 In Evaluierung 0%
🟢 Keine Auswirkung 0%
Spannende Innovation 0%
Verwandte Story-Cluster & Quellen (Vektor-KI)
Port 8095 Engine
2 Quellen
Could A.I. Really Kill All Humans?
1 Quelle
The Next Terrorist Attack Is Predictable
1 Quelle
Anthropic Says It Blocked Possible Efforts to Build Biological Weapons
Ähnliche Beiträge
🔍 Verwandte News

Auch interessante Nachrichten Tombstone v1.3-v1.4: Resilience Was the Easy Layer

Thematisch verwandte Begriffe: Tombstone, v13v14, Resilience, Easy · 6 Treffer

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...