I Revived a Broken MLOps Platform — Now It's Self-Service, Policy-Guarded, and Operationally Credible

Repo:

This is not a claim. The video below shows every check running live against a real k3d cluster.

━━━ Milestone A — GitOps Spine (ArgoCD) ━━━

  [✓ PASS] All ArgoCD pods are Running

  [✓ PASS] ArgoCD Applications: 7 Healthy, 0 Progressing, 7 total

  [✓ PASS] ArgoCD sync visibility: no Unknown states (7/7 Synced)

  [✓ PASS] Drift self-heal: nginx-test recreated and Ready in ~20s



━━━ Milestone B — AI Serving Baseline (KServe) ━━━

  [✓ PASS] KServe controller-manager: 1 replica available

  [✓ PASS] InferenceServices: 2/2 Ready=True

  [✓ PASS] Inference request: demo-iris-2 returned predictions

           ↳ Response: {"predictions":[1,1]}



━━━ Milestone C — Golden Path (Backstage) ━━━

  [✓ PASS] Backstage deployment: 1 replica available

  [✓ PASS] demo-iris-2 InferenceService exists (scaffolder output)

  [✓ PASS] demo-iris-2 ArgoCD Application exists (ApplicationSet output)



━━━ Milestone D — Guardrails (Kyverno + CI) ━━━

  [✓ PASS] Kyverno pods running: 3

  [✓ PASS] Kyverno ClusterPolicies installed: 5 policies

  [✓ PASS] Admission block: non-compliant InferenceService correctly denied



━━━ Milestone F — Production Hardening ━━━

  [✓ PASS] ApplicationSet neuroscale-model-endpoints exists

  [✓ PASS] ArgoCD has 7 Applications (ApplicationSet + static)

  [✓ PASS] ResourceQuota exists in namespace default

  [✓ PASS] LimitRange exists in namespace default

  [✓ PASS] Non-root admission block: root-container Deployment denied

  [✓ PASS] OpenCost deployment healthy: 1 replica available



  PASS 21 / FAIL 0 / SKIP 1

  ✓ All checks passed. Platform is healthy and ready to demo.

The single SKIP is the drift self-heal pre-condition check — normal after a previous test run. The drift self-heal itself passed, visible in the output above and in the video.

Reproducible on any machine:

bash scripts/bootstrap.sh     # ~5 minutes — requires Docker + k3d

bash scripts/smoke-test.sh    # 21 checks, all green

The Problem: A Platform That Was Abandoned and Dangerous

NeuroScale started in February 2026 as an AI inference platform on Kubernetes. By early April it was abandoned — Backstage crashing, ArgoCD broken, KServe unable to serve a single model. The last commit before this challenge was April 4th. Then 48 days of silence.

Here's what I found when I came back:

• 
  Backstage: CrashLoopBackOff — 14 restarts. A Helm values nesting bug caused probe timings to be silently ignored.


• 
  ArgoCD repo-server: CrashLoopBackOff — every application showed Unknown, meaning ArgoCD couldn't even evaluate their state.


• 
  KServe: READY=False — default config assumed Istio for ingress, but the cluster ran Kourier. Error: "virtual service not found".


• 
  Policy enforcement: None. Root containers, no resource limits, :latest tags — deployed freely.


• 
  Drift detection: None. Manual kubectl changes accumulated silently.

The deployment process was vim → kubectl apply → hope. Developers feared deploying models. The platform was technically worse than not having one.

What I Built: Five Enforcement Layers

generates a compliant InferenceService manifest, opens a PR, and the neuroscale-model-endpoints ApplicationSet auto-discovers it.

Two steps, 9 seconds total. PR opened, ApplicationSet picks it up on next ArgoCD sync.

Layer 2: GitOps Drift Control

Git is the source of truth. Drift is auto-corrected.

$ kubectl delete deploy nginx-test -n default

# 20 seconds later...

$ kubectl get deploy nginx-test -n default

NAME         READY   UP-TO-DATE   AVAILABLE   AGE

nginx-test   1/1     1            1           8s   # Auto-recreated by ArgoCD

selfHeal: true and prune: true. Manual cluster changes cannot persist.

Layer 3: Policy Guardrails — Shift-Left + Shift-Down

At PR time (CI): kubeconform validates schemas. kyverno-cli simulates all 5 policies against rendered manifests with a dual exit-code + stdout check to guard against false-greens. Full pipeline at

Five enforced policies:

Layer 4: Cost Attribution

Every workload carries owner and cost-center labels enforced by Kyverno — you can't deploy without them. OpenCost reads these via Prometheus for per-team cost breakdowns. The CI pipeline also .

Copilot Partnership: Three Moments That Mattered

Copilot didn't write this platform. It functioned as a senior infrastructure advisor at three exact moments where I could have stayed stuck for days.

Moment 1: The Architectural Decision — Kourier vs Istio

Problem: KServe stuck at READY=False. Error: "virtual service not found" — an Istio concept on a cluster running Kourier.

Copilot searched the actual repo files, confirmed the non-Istio setup was already correct, and identified the root cause: stale cached config. Critical tradeoff it surfaced — Istio adds ~1GB memory overhead; Kourier is under 200MB. On a shared 8GB dev node, Istio would have killed reproducibility.

Fix: Reapply the serving-stack overlay, verify disableIstioVirtualHost=true in ConfigMaps, restart control plane pods. Result: working inference, 800MB freed.

Moment 2: The Silent Bug — CI Guardrails That Can't False-Green

Problem: kyverno-cli apply looked green in CI. Then I tested with a deliberately non-compliant manifest. It still passed. The guardrail was checking nothing.

Two undocumented kyverno-cli behaviors Copilot surfaced:

1. A single --resource flag with multiple paths silently ignores every path after the first.


2. Exit code is 0 even when violations are printed to stdout.

The fix (live in

. The platform doesn't just work — it's recoverable. That's the difference between a demo and a real platform.

Before vs After

For Developers

For the Platform

What Made This Real: The Failures

This was not built on the happy path. Every milestone hit real failures:

Full failure log and recovery steps in docs/runbook.md.

Try It Yourself

# Clone and bootstrap (requires Docker + k3d + kubectl + helm)

git clone https://github.com/sodiq-code/neuroscale-platform.git

cd neuroscale-platform

bash scripts/bootstrap.sh     # ~5 minutes



# Verify everything works

bash scripts/smoke-test.sh    # 21 checks, 0 failures



# Open all UIs

bash scripts/port-forward-all.sh

After port-forward-all.sh:

• 
  Backstage at http://localhost:7010 — developer portal


• 
  ArgoCD at http://localhost:8080 — 7 synced applications


• 
  OpenCost at http://localhost:9090 — per-workload cost attribution

5 minutes from git clone to a fully working platform. The smoke test proves it all.

The Bottom Line

Copilot helped at the exact points where strong engineering judgment mattered most: an architectural tradeoff that saved 800MB of memory, a silent CI bug that every kyverno-cli user faces, and operational recovery that turns a 2-hour outage into a 2-minute runbook.

21 checks. 0 failures. Reproducible on any machine.

What's one abandoned project you wish you had finished? Drop it in the comments.