Web TippsUse custom web fonts in Google Sheets charts(08.09.2026 um 17:05 Uhr)
Web TippsIntroducing the new 1Password App for Google Chat(08.09.2026 um 18:02 Uhr)
Web TippsUse custom web fonts in Google Sheets charts(08.09.2026 um 17:05 Uhr)
Web TippsIntroducing the new 1Password App for Google Chat(08.09.2026 um 18:02 Uhr)

🔧 Programmierung 🕛 vor 2 Monaten 12 Min Lesezeit
0

Surviving N4 stockouts in GKE

↗ Quelle (dev.to)
🗣️ Stimme:
📑 Inhaltsübersicht




The problem I ran into



I was bringing up the startup infrastructure for a solution standardized on the N4 machine series. N4 is a great general-purpose default for us, so the design assumed N4 capacity would be there when we asked for it.



It wasn't. In several of our target regions I simply could not get N4 nodes when I needed them. New nodes wouldn't come up during scale-up events, and worse, I couldn't even create the initial infrastructure in some zones. The cluster autoscaler kept logging the now-familiar Compute Engine signal:




CODE
ZONE_RESOURCE_POOL_EXHAUSTED
The zone 'projects/PROJECT_ID/zones/ZONE' does not have enough resources
available to fulfill the request. Try a different zone, or try again later.






This is not a quota problem and not a config problem. It's a capacity problem: GCP didn't have the N4 hardware free in that zone at that moment. A few things made this clear to me as I dug in.



Google's own resource-availability troubleshooting page is explicit that these errors happen when you request resources in a zone that can't currently accommodate them, that they're unrelated to your quota, and that they only apply to new resource requests — existing VMs keep running fine. Their recommended mitigations are to retry in another zone, retry in another region, retry later, or change the hardware configuration you're asking for. ()



And I'm clearly not alone. The lack of capacity for specific machine families in specific zones is a long-running, well-documented reality on GCP. You can find it discussed in places like the GCE discussion group thread on , and the Google developer-forum report of quotes Delivery Hero saying the thing I wanted to be able to say: that with fallback rules they stopped having to worry about instance availability at all.



So I stopped trying to force N4 and instead made my infrastructure resilient to N4 being temporarily unavailable.






The fix: a dedicated cluster with a cluster-wide default ComputeClass



My solution doesn't bolt a ComputeClass onto an existing shared cluster, and it deliberately does not attach the class to individual workloads with per-Pod selectors. Instead, I provision a specific, dedicated GKE cluster for the whole solution, and I make a single custom ComputeClass the cluster-level default. Every Pod that lands on that cluster inherits the N4→C4 behavior automatically, whether or not it asks for it. The workload manifests stay completely clean — no nodeSelector, no per-namespace labels, nothing for app teams to remember.



A ComputeClass is a Kubernetes Custom Resource (CRD, apiVersion: cloud.google.com/v1, kind: ComputeClass) that lets me declare an ordered list of node configurations — "priorities" — that GKE walks through when it needs to scale up. ()




The reason the dedicated cluster + cluster-level default combination matters — and isn't just a stylistic choice — is covered in the next section. The short version: it's the only configuration where my N4 fail-back actually fires for the whole solution.






The default ComputeClass I deployed



To make a custom ComputeClass the cluster-wide default, you give it the reserved name default. GKE then uses its priority rules as the autoscaling rules for any Pod that doesn't select a different class. () I use DoNotScaleUp where I'd rather alert on Pending Pods; if you'd rather keep running on anything, use ScaleUpAnyway.



Because this is the cluster default, workloads need no changes at all — there are no nodeSelector blocks and no namespace labels to apply. One caution that informed the dedicated-cluster decision: GKE recommends you use either ComputeClasses or individual node selectors, but not both, since mixing them causes unexpected scheduling. A dedicated cluster where the default class governs everything avoids that conflict entirely. ()





  • Cluster-level default (the class named default): requires GKE 1.33.1-gke.1744000 or later.


  • Namespace-level default (a cloud.google.com/default-compute-class label on the namespace), applied to only non-DaemonSet Pods: requires GKE 1.33.1-gke.1788000 or later.



The behavioral difference is the trap, and it's the reason a dedicated cluster with a cluster-level default is the right shape for my solution rather than a namespace label on a shared cluster:




  • With a cluster-level default, activeMigration.optimizeRulePriority does fire — if lower-priority C4 nodes exist and GKE can now create N4 nodes, it migrates. That's exactly my N4 fail-back.

  • With a namespace-level default, active migration is not triggered at all, because GKE only injects the ComputeClass selector into newly created Pods. Existing Pods are never re-created with the selector, so they never migrate back to N4. (, , )



    On the tooling side, the versions I standardized on:





    • Terraform CLI ≥ 1.5 (I run 1.15.x).


    • hashicorp/google provider 7.x — I pin ~> 7.36. (The 7.0 line has been GA since 2025; 7.36 was current when I wrote this.)


    • hashicorp/kubernetes provider ≥ 2.30 to apply the ComputeClass CRD through Terraform.

    • If you use the community terraform-google-modules/terraform-google-kubernetes-engine module with its enable_default_compute_class plumbing, it requires the google provider ≥ 7.10.






    Terraform



    I provision the dedicated cluster and its default ComputeClass as code. Three pieces: providers (pinned), the dedicated cluster (with the version floor and autoscaling/NAP settings), and the default ComputeClass manifest.






    Providers and version pinning






    CODE
    terraform {
    required_version = ">= 1.5"

    required_providers {
    google = {
    source = "hashicorp/google"
    version = "~> 7.36"
    }
    kubernetes = {
    source = "hashicorp/kubernetes"
    version = "~> 2.30"
    }
    }
    }

    provider "google" {
    project = var.project_id
    region = var.region
    }









    The dedicated cluster — pin the version, turn on autoscaling / NAP






    CODE
    # A cluster built specifically and exclusively for this solution.
    resource "google_container_cluster" "solution" {
    name = "n4-solution-euw3"
    location = var.region

    # Rapid channel keeps us on a version new enough for both the
    # cluster-level default ComputeClass AND ComputeClass-driven
    # node-pool auto-creation without cluster-level NAP.
    release_channel {
    channel = "RAPID"
    }

    # Insist on a control-plane version that supports the cluster-level
    # default ComputeClass and auto-creation. This is the binding floor.
    min_master_version = "1.33.3-gke.1136000"

    remove_default_node_pool = true
    initial_node_count = 1

    # Cluster-level node auto-provisioning. REQUIRED if your control plane is
    # older than 1.33.3-gke.1136000; optional (but harmless) at/after it.
    cluster_autoscaling {
    enabled = true

    resource_limits {
    resource_type = "cpu"
    minimum = 4
    maximum = 512
    }
    resource_limits {
    resource_type = "memory"
    minimum = 16
    maximum = 2048
    }

    auto_provisioning_defaults {
    management {
    auto_repair = true
    auto_upgrade = true
    }
    }
    }
    }

    # A small autoscaling system pool so the Standard cluster satisfies the
    # "at least one node pool has autoscaling enabled" requirement.
    resource "google_container_node_pool" "system" {
    name = "system"
    cluster = google_container_cluster.solution.id
    location = var.region

    autoscaling {
    min_node_count = 1
    max_node_count = 3
    }

    node_config {
    machine_type = "e2-standard-4"
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
    }
    }









    The cluster-level default ComputeClass as a Terraform-managed manifest






    CODE
    data "google_client_config" "default" {}

    provider "kubernetes" {
    host = "https://${google_container_cluster.solution.endpoint}"
    cluster_ca_certificate = base64decode(
    google_container_cluster.solution.master_auth[0].cluster_ca_certificate
    )
    token = data.google_client_config.default.access_token
    }

    resource "kubernetes_manifest" "default_compute_class" {
    manifest = {
    apiVersion = "cloud.google.com/v1"
    kind = "ComputeClass"
    metadata = {
    # "default" => cluster-wide default. Requires GKE 1.33.1-gke.1744000+.
    name = "default"
    }
    spec = {
    priorities = [
    { machineFamily = "n4", spot = false },
    { machineFamily = "c4", spot = false },
    ]
    # Fires for a CLUSTER-level default; would NOT fire for a namespace default.
    activeMigration = {
    optimizeRulePriority = true
    }
    nodePoolAutoCreation = {
    enabled = true
    }
    whenUnsatisfiable = "DoNotScaleUp"
    }
    }

    depends_on = [google_container_cluster.solution]
    }







    Gotcha worth knowing: kubernetes_manifest validates against the cluster's API at plan time, so the ComputeClass CRD must already exist on the control plane when you plan. On a brand-new cluster that means a two-phase apply (cluster first, then the manifest), or driving the kubectl apply from a null_resource/local-exec against the rendered YAML. On GKE the CRD ships with the control plane, so once the cluster exists, the manifest applies cleanly.




    Because this is the cluster default, there are no workload-side Terraform or manifest changes. Anything scheduled on this dedicated cluster inherits N4→C4→N4 automatically.






    What I'd tell my past self



    The N4 stockouts were never going to be "fixed" by retrying harder — capacity in a given zone isn't something I control. What I could control was whether a missing N4 node meant a stalled rollout or a transparent fall-through to C4. Building a dedicated cluster with a cluster-level default ComputeClass gave me a declarative, GitOps-friendly way to express "prefer N4, accept C4, return to N4" for the entire solution at once — no per-workload wiring, and no risk of mixing class selection with node selectors.



    Three things genuinely mattered in practice: set whenUnsatisfiable explicitly so an upgrade never changes my failure mode behind my back; use a cluster-level default, not a namespace label, because only the cluster-level default actually triggers the active migration back to N4; and pin the GKE version to at least 1.33.3-gke.1136000 on Rapid, because the default-ComputeClass and auto-creation features I'm relying on simply don't exist on older clusters. Get those right and the stockouts stop being incidents and become a non-event.

    Vollständiger Original-Bericht
    Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
    ↗ Original-Artikel auf dev.to lesen
Wie bewertest du diesen Beitrag?
1 Klick Feedback
Teilen mit Netzwerk & Team:

Community-Analysen & Experten-Meinungen 0

Verfasse deine eigene Analyse, teile Workarounds oder diskutiere diesen Vorfall im Blog.
Noch keine Community-Analyse verfasst. Markiere einen Textabschnitt oder klicke oben auf Eigene Analyse verfassen“!
Community Pulse: Relevanz-Einschätzung
1 Klick Experten-Votum
🔴 Akute Relevanz 0%
🟡 In Evaluierung 0%
🟢 Keine Auswirkung 0%
Spannende Innovation 0%
Verwandte Story-Cluster & Quellen (Vektor-KI)
Port 8095 Engine
3 Quellen
Use custom web fonts in Google Sheets charts
2 Quellen
Introducing the new 1Password App for Google Chat
1 Quelle
Context-aware access controls are available for Gemini Enterprise in the Admin console
Ähnliche Beiträge
🔍 Verwandte News

Auch interessante Nachrichten Surviving N4 stockouts in GKE

Thematisch verwandte Begriffe: Surviving, stockouts · 6 Treffer

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...