This is part of the Platform Engineering with Go series. This post builds on the minikube cluster set up in post 2. Read post 2 first if you haven't yet.
kubectl is great for humans. But what about programs?
When you run kubectl get pods, something has to talk to the Kubernetes API server, deserialize the JSON response, and format it for your terminal. That something is client-go, the official Go client library for Kubernetes.
Every tool in the Kubernetes ecosystem is built on it. kubectl itself, helm, ArgoCD, controller-runtime, Tekton. If it talks to a Kubernetes cluster and it's written in Go, it's using client-go.
In this post, we'll learn how client-go works by building something immediately useful: a cluster health reporter. One Go program that connects to your minikube cluster, scans every pod across every namespace, and prints a complete health summary, the kind of view that usually takes four or five kubectl commands to assemble.
Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC
NAMESPACE TOTAL RUNNING FAILING PENDING
──────────────────────────────────────────────────
default 3 3 0 0
kube-system 6 6 0 0
✓ All pods healthy
And if something is wrong:
Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC
NAMESPACE TOTAL RUNNING FAILING PENDING
──────────────────────────────────────────────────
default 3 1 2 0
kube-system 6 6 0 0
FAILING PODS
──────────────────────────────────────────────────
default go-api-7d6b9f8c4-mn9rt CrashLoopBackOff 14 restarts
default go-api-7d6b9f8c4-p8wvz OOMKilled 3 restarts
What you'll learn
What client-go is and how it relates to the Kubernetes API.
The four key packages and what each one contains.
How to connect to a cluster, out-of-cluster and in-cluster.
How the Clientset is organized, API groups explained.
How to list, filter, and inspect Kubernetes resources from Go.
How to build the cluster health reporter step by step.
How to test client-go code, two approaches with clear rules for when to use each.
Prerequisites
- minikube running with the
go-apideployment from post 2
What client-go is
The Kubernetes API is a REST API. When you run kubectl get pods -n default, kubectl makes an HTTP GET request to something like:
GET https://kubernetes/api/v1/namespaces/default/pods
The API server returns JSON. kubectl deserializes it and formats it for your terminal.
client-go is the library that handles all of this in Go: the HTTP calls, TLS, authentication, rate limiting, and deserialization into typed Go structs. Instead of working with raw JSON, you get a *corev1.PodList with typed fields you can navigate in your editor with autocomplete.
Your Go program
│
│ client-go
│ ├── kubeconfig loading
│ ├── TLS + authentication
│ ├── HTTP client + rate limiting
│ └── JSON → typed Go structs
│
▼
Kubernetes API server
│
▼
etcd (the cluster's data store)
Everything you can do with kubectl, you can do with client-go, and more. kubectl is itself a client-go program.
Project structure
Here is the Go project structure we are going to build; remember, you can see how to organize the code for your own projects here.
You can find the entire project here.
go-k8s-health-reporter/
├── go.mod
├── main.go ← entry point — wires dependencies, calls reporter
└── internal/
├── reporter/
│ ├── types.go ← data types — PodStatus, NamespaceSummary, Report
│ ├── reporter.go ← Reporter struct, Generate()
│ ├── inspector.go ← pod health detection — inspectPod(), pendingReason()
│ └── printer.go ← output formatting — Print(), tabwriter tables
└── kubernetes/
└── client.go ← KubeClient interface + real implementation
Installing the packages
Before writing any code, initialize the module and install the dependencies. The version numbers must match your Kubernetes cluster; since we're running Kubernetes 1.36.2, we use v0.36.2 for all three packages. Mixing versions causes compilation errors because the types won't match across packages. Remember to rename the module's name to your own module name.
# Initialize the Go module
mkdir go-k8s-health-reporter && cd go-k8s-health-reporter
go mod init github.com/FerRiosCosta/go-k8s-health-reporter
# Install the four packages pinned to Kubernetes 1.36.2
go get k8s.io/[email protected]
go get k8s.io/[email protected]
go get k8s.io/[email protected]
# Tidy up — removes unused dependencies and ensures versions are consistent
go mod tidy
If you see a
missing go.sum entryerror after installing the packages, it means some transitive dependencies, packages that client-go depends on, likek8s.io/klog/v2, weren't added togo.sumautomatically. Fix it with:
bash
go get ./...
go mod tidy
go get ./...scans all packages in your module and fetches every dependency recursively, including transitive ones.go mod tidythen writes the completego.sum.
Why pin to v0.36.2 instead of @latest? The client-go versioning scheme directly mirrors Kubernetes releases; v0.36.2 corresponds to Kubernetes v1.36.2. Always match your client-go version to your cluster version. If you run @latest you might get a newer client-go version than your cluster supports, which can cause unexpected API behavior.
You can verify your cluster version at any time:
kubectl version
# Client Version: v1.36.2
# Server Version: v1.36.2
Both should match (ideally). If they don't, pin your packages to match the server version.
Note:
k8s.io/metricsis not needed for the health reporter in this post, but you will need it for the next posts.
go get k8s.io/[email protected]
go mod tidy
Here's the go.mod:
module github.com/FerRiosCosta/go-k8s-health-reporter
go 1.26
require (
k8s.io/api v0.36.2
k8s.io/apimachinery v0.36.2
k8s.io/client-go v0.36.2
)
The four key packages
Before writing any code, let's understand the four packages client-go code almost always imports. This is the section that makes every future client-go import immediately readable.
k8s.io/api: the Go type definitions
import (
corev1 "k8s.io/api/core/v1" // Pod, Service, ConfigMap, Secret, Namespace, Node
appsv1 "k8s.io/api/apps/v1" // Deployment, StatefulSet, DaemonSet, ReplicaSet
batchv1 "k8s.io/api/batch/v1" // Job, CronJob
)
This package contains the Go struct definitions for every Kubernetes resource. When you write corev1.Pod{} or appsv1.Deployment{}, you're using this package. The aliases (corev1, appsv1) are conventions, the actual package names are just v1, which is too generic to use unaliased.
k8s.io/apimachinery: the shared building blocks
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ObjectMeta, ListOptions, Time
"k8s.io/apimachinery/pkg/api/resource" // resource.Quantity (CPU, memory)
)
This package contains types shared across all Kubernetes API groups. metav1.ObjectMeta is the block every resource has: name, namespace, labels, annotations, creation timestamp. metav1.ListOptions lets you filter API calls by label selectors and field selectors. You'll import something from this package in virtually every file that touches the Kubernetes API.
k8s.io/client-go: the actual client
import (
"k8s.io/client-go/kubernetes" // the Clientset — entry point to all APIs
"k8s.io/client-go/rest" // rest.Config — holds connection details
"k8s.io/client-go/tools/clientcmd" // reads and parses kubeconfig files
)
This is the main package. kubernetes.NewForConfig() creates the Clientset, your entry point to every Kubernetes API. clientcmd reads the ~/.kube/config file that minikube wrote in post 2. rest.Config holds the connection details: API server URL, TLS certificates, auth token.
k8s.io/metrics: the metrics-server API
import (
metrics "k8s.io/metrics/pkg/client/clientset/versioned" // metrics Clientset
)
This is a separate package because metrics-server is an extension API, not part of the core Kubernetes API.
# Install in post 4 — pinned to the same version as the rest
go get k8s.io/[email protected]
go mod tidy
Connecting to the cluster
Every client-go program starts the same way: build a rest.Config, then create a Clientset from it.
// internal/kubernetes/client.go
package kubernetes
import (
"context"
"fmt"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
)
// KubeClient defines the Kubernetes operations the health reporter uses.
// Wrapping the Clientset behind an interface makes the reporter testable
// without a real cluster, the same pattern used throughout this series.
type KubeClient interface {
// ListPods returns all pods in the given namespace.
// An empty namespace string returns pods from all namespaces.
ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)
}
// client is the real implementation backed by the official Kubernetes Clientset.
// It is unexported, callers always use the KubeClient interface.
type client struct {
clientset kubernetes.Interface
}
// NewClient creates a KubeClient connected to the cluster in kubeconfig.
func NewClient(kubeconfigPath string) (KubeClient, error) {
config, err := buildConfig(kubeconfigPath)
if err != nil {
return nil, fmt.Errorf("build config: %w", err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, fmt.Errorf("create clientset: %w", err)
}
return &client{clientset: clientset}, nil
}
// buildConfig tries in-cluster config first, then falls back to kubeconfig.
// This means the same binary works both locally (reads ~/.kube/config)
// and when deployed as a pod inside a cluster (reads the mounted service account token).
func buildConfig(kubeconfigPath string) (*rest.Config, error) {
// in-cluster config: reads /var/run/secrets/kubernetes.io/serviceaccount/token
// automatically mounted inside every pod by Kubernetes.
if config, err := rest.InClusterConfig(); err == nil {
return config, nil
}
// out-of-cluster config: reads the kubeconfig file from disk.
// This is the file minikube wrote to ~/.kube/config in post 2.
return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}
// ListPods retrieves all pods in the given namespace from the Kubernetes API.
// Passing an empty string returns pods from all namespaces.
func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
if err != nil {
return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
}
return pods, nil
}
The Clientset and API groups
c.clientset.CoreV1().Pods(namespace).List(ctx, opts), let's break this chain down because you'll see it everywhere in client-go code:
c.clientset // the Clientset, entry point to all Kubernetes APIs
.CoreV1() // select the "core" API group, version 1
.Pods(namespace) // select the Pods resource in the given namespace
.List(ctx, opts) // call the List operation with options
Kubernetes organizes resources into API groups. Each group contains related resources:
CoreV1() → Pods, Services, ConfigMaps, Secrets, Nodes, Namespaces
AppsV1() → Deployments, StatefulSets, DaemonSets, ReplicaSets
BatchV1() → Jobs, CronJobs
The pattern is always the same: clientset.APIGroup().ResourceType(namespace).Operation(). Once you know this pattern, any client-go call is readable.
Building the cluster health reporter
Now let's build the reporter. Instead of putting everything in one file, we'll split it across four focused files from the start, each with one clear responsibility.
internal/reporter/
├── types.go ← data types only — PodStatus, NamespaceSummary, Report
├── reporter.go ← Reporter struct, constructor, Generate()
├── inspector.go ← pod inspection logic — detects failing and pending states
└── printer.go ← output formatting — tabwriter tables
Each file has one reason to change:
New data field →
types.goNew data source →
reporter.goNew failure reason to detect →
inspector.goNew output format (JSON, CSV) →
printer.go
types.go, data types only
// internal/reporter/types.go
package reporter
import "time"
// PodStatus represents the health state of a single pod.
type PodStatus struct {
Namespace string
Name string
Phase string // Running, Pending, Failed, Succeeded
Reason string // CrashLoopBackOff, OOMKilled, etc. — more specific than Phase
Restarts int32 // total restarts across all containers
IsFailing bool // true if the pod is in a non-running problem state
IsPending bool // true if the pod is waiting to be scheduled or started
}
// NamespaceSummary holds the pod counts for a single namespace.
type NamespaceSummary struct {
Namespace string
Total int
Running int
Failing int
Pending int
}
// Report is the complete cluster health picture.
type Report struct {
GeneratedAt time.Time
Namespaces []NamespaceSummary
Failing []PodStatus
Pending []PodStatus
}
Thirty lines. No imports beyond time. If you need to add a field to a pod's health state, you open this file and only this file.
inspector.go, pod health detection
// internal/reporter/inspector.go
package reporter
import corev1 "k8s.io/api/core/v1"
// inspectPod examines a pod's status and returns a detailed PodStatus.
// It checks container states first, a pod can be in "Running" phase
// while individual containers are in CrashLoopBackOff or OOMKilled.
func inspectPod(pod *corev1.Pod) PodStatus {
status := PodStatus{
Namespace: pod.Namespace,
Name: pod.Name,
Phase: string(pod.Status.Phase),
}
// Sum restart counts and check container states.
// Must check containers before pod phase, a pod can be "Running"
// while a container inside it is waiting to restart.
for _, cs := range pod.Status.ContainerStatuses {
status.Restarts += cs.RestartCount
if cs.State.Waiting != nil {
switch cs.State.Waiting.Reason {
case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
"ErrImagePull", "CreateContainerConfigError":
status.Reason = cs.State.Waiting.Reason
status.IsFailing = true
return status
}
}
if cs.State.Terminated != nil {
reason := cs.State.Terminated.Reason
if reason == "OOMKilled" || cs.State.Terminated.ExitCode != 0 {
status.Reason = reason
status.IsFailing = true
return status
}
}
}
// Check pod-level phase for broader failure states.
switch pod.Status.Phase {
case corev1.PodFailed:
status.Reason = pod.Status.Reason
status.IsFailing = true
case corev1.PodPending:
status.Reason = pendingReason(pod)
status.IsPending = true
}
return status
}
// pendingReason extracts why a pod is pending from its conditions.
// The PodScheduled condition contains the human-readable scheduling
// failure message; e.g. "0/1 nodes are available: 1 Insufficient memory."
func pendingReason(pod *corev1.Pod) string {
for _, condition := range pod.Status.Conditions {
if condition.Type == corev1.PodScheduled &&
condition.Status == corev1.ConditionFalse {
return condition.Message
}
}
return string(pod.Status.Phase)
}
Understanding container states
The inspectPod function works by reading pod.Status.ContainerStatuses, a slice that contains one entry per container in the pod. Each entry has a State field that holds exactly one of three possible states at any given moment:
Waiting: the container is not yet running. This is where most failure reasons live:
| Reason | What it means |
|---|---|
ContainerCreating | Image is being pulled, or volumes are being mounted |
PodInitializing | Init containers are still running, main container hasn't started |
CrashLoopBackOff | Container keeps crashing, Kubernetes is backing off before restarting |
ImagePullBackOff | Image pull failed, Kubernetes is backing off before retrying |
ErrImagePull | The immediate image pull error, before backoff kicks in |
CreateContainerConfigError | Bad config, missing ConfigMap, Secret, or invalid env var |
OOMKilled | Killed by the OOM killer, waiting to restart |
Running: the container process is alive. Note that this doesn't mean the application inside it is healthy, it just means the process is executing. Your app could still be returning 500s or stuck in a deadlock.
Terminated: the container has finished. The most important fields are:
| Field | What it means |
|---|---|
ExitCode | 0 = clean exit, anything else = failure |
Reason | Completed (clean) or OOMKilled (killed by OOM killer) |
Common exit codes worth knowing:
| Exit code | Meaning |
|---|---|
0 | Clean exit, completed successfully |
1 | Application error, the process itself returned an error |
137 | SIGKILL, OOM killer fired, or forcefully killed |
143 | SIGTERM, graceful shutdown (normal during pod termination) |
This is exactly why inspectPod checks cs.State.Waiting and cs.State.Terminated separately; each state carries different information in different fields.
Init containers, the hidden failure
There's one more status field worth knowing about:
pod.Status.ContainerStatuses // current state of main containers
pod.Status.InitContainerStatuses // current state of init containers
Init containers run before your main containers and go through the same three states. If an init container is stuck in CrashLoopBackOff, your main container will never start, and pod.Status.ContainerStatuses will be empty, giving you no indication of what's wrong.
The current inspectPod implementation only checks main container statuses. A more complete version would check init containers first:
// A more complete inspectPod that also catches stuck init containers.
// This is left as an exercise — the pattern is identical to the main
// container check, just reading from InitContainerStatuses instead.
func inspectPod(pod *corev1.Pod) PodStatus {
status := PodStatus{
Namespace: pod.Namespace,
Name: pod.Name,
Phase: string(pod.Status.Phase),
}
// Check init containers first — if one is stuck, main containers
// will never start and ContainerStatuses will be empty.
for _, cs := range pod.Status.InitContainerStatuses {
status.Restarts += cs.RestartCount
if cs.State.Waiting != nil {
switch cs.State.Waiting.Reason {
case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
"ErrImagePull", "CreateContainerConfigError":
status.Reason = "init: " + cs.State.Waiting.Reason
status.IsFailing = true
return status
}
}
}
// Then check main containers as before.
for _, cs := range pod.Status.ContainerStatuses {
// ... same logic as above
}
// ... rest of the function unchanged
}
Prefixing the reason with "init: " makes it immediately clear in the report that it's an init container failing, not the main application.
You can add this code block before checking the main container in your project if you want.
printer.go, output formatting
// internal/reporter/printer.go
package reporter
import (
"fmt"
"io"
"strings"
"text/tabwriter"
)
// Print writes the health report as a formatted table to w.
// Accepting io.Writer instead of os.Stdout directly keeps this
// testable — callers can pass a bytes.Buffer to inspect the output.
func (r *Reporter) Print(w io.Writer, report *Report) {
fmt.Fprintf(w, "Cluster Health Report\n")
fmt.Fprintf(w, "Generated: %s\n\n",
report.GeneratedAt.Format("2006-01-02 15:04:05 UTC"),
)
printNamespaceSummary(w, report.Namespaces)
if len(report.Failing) > 0 {
printFailingPods(w, report.Failing)
return
}
if len(report.Pending) > 0 {
printPendingPods(w, report.Pending)
return
}
fmt.Fprintf(w, "\n✓ All pods healthy\n")
}
// printNamespaceSummary writes the namespace totals table.
func printNamespaceSummary(w io.Writer, namespaces []NamespaceSummary) {
tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
fmt.Fprintln(tw, "NAMESPACE\tTOTAL\tRUNNING\tFAILING\tPENDING")
fmt.Fprintln(tw, strings.Repeat("─", 52))
for _, ns := range namespaces {
fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%d\n",
ns.Namespace, ns.Total, ns.Running, ns.Failing, ns.Pending,
)
}
tw.Flush()
}
// printFailingPods writes the failing pods detail section.
func printFailingPods(w io.Writer, pods []PodStatus) {
fmt.Fprintf(w, "\nFAILING PODS\n")
fmt.Fprintln(w, strings.Repeat("─", 52))
tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
for _, p := range pods {
fmt.Fprintf(tw, "%s\t%s\t%s\t%d restarts\n",
p.Namespace, p.Name, p.Reason, p.Restarts,
)
}
tw.Flush()
}
// printPendingPods writes the pending pods detail section.
func printPendingPods(w io.Writer, pods []PodStatus) {
fmt.Fprintf(w, "\nPENDING PODS\n")
fmt.Fprintln(w, strings.Repeat("─", 52))
tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
for _, p := range pods {
fmt.Fprintf(tw, "%s\t%s\t%s\n",
p.Namespace, p.Name, p.Reason,
)
}
tw.Flush()
}
reporter.go, orchestration only
// internal/reporter/reporter.go
package reporter
import (
"context"
"fmt"
"time"
k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
)
// Reporter builds health reports from Kubernetes data.
// It depends on the KubeClient interface, never on a concrete
// client-go type, so it works with both real and fake clients.
type Reporter struct {
kube k8s.KubeClient
}
// New creates a Reporter with the given KubeClient injected.
func New(kube k8s.KubeClient) *Reporter {
return &Reporter{kube: kube}
}
// Generate fetches all pods from the cluster and builds a health report.
func (r *Reporter) Generate(ctx context.Context) (*Report, error) {
// Fetch all pods across all namespaces.
// Passing "" to ListPods returns pods from every namespace.
pods, err := r.kube.ListPods(ctx, "")
if err != nil {
return nil, fmt.Errorf("fetch pods: %w", err)
}
report := &Report{
GeneratedAt: time.Now().UTC(),
}
// Group pods by namespace using a map.
summaries := make(map[string]*NamespaceSummary)
for _, pod := range pods.Items {
// getOrCreate avoids the boilerplate of checking if the
// namespace key exists before writing to the map.
summary := getOrCreate(summaries, pod.Namespace)
summary.Total++
// inspectPod is defined in inspector.go, same package,
// no import needed.
status := inspectPod(&pod)
switch {
case status.IsFailing:
summary.Failing++
report.Failing = append(report.Failing, status)
case status.IsPending:
summary.Pending++
report.Pending = append(report.Pending, status)
default:
summary.Running++
}
}
for _, summary := range summaries {
report.Namespaces = append(report.Namespaces, *summary)
}
return report, nil
}
// getOrCreate returns the existing NamespaceSummary for the given namespace
// or creates a new one and registers it in the map.
// Extracted into its own function to keep Generate() readable.
func getOrCreate(m map[string]*NamespaceSummary, namespace string) *NamespaceSummary {
if s, ok := m[namespace]; ok {
return s
}
s := &NamespaceSummary{Namespace: namespace}
m[namespace] = s
return s
}
Notice that reporter.go now has no output imports (tabwriter, strings, io) and no Kubernetes type imports beyond what it needs for ListPods. The corev1 import moved entirely to inspector.go where it belongs. The reporter package is still one package, Go compiles all four files together, but each file is now independently readable.
Wire it together in main.go
// main.go
package main
import (
"context"
"fmt"
"os"
k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
"github.com/FerRiosCosta/go-k8s-health-reporter/internal/reporter"
)
func main() {
// Build the Kubernetes client using the kubeconfig minikube wrote in post 2.
// This is the same file kubectl uses, no extra setup needed.
kubeconfigPath := os.Getenv("HOME") + "/.kube/config"
kube, err := k8s.NewClient(kubeconfigPath)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to connect to cluster: %v\n", err)
os.Exit(1)
}
// Build the reporter with the client injected.
r := reporter.New(kube)
// Generate and print the report.
ctx := context.Background()
report, err := r.Generate(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to generate report: %v\n", err)
os.Exit(1)
}
r.Print(os.Stdout, report)
}
Verifying against minikube
The best way to verify the health reporter works is to run it against your real minikube cluster. No fakes, no mocks, just your actual cluster from post 2.
Healthy cluster
Make sure minikube is running and the go-api deployment from post 2 is up:
kubectl get pods
# NAME READY STATUS RESTARTS AGE
# go-api-7d6b9f8c4-xk2pq 1/1 Running 0 2d
# go-api-7d6b9f8c4-mn9rt 1/1 Running 0 2d
# go-api-7d6b9f8c4-p8wvz 1/1 Running 0 2d
Run the reporter:
go run ./main.go
Cluster Health Report
Generated: 2026-06-29 15:58:43 UTC
NAMESPACE TOTAL RUNNING FAILING PENDING
────────────────────────────────────────────────────
default 3 3 0 0
kube-system 8 8 0 0
kubernetes-dashboard 2 2 0 0
✓ All pods healthy
Simulating a failing pod
To verify the failing pod detection works we need a pod that reliably OOMKills. Our go-api binary turns out to be too lean to reliably trigger OOMKill at the Docker minimum, the multi-stage Dockerfile produced an extremely efficient binary. That's good engineering, but it makes for a tricky simulation.
Instead, let's deploy a pod that deliberately holds memory using Python's bytearray. Unlike shell tools like dd which stream data through memory and release it immediately, bytearray allocates a contiguous block and holds it, guaranteeing an OOMKill with a tight limit.
Create a file called memory-hog.yaml:
# memory-hog.yaml
apiVersion: v1
kind: Pod
metadata:
name: memory-hog
namespace: default
spec:
# restartPolicy: Always ensures Kubernetes keeps restarting after each
# OOMKill, driving the pod into CrashLoopBackOff.
restartPolicy: Always
containers:
- name: hog
image: python:3.12-alpine
command: ["python3", "-c"]
args:
- |
import time
# bytearray allocates 100MB and holds it in physical memory.
# The moment this runs, usage exceeds the 10Mi limit
# and the OOM killer fires immediately.
data = bytearray(100 * 1024 * 1024)
print("allocated 100MB, holding...", flush=True)
time.sleep(3600)
resources:
requests:
memory: "10Mi"
limits:
memory: "10Mi"
Apply it:
kubectl apply -f memory-hog.yaml
Watch the pod:
kubectl get pods -w
# NAME READY STATUS RESTARTS AGE
# memory-hog 0/1 ContainerCreating 0 1s
# memory-hog 0/1 OOMKilled 0 4s ← transient
# memory-hog 0/1 CrashLoopBackOff 1 10s ← stable, run the reporter now
Once you see CrashLoopBackOff, run the reporter:
go run ./main.go
Cluster Health Report
Generated: 2026-07-01 21:37:47 UTC
NAMESPACE TOTAL RUNNING FAILING PENDING
────────────────────────────────────────────────────
kubernetes-dashboard 2 2 0 0
default 4 3 1 0
kube-system 8 8 0 0
FAILING PODS
────────────────────────────────────────────────────
default memory-hog CrashLoopBackOff 13 restarts
The reporter correctly identifies the failing pod, the reason, and the restart count.
Clean up when done:
kubectl delete pod memory-hog
Simulating a pending pod
To verify pending detection, set a memory request higher than your node can provide. Again, both requests and limits must be set together:
kubectl set resources deployment/go-api \
--requests=memory=100Gi \
--limits=memory=100Gi \
-n default
The pods can't be scheduled, no node has 100Gi available. Run the reporter:
go run ./main.go
Cluster Health Report
Generated: 2026-07-01 22:03:51 UTC
NAMESPACE TOTAL RUNNING FAILING PENDING
────────────────────────────────────────────────────
default 4 3 0 1
kube-system 8 8 0 0
kubernetes-dashboard 2 2 0 0
PENDING PODS
────────────────────────────────────────────────────
default go-api-6ff6496cb5-dnq6m 0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
The reason message comes directly from the pod's scheduling condition, exactly what the pendingReason() function extracts. Restore the correct values when done:
kubectl set resources deployment/go-api \
--requests=memory=32Mi \
--limits=memory=64Mi \
-n default
Note on testing with fakes: In a future post we'll cover how to write unit tests for this reporter using interface fakes and client-go's built-in fake clientset, so the test suite runs without a real cluster. For now, minikube is our test environment.
Common gotchas
Wrong kubeconfig context
The most common mistake when starting out with client-go. If you have multiple clusters in your kubeconfig, always verify before running Go code:
kubectl config current-context
# Should print: minikube
If it shows a production cluster, switch back:
kubectl config use-context minikube
Passing "" vs "default" to namespace-scoped resources
// Returns pods from ALL namespaces
clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{})
// Returns pods from the "default" namespace only
clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})
These behave completely differently. The empty string means "all namespaces", not "the default namespace." This is one of the most common sources of confusion in client-go code.
Pod phase vs container state, the big one
A pod in Running phase can have containers in CrashLoopBackOff. The phase is the pod-level state; the container state is what's actually happening inside. Always check both:
// WRONG: misses CrashLoopBackOff pods
if pod.Status.Phase != corev1.PodRunning {
// handle failing pod
}
// RIGHT: catches CrashLoopBackOff even when phase is Running
for _, cs := range pod.Status.ContainerStatuses {
if cs.State.Waiting != nil && cs.State.Waiting.Reason == "CrashLoopBackOff" {
// handle failing container
}
}
Creating a new Clientset per function call
client-go's Clientset maintains an internal HTTP connection pool and rate limiter. Creating a new one on every call throws away the connection pool and can trigger API server rate limits. Always create the Clientset once at startup and reuse it:
// WRONG: new Clientset every call, no connection reuse
func listPods() {
clientset, _ := kubernetes.NewForConfig(config)
clientset.CoreV1().Pods("").List(...)
}
// RIGHT: Clientset created once, reused everywhere
var clientset kubernetes.Interface
func init() {
clientset, _ = kubernetes.NewForConfig(config)
}
func listPods() {
clientset.CoreV1().Pods("").List(...)
}
API version mismatches, always pin all three packages together
All three packages, k8s.io/api, k8s.io/apimachinery, and k8s.io/client-go, must be on the same minor version. They are developed and released together as part of each Kubernetes release cycle. Mixing versions causes compile errors because types from one package won't match types expected by another.
The version number maps directly to your Kubernetes cluster: v0.36.2 = Kubernetes v1.36.2. Since we're running Kubernetes 1.36.2 in minikube, v0.36.2 is the right choice throughout this series:
# WRONG: mixing minor versions causes type mismatch errors at compile time
go get k8s.io/[email protected]
go get k8s.io/[email protected] # different minor version — will break
go get k8s.io/[email protected] # different minor version — will break
# RIGHT: all three pinned to match Kubernetes 1.36.2
go get k8s.io/[email protected]
go get k8s.io/[email protected]
go get k8s.io/[email protected]
go mod tidy
Always verify your cluster version before pinning:
kubectl version
# Client Version: v1.36.2
# Server Version: v1.36.2
If your client and server versions differ, for example
Client Version: v1.36.2andServer Version: v1.35.1, that's completely fine. Kubernetes has an official version skew policy:kubectlcan be up to one minor version ahead or behind the server. Pin your client-go packages to match whichever version you're running in the cluster.
Summary
client-go is the foundation everything in the Kubernetes Go ecosystem is built on. Four packages, one Clientset, typed access to every resource the API server exposes.
Three things to take away:
The Clientset pattern:
clientset.APIGroup().Resource(namespace).Operation(), is consistent across every resource type. Learn it once, read any client-go code.Always check container states, not just pod phase, a "Running" pod can have failing containers.
Wrap the Clientset behind an interface from the start, it costs nothing and makes every function that uses it instantly testable.
What's next
The health reporter we built here is a real tool. Run it on any cluster, and it gives you immediate operational value. It's also the direct foundation for a new tool we are going to build in the later chapter:ferctl top: a full CLI with Cobra, flags, tabwriter output, and near-limit warnings. The health reporter becomes a subcommand. The KubeClient interface gets extended with metrics.
Every pattern, every import, and every interface in this post appears in the next post.
SOCIAL SHARE CARD GENERATOR