Skip to content
KubeClientlings

Catalog

All 52 exercises in one place — filter by topic or search. Click any row for the run command, the code and the hint; expand a topic header for what it teaches.

  • 52Exercises
  • 17Topics
  • 8Tiers
How exercises pass: Every exercise ships broken on purpose — it passes once it compiles, the linter is clean, and its checks against the local kind cluster succeed. Remove the I AM NOT DONE marker once it's fixed.
Topic Exercise Description
Setup setup1 Every client-go program starts the same way: find the kubeconfig, build a *rest.Config from it. kubectl — and every controller you will ever write — resolves credentials via the standard loading rules: the $KUBECONFIG env var first, then ~/.kube/config. Hard-coding a path breaks the moment the code runs on another machine. Make this program load the kubeconfig using the standard rules, pinned to the kind-kubeclientlings context.▸
Setup setup2 A *rest.Config is just credentials + an address. To actually talk to the API you wrap it in a clientset: one typed client per API group/version (CoreV1(), AppsV1(), ...). Two classic mistakes are made on this line: passing the wrong config, and throwing the error away. Create the clientset from the loaded config, handle the error, and list the cluster's nodes.▸
Setup setup3 The zero-value rest.Config works, but production clients always tune it: - QPS / Burst: client-side rate limits (default 5/10 — far too low for controllers; starved clients show mysterious multi-second latency) - Timeout: without it a hung API server hangs your program forever - UserAgent: shows up in API server audit logs, so operators can tell WHICH client is hammering the API From here on, exkit.MustRESTConfig() does the loading dance from setup1 for you. Tune the returned config so the checks below pass, then prove it still works with a real round-trip.▸
Pods pods1 The most fundamental client-go operation: create an object, read it back. Namespaced clients take the namespace as an ARGUMENT — Pods(ns) — and that argument, not the object's metadata, decides which URL the request hits. Create the pod in the exercise namespace, then Get it back.▸
Pods pods2 List() returns everything in the namespace unless you narrow it down. The server does the filtering — you express it in ListOptions with a label selector string, exactly the same syntax as `kubectl get -l`. List only the pods labeled app=web.▸
Pods pods3 Update() is optimistic concurrency in action: the object you send must carry the resourceVersion of the object you read. You never build an update by hand — you Get the current object, mutate it, and send it back. And because ANYONE can write between your Get and your Update (the kubelet writes pod status constantly), production code wraps the cycle in retry.RetryOnConflict. Add the label tier=frontend to the existing pod, the right way.▸
Pods pods4 Patch() changes just the fields you send — no read-modify-write, no resourceVersion conflicts. A strategic merge patch is plain JSON shaped like the object itself, and the patch TYPE argument tells the server how to merge it. Patch the label tier=frontend onto the pod.▸
Pods pods5 Delete() only STARTS a deletion — the object lingers while finalizers and the kubelet do their work. Code that needs the object gone must poll until Get returns a NotFound error. And NotFound here is not a failure: it is exactly the signal you are waiting for. apierrors.IsNotFound() is how you tell it apart from real errors. Delete the pod and wait until it is truly gone.▸
Deployments deploy1 A Deployment's selector must match its pod template's labels — that is the contract that ties the Deployment to the pods it owns. The API server enforces it: a mismatch is rejected with a 422 "selector does not match template labels" before anything is created. Fix the labels so the Deployment is accepted.▸
Deployments deploy2 Scaling is just updating .spec.replicas — but the field is a *int32, a pointer, so "unset" can be told apart from "zero". Go won't let you take a pointer to an untyped constant or assign *int where *int32 is wanted; this trips up everyone exactly once. Scale the deployment to 3 replicas.▸
Deployments deploy3 Patching a container image is the everyday "deploy a new version" call. containers is a LIST, and strategic merge patch needs the list's merge key — the container NAME — to know which element you mean. Omit it and the server rejects the patch. Patch the web container to run a newer nginx.▸
Deployments deploy4 "Is the rollout done?" is subtler than it looks. Status.Replicas counts pods that EXIST, not pods that are READY — and the status you read might describe an older generation of the spec. A correct rollout wait checks: - ObservedGeneration >= Generation (status describes MY change) - UpdatedReplicas == desired (all pods are on the new template) - ReadyReplicas == desired (and they all pass readiness) Wait for the rollout properly. The pods take ~5s to become ready (readiness probe delay), which is exactly the window where sloppy checks declare victory too early.▸
Services svc1 A Service finds its backends with a label selector. Nothing validates it at create time — a Service with a selector that matches no pods is perfectly legal and silently routes to nowhere. This is one of the most common "why is my service not working" bugs in real clusters. Fix the Service's selector so it actually selects the web pods.▸
Services svc2 Endpoints moved to the discovery.k8s.io/v1 API: EndpointSlices. You don't Get them by service name — you LIST them by the well-known label kubernetes.io/service-name (exported as discoveryv1.LabelServiceName). Guessing that label wrong returns an empty list, not an error. List the EndpointSlices that belong to the web service.▸
Options opt1 Building selector strings by hand gets error-prone fast. The labels package builds them programmatically: labels.Set is a map, and SelectorFromSet turns it into a selector that requires EVERY key=value in the set (logical AND). Select exactly the prod web pods.▸
Options opt2 Field selectors filter on OBJECT FIELDS instead of labels — but only on the handful of fields the API server indexes (metadata.name, metadata.namespace, status.phase, spec.nodeName, ...). Ask for anything else and the server rejects the whole request. Fetch just the pod named web-2 using a field selector.▸
Options opt3 Server-side apply (SSA) is the modern way to manage objects: you declare the fields you care about, the server merges, and every field remembers its OWNER (the field manager). That's why ApplyOptions.FieldManager is mandatory — apply without an identity is meaningless. Apply the pod with a field manager so the server knows who owns what.▸
Options opt4 ownerReferences are how Kubernetes garbage collection works: delete the owner and everything pointing at it gets cleaned up. A valid reference needs the owner's UID — the value the server assigned at creation — not just its name. This is exactly what controllers set on every object they create. Give the child a proper owner reference, then watch GC cascade.▸
Watch watch1 Watch() streams changes as they happen. Every event on the channel has a TYPE — Added, Modified, Deleted — and reacting to the wrong type is one of the quietest bugs in Kubernetes tooling: the code compiles, runs, and waits forever. Observe the pod being created and then deleted.▸
Watch watch2 Where a watch STARTS matters. ResourceVersion "0" replays the server's cached state first — every existing object arrives as a fresh Added event. The list-then-watch pattern avoids that: List, remember the list's resourceVersion, watch from exactly there. Only genuinely new changes flow. Start the watch from the right resourceVersion so only the new pod's Added event arrives.▸
Watch watch3 Raw watches die: API servers close them, networks drop them, timeouts fire. RetryWatcher re-establishes the stream automatically — but it refuses to guess where to start. It demands a real resourceVersion ("" and "0" are rejected), which forces you into the correct list-then-watch pattern. Give the RetryWatcher a real starting point.▸
Watch watch4 Events — the things `kubectl describe` shows under "Events:" — don't appear by magic. A controller wires up an EventBroadcaster, points it at the API server with StartRecordingToSink, and gets an EventRecorder to emit through. Skip the sink and every event vanishes into the void: no errors, nothing stored. Wire the broadcaster to the API server so the event actually lands.▸
Dynamic dyn1 The dynamic client works on ANY resource — no typed structs, just unstructured maps. The price: you must name the resource yourself with a GroupVersionResource, and the resource is the lowercase PLURAL from the URL path ("pods"), never the Kind ("Pod"). The server has no idea what /api/v1/namespaces/x/Pod is. List the pods through the dynamic client with the right GVR.▸
Dynamic dyn2 Unstructured objects are map[string]any all the way down. Digging fields out by hand means type assertions at every level — the unstructured package's Nested* helpers do it safely: give the FULL path, get back (value, found, err). found=false is not an error; it just means nothing lives at that path. Read the ConfigMap value through the correct nested path.▸
Dynamic dyn3 CRDs extend the API at runtime — and the dynamic client is how you talk to them without generated code. One catch: a custom resource must be created with a GVR the CRD actually SERVES. The version in the GVR must be one of the CRD's spec.versions with served=true, or the server 404s as if the resource didn't exist at all. Create the Widget with the version the CRD serves.▸
Dynamic dyn4 How does kubectl know what resources exist? It ASKS — the discovery API lists every group/version the server serves and every resource in them. But you must ask for a groupVersion that exists: apps/v1beta1 died in Kubernetes 1.16, and asking for it is an error, not an empty list. Discover where deployments live today.▸
CRDs crd1 The dynamic client hands you unstructured maps, but your code wants a real Go struct. runtime.DefaultUnstructuredConverter bridges the two — and it maps fields by their `json` tags, exactly like encoding/json. A tag that does not match the key in the unstructured object leaves that field zero. Read a custom resource's spec into a typed struct.▸
CRDs crd2 Going the other way — typed struct to unstructured — is how you hand a Go value to the dynamic client. The catch: the dynamic client routes the request by apiVersion and kind, and those come from the embedded TypeMeta. Forget to set TypeMeta and the resulting object has no kind, so the client has no idea what endpoint to send it to. Convert a typed Widget into an unstructured object the dynamic client can send.▸
Informers inf1 Informers replace polling: List once, Watch forever, keep a local cache you read for free. But the cache starts EMPTY — Start() only launches the machinery in the background. Read before the initial sync finishes and you get whatever happens to be there: usually nothing. Wait for the cache to sync before reading from the lister.▸
Informers inf2 Informers PUSH: AddEventHandler registers callbacks for every Add, Update and Delete the watch sees. But registering a handler wires nothing up by itself — until the factory is STARTED no goroutine runs, no watch opens, and no handler ever fires. Forgetting Start() is the classic silent informer bug. Start the factory so the handler sees the pod appear.▸
Informers inf3 Objects in an informer cache are indexed by KEY, and for namespaced objects the key is "namespace/name" — that's what MetaNamespaceKeyFunc produces and what every controller's workqueue carries around. Ask the store for a bare name and it politely finds nothing: no error, just exists=false. Look the pod up by its real cache key.▸
Informers inf4 A factory built with no options watches the WHOLE cluster — kube-system, other tenants, everything. That's cache memory and API load you pay for objects you will never touch. WithNamespace scopes the underlying List+Watch so only your namespace ever crosses the wire. Scope the informer so the cache holds exactly this exercise's pods.▸
Workqueue wq1 A rate-limiting queue tracks how many times each key has been requeued, so repeated failures back off. Two different calls "release" a key, and they are NOT the same: Done — "I stopped processing this item" (from Get/Done). Says nothing about retries. Forget — "clear this key's retry history" so its next failure starts from the short base delay again, not a long one. A successful reconcile must reset the backoff so the next failure starts fresh.▸
Workqueue wq2 Why do controllers back off exponentially? Because a resource that fails once often keeps failing, and hammering the API server helps no one. The exponential rate limiter answers When(key) with a delay that DOUBLES on every failure — base, 2·base, 4·base … capped at a max. Pick the rate limiter whose delays actually grow, and watch them.▸
Workqueue wq3 A poison item — one that fails no matter what — must not requeue forever. Every real controller has a handleErr that caps retries: requeue with backoff while NumRequeues is under a limit, and once it hits the limit, Forget the key and drop it so it stops consuming the queue. Implement the retry cap so a permanently failing key is eventually dropped.▸
Controllers ctrl1 The workqueue is the beating heart of every controller. It has a contract: Get hands you an item and marks it PROCESSING; you MUST call Done when finished. If the same key is Added while you're processing it, the queue holds it back — and re-delivers only after your Done. Skip Done and that key is stuck forever: Get never returns it again. Honor the Get/Done contract so the requeued item comes back.▸
Controllers ctrl2 This is a whole controller in one file: informer → workqueue → worker. The glue is the KEY. Handlers push "namespace/name" strings (never whole objects — the cache dedupes by key), and the worker splits the key back apart to act. Push the wrong string and the worker reconciles a pod that doesn't exist. Enqueue proper cache keys so the worker can find the pod.▸
Controllers ctrl3 Run two replicas of a controller and they fight — unless exactly one leads. Leader election settles it with a Lease object. The timings form a contract the library ENFORCES: LeaseDuration (how long a lost leader's lease lingers) must be longer than RenewDeadline (how long the leader keeps trying to renew), which must be longer than RetryPeriod. Fix the timing contract so the candidate can run for office.▸
Server-Side Apply ssa1 Server-Side Apply lets you declare the state you want and let the apiserver merge it — but it tracks WHO owns each field, so every apply must name a field manager. Leave it empty and the apiserver rejects the request: it has nowhere to record ownership. Apply a ConfigMap with a field manager.▸
Server-Side Apply ssa2 Two managers, one field: when a second manager applies a value to a field the first already owns, the apiserver returns a conflict — that is SSA protecting you from silent clobbering. To deliberately take ownership, you set Force. It resolves the conflict and moves the field into your name. Take over a field a different manager owns.▸
Subresources sub1 Replicas live behind the /scale subresource, not the main object. That is why `kubectl scale` and an HPA can resize a Deployment without touching its spec: they GET the Scale, set Scale.Spec.Replicas — the DESIRED count — and UpdateScale. Scale.Status.Replicas is the observed count and is read-only on write; setting it does nothing. Resize a Deployment through its scale subresource.▸
Subresources sub2 When a resource has the status subresource enabled, status becomes a separate write path. The main endpoint IGNORES any change to status, and only UpdateStatus (the /status subresource) can write it. This split is what stops a controller writing status and a user writing spec from clobbering each other. A custom resource is perfect for seeing this cleanly — nothing else reconciles it. Write the Widget's status through the right endpoint.▸
Finalizers fin1 A finalizer is how an operator says "wait — I have cleanup to do before this object goes away." While a finalizer is present, a Delete does NOT remove the object: the apiserver just stamps it with a DeletionTimestamp and leaves it, terminating, until the finalizer is cleared. Protect an object from deletion with a finalizer.▸
Finalizers fin2 The other half of the finalizer contract: once your cleanup is done, you remove your finalizer, and only then does the apiserver actually delete the object. A terminating object with a non-empty finalizer list lingers forever — clearing the list is what lets it go. Release a terminating object by removing its finalizer.▸
Webhooks wh1 A validating webhook is a plain function: apiserver sends an AdmissionReview describing the object under review, your handler returns an AdmissionResponse that allows or denies it. Two rules that trip everyone up: you MUST echo the request UID back, and Allowed defaults to false — so an "allow" path has to set it explicitly. This handler must deny any pod that lacks a "team" label. Test it directly — no cluster, no TLS, just the function.▸
Webhooks wh2 A mutating webhook changes the object by returning a JSON patch. The gotcha is invisible: the AdmissionResponse must announce its PatchType, or the apiserver silently ignores the patch bytes — no error, the mutation just never happens. A JSON patch always travels with PatchType = JSONPatch. Build a response that injects a default label, and declare its patch type.▸
Testing test1 You do not need a cluster to test client-go code — the fake clientset implements the same typed interface backed by an in-memory object tracker. Seed it with objects and every Get/List/Update works against that tracker. The one rule: the tracker keys objects by namespace AND name, exactly as seeded. Query the wrong namespace and you get an ordinary NotFound.▸
Testing test2 Real code must survive API failures — but you cannot make a live server return "etcd unavailable" on demand. Reactors can: a reactor intercepts matching calls on the fake client and returns whatever you want. That is how you test the sad path. A reactor matches on (verb, resource). PrependReactor puts it at the front of the chain so it fires first, and returning handled=true short-circuits the default tracker.▸
Testing test3 The fake client records every call it receives. Actions() returns them in order, and each action knows its verb and resource (Matches) and — for writes — carries the object that was sent. This is how you assert that a controller did the right WRITE, not just that it did not error. Inspect the recorded action to prove what was created.▸
Operator op1 Your first controller-runtime Reconciler. The manager owns all the machinery you built by hand in ctrl2 (informer → queue → worker); you only write Reconcile. The contract: a Request names one object, you return a Result plus error. NotFound is NORMAL — the object was deleted and the cache already forgot it. Returning an error there makes the manager retry a reconcile that can never succeed. Fetch the object the request names, and treat NotFound as "done".▸
Operator op2 Owns() is how a controller notices its children. It maps an event on a child object back to a reconcile request for the OWNER — by reading the child's controller ownerRef. No ownerRef, no mapping: delete the child and nobody hears it. SetControllerReference is what stamps that ref. Stamp the parent's controller reference on the child before creating it.▸
Operator op3 controllerutil.CreateOrUpdate is the reconciler's read-modify-write: it GETS the live object into your struct, runs your mutate() to lay desired state over it, then Creates or Updates only if something changed. Desired state set OUTSIDE mutate() is wiped by that Get — creation works once, but the update path becomes a no-op and drift is never repaired. Move the desired state (and the ownerRef) into the mutate function.▸

▶ Run it

From the repo root: — or mise run watch to work through everything in order with instant feedback.