Catalog

All 56 exercises in one place — filter by topic or search. Click any row for the run command, the code and the hint; expand a topic header for what it teaches.

56Exercises
17Topics
8Tiers

How exercises pass: Every exercise ships broken on purpose — it passes once it compiles, the linter is clean, and its checks against the local kind cluster succeed. Remove the I AM NOT DONE marker once it's fixed.

Topic	Exercise	Description
Setup Beginner · Setup 3 exercises
Setup	Chapter	Every client-go program — a one-off script, a `kubectl` plugin, a controller that runs forever — begins the same way: find credentials, turn them into a `rest.Config`, and wrap that in a client. Nothing above this layer works until this layer is right, and when it is wrong the failure almost never says so. An empty config builds a perfectly valid clientset; you find out on the first request, as a dial error against an empty URL. The chain is three links: loading rules* decide which files to read, `*rest.Config` is the credential bundle they collapse into, and `kubernetes.NewForConfig` turns that bundle into typed clients that share one HTTP transport. This chapter follows the chain, then tunes it for a process that has to survive contact with a real cluster.▸
Setup	setup1	Every client-go program starts the same way: find the kubeconfig, build a *rest.Config from it. kubectl — and every controller you will ever write — resolves credentials via the standard loading rules: the $KUBECONFIG env var first, then ~/.kube/config. Hard-coding a path breaks the moment the code runs on another machine. Make this program load the kubeconfig using the standard rules, pinned to the kind-kubeclientlings context.▸
Setup	setup2	A *rest.Config is just credentials + an address. To actually talk to the API you wrap it in a clientset: one typed client per API group/version (CoreV1(), AppsV1(), ...). Two classic mistakes are made on this line: passing the wrong config, and throwing the error away. Create the clientset from the loaded config, handle the error, and list the cluster's nodes.▸
Setup	setup3	The zero-value rest.Config works, but production clients always tune it: - QPS / Burst: client-side rate limits (default 5/10 — far too low for controllers; starved clients show mysterious multi-second latency) - Timeout: without it a hung API server hangs your program forever - UserAgent: shows up in API server audit logs, so operators can tell WHICH client is hammering the API From here on, exkit.MustRESTConfig() does the loading dance from setup1 for you. Tune the returned config so the checks below pass, then prove it still works with a real round-trip.▸
Pods Beginner · Core Workloads 7 exercises
Pods	Chapter	Setup ended with a clientset. This topic is what you do with one: the full write path against the core/v1 API, using the resource everybody already understands. Nothing here is Pod-specific — Create, Get, List, Update, Patch and Delete behave identically for Deployments, ConfigMaps and your own CRDs. Learn the shapes once. The typed client is a thin, generated wrapper over a URL. `cs.CoreV1()` picks the group and version, `.Pods(ns)` picks the resource and namespace, and together they build `/api/v1/namespaces/<ns>/pods`. Every method on that interface is one HTTP verb against that path. Once you can see the URL behind the call, most of client-go's surprises stop being surprising.▸
Pods	pods1	The most fundamental client-go operation: create an object, read it back. Namespaced clients take the namespace as an ARGUMENT — Pods(ns) — and that argument, not the object's metadata, decides which URL the request hits. Create the pod in the exercise namespace, then Get it back.▸
Pods	pods2	List() returns everything in the namespace unless you narrow it down. The server does the filtering — you express it in ListOptions with a label selector string, exactly the same syntax as `kubectl get -l`. List only the pods labeled app=web.▸
Pods	pods3	Update() is optimistic concurrency in action: the object you send must carry the resourceVersion of the object you read. You never build an update by hand — you Get the current object, mutate it, and send it back. And because ANYONE can write between your Get and your Update (the kubelet writes pod status constantly), production code wraps the cycle in retry.RetryOnConflict. Add the label tier=frontend to the existing pod, the right way.▸
Pods	pods4	Patch() changes just the fields you send — no read-modify-write, no resourceVersion conflicts. A strategic merge patch is plain JSON shaped like the object itself, and the patch TYPE argument tells the server how to merge it. Patch the label tier=frontend onto the pod.▸
Pods	pods5	Delete() only STARTS a deletion — the object lingers while finalizers and the kubelet do their work. Code that needs the object gone must poll until Get returns a NotFound error. And NotFound here is not a failure: it is exactly the signal you are waiting for. apierrors.IsNotFound() is how you tell it apart from real errors. Delete the pod and wait until it is truly gone.▸
Pods	pods6	Reading logs is not an object Get — there is no "Log" resource. GetLogs returns a *rest.Request you have to STREAM, because a log is an open pipe, not a value. And when a pod has more than one container the apiserver refuses to guess which one you meant: PodLogOptions.Container is mandatory the moment there is more than one. Read the marker line out of the writer container's log.▸
Pods	pods7	`kubectl exec` is not an API call you can make with the typed clientset — it is a protocol UPGRADE. You hand-build a request to the pod's exec subresource, then hand its URL to an executor that switches the connection to SPDY and multiplexes stdin/stdout/stderr over it. The trap: PodExecOptions and StreamOptions BOTH mention stdout, and they mean different things. StreamOptions says where to put the bytes locally; PodExecOptions asks the apiserver to send them at all. You need both. Capture the command's stdout.▸
Deployments Beginner · Core Workloads 4 exercises
Deployments	Chapter	A Pod is a leaf. A Deployment is the first object in this course that is managed by a controller, and that changes what you have to understand. Nothing you write to a Deployment happens immediately: you record an intention in `spec`, a controller acts on it, and it reports back in `status`. Reading that loop correctly — knowing which of the four replica counters answers your question, and when `status` is even describing the spec you just wrote — is most of what this topic is about. The other half is structural: a Deployment does not hold references to its pods. It finds them by label query, which makes labels a contract rather than decoration, and makes one particular mistake unfixable after creation.▸
Deployments	deploy1	A Deployment's selector must match its pod template's labels — that is the contract that ties the Deployment to the pods it owns. The API server enforces it: a mismatch is rejected with a 422 "selector does not match template labels" before anything is created. Fix the labels so the Deployment is accepted.▸
Deployments	deploy2	Scaling is just updating .spec.replicas — but the field is a int32, a pointer, so "unset" can be told apart from "zero". Go won't let you take a pointer to an untyped constant or assign int where *int32 is wanted; this trips up everyone exactly once. Scale the deployment to 3 replicas.▸
Deployments	deploy3	Patching a container image is the everyday "deploy a new version" call. containers is a LIST, and strategic merge patch needs the list's merge key — the container NAME — to know which element you mean. Omit it and the server rejects the patch. Patch the web container to run a newer nginx.▸
Deployments	deploy4	"Is the rollout done?" is subtler than it looks. Status.Replicas counts pods that EXIST, not pods that are READY — and the status you read might describe an older generation of the spec. A correct rollout wait checks: - ObservedGeneration >= Generation (status describes MY change) - UpdatedReplicas == desired (all pods are on the new template) - ReadyReplicas == desired (and they all pass readiness) Wait for the rollout properly. The pods take ~5s to become ready (readiness probe delay), which is exactly the window where sloppy checks declare victory too early.▸
Services Beginner · Core Workloads 2 exercises
Services	Chapter	Deployments showed a controller acting on a label selector. A Service is the same idea pointed at the network — and it is the place where Kubernetes' habit of failing silently is most expensive. A Service whose selector matches nothing is a completely legal object. It gets a ClusterIP, DNS resolves it, and every connection to it fails. Nothing warns you. So this topic is about not trusting the create. You write the selector, then you go and read what the control plane produced from it: the EndpointSlices that carry the actual pod IPs. Those slices are not named after the Service and there is more than one of them, which means finding them takes a label query of its own.▸
Services	svc1	A Service finds its backends with a label selector. Nothing validates it at create time — a Service with a selector that matches no pods is perfectly legal and silently routes to nowhere. This is one of the most common "why is my service not working" bugs in real clusters. Fix the Service's selector so it actually selects the web pods.▸
Services	svc2	Endpoints moved to the discovery.k8s.io/v1 API: EndpointSlices. You don't Get them by service name — you LIST them by the well-known label kubernetes.io/service-name (exported as discoveryv1.LabelServiceName). Guessing that label wrong returns an empty list, not an error. List the EndpointSlices that belong to the web service.▸
Options Intermediate · Requests & Watch 5 exercises
Options	Chapter	Every client-go call ends in an options struct, and until now they have mostly been empty. `ListOptions`, `ApplyOptions`, `DeleteOptions` are where you shape the request: what to filter on, who you are claiming to be, how much to ask for at a time. Getting them right is the difference between a controller that a cluster barely notices and one that operators come looking for. Two ideas run through the topic. The first is that filtering happens on two independent axes — labels and fields — that look interchangeable and behave nothing alike, including on typos. The second is identity: server-side apply needs to know who owns each field, and the garbage collector needs to know exactly which instance owns each object. Both are answered with values you can only get from the server.▸
Options	opt1	Building selector strings by hand gets error-prone fast. The labels package builds them programmatically: labels.Set is a map, and SelectorFromSet turns it into a selector that requires EVERY key=value in the set (logical AND). Select exactly the prod web pods.▸
Options	opt2	Field selectors filter on OBJECT FIELDS instead of labels — but only on the handful of fields the API server indexes (metadata.name, metadata.namespace, status.phase, spec.nodeName, ...). Ask for anything else and the server rejects the whole request. Fetch just the pod named web-2 using a field selector.▸
Options	opt3	Server-side apply (SSA) is the modern way to manage objects: you declare the fields you care about, the server merges, and every field remembers its OWNER (the field manager). That's why ApplyOptions.FieldManager is mandatory — apply without an identity is meaningless. Apply the pod with a field manager so the server knows who owns what.▸
Options	opt4	ownerReferences are how Kubernetes garbage collection works: delete the owner and everything pointing at it gets cleaned up. A valid reference needs the owner's UID — the value the server assigned at creation — not just its name. This is exactly what controllers set on every object they create. Give the child a proper owner reference, then watch GC cascade.▸
Options	opt5	A List with no Limit fetches EVERYTHING in one response. On a namespace with 200k pods that is a multi-hundred-megabyte body the apiserver has to hold in memory, serialize and send — the single easiest way for a client to hurt a cluster. Limit asks for one page; the response comes back with a CONTINUE token, and you keep passing it back until it is empty. Page through all the configmaps instead of demanding them in one gulp.▸
Watch Intermediate · Requests & Watch 4 exercises
Watch	Chapter	Everything so far has been a request and a reply. Watch is the other shape: a long-lived stream of changes, and the primitive under every controller in Kubernetes. It is also where the API stops being forgiving. A watch is expected to end — the API server kills each one on a timer by design — and starting it at the wrong point in history silently gives you either duplicates or a hole. The correct pattern has a name, list-then-watch, and it exists because a `List` response tells you the exact point in history it represents. Get that handoff right and you have a coherent view of the cluster; get it wrong and you have a controller acting on a world that never existed. This chapter builds the pattern up, then layers on reconnection, then ends with how a controller tells humans what it did.▸
Watch	watch1	Watch() streams changes as they happen. Every event on the channel has a TYPE — Added, Modified, Deleted — and reacting to the wrong type is one of the quietest bugs in Kubernetes tooling: the code compiles, runs, and waits forever. Observe the pod being created and then deleted.▸
Watch	watch2	Where a watch STARTS matters. ResourceVersion "0" replays the server's cached state first — every existing object arrives as a fresh Added event. The list-then-watch pattern avoids that: List, remember the list's resourceVersion, watch from exactly there. Only genuinely new changes flow. Start the watch from the right resourceVersion so only the new pod's Added event arrives.▸
Watch	watch3	Raw watches die: API servers close them, networks drop them, timeouts fire. RetryWatcher re-establishes the stream automatically — but it refuses to guess where to start. It demands a real resourceVersion ("" and "0" are rejected), which forces you into the correct list-then-watch pattern. Give the RetryWatcher a real starting point.▸
Watch	watch4	Events — the things `kubectl describe` shows under "Events:" — don't appear by magic. A controller wires up an EventBroadcaster, points it at the API server with StartRecordingToSink, and gets an EventRecorder to emit through. Skip the sink and every event vanishes into the void: no errors, nothing stored. Wire the broadcaster to the API server so the event actually lands.▸
Dynamic Intermediate · Dynamic & CRDs 4 exercises
Dynamic	Chapter	The typed clientset knows every path at compile time, which is exactly why it cannot talk to a type invented after it was built. The dynamic client knows nothing and asks you for everything: you name the endpoint, you get back a `map[string]any`, and nothing about your type exists in Go at all. That is the price of working with CRDs, with objects whose type is only known at runtime, and with anything that has to operate on resources generically — which is to say, most of the interesting tooling in the ecosystem, `kubectl` included. Two vocabularies collide here and confusing them produces 404s on URLs nobody serves. Kind names objects; Resource names URLs. This chapter starts there, then covers reading unstructured objects without panicking, defining a CRD and talking to it, and finally asking the server what it actually serves instead of guessing.▸
Dynamic	dyn1	The dynamic client works on ANY resource — no typed structs, just unstructured maps. The price: you must name the resource yourself with a GroupVersionResource, and the resource is the lowercase PLURAL from the URL path ("pods"), never the Kind ("Pod"). The server has no idea what /api/v1/namespaces/x/Pod is. List the pods through the dynamic client with the right GVR.▸
Dynamic	dyn2	Unstructured objects are map[string]any all the way down. Digging fields out by hand means type assertions at every level — the unstructured package's Nested* helpers do it safely: give the FULL path, get back (value, found, err). found=false is not an error; it just means nothing lives at that path. Read the ConfigMap value through the correct nested path.▸
Dynamic	dyn3	CRDs extend the API at runtime — and the dynamic client is how you talk to them without generated code. One catch: a custom resource must be created with a GVR the CRD actually SERVES. The version in the GVR must be one of the CRD's spec.versions with served=true, or the server 404s as if the resource didn't exist at all. Create the Widget with the version the CRD serves.▸
Dynamic	dyn4	How does kubectl know what resources exist? It ASKS — the discovery API lists every group/version the server serves and every resource in them. But you must ask for a groupVersion that exists: apps/v1beta1 died in Kubernetes 1.16, and asking for it is an error, not an empty list. Discover where deployments live today.▸
CRDs Intermediate · Dynamic & CRDs 2 exercises
CRDs	Chapter	Dynamic left you holding a `map[string]any`. That is fine at the edges of a program and miserable in the middle of one: no field names, no compiler, no completion, and `int64` assertions everywhere. The usual answer is code generation, which means a build step and generated clients per CRD. This topic is the pragmatic middle ground — `runtime.DefaultUnstructuredConverter`, which moves between unstructured maps and ordinary Go structs in both directions, driven entirely by `json` struct tags. The two directions have opposite failure characters, and that asymmetry is the whole lesson. Reading in, a mismatched tag is silent — you get a zero value and no error. Writing out, an unrepresentable type is loud. Meanwhile the field that decides whether the object is routable at all, `TypeMeta`, is one the typed clientset fills in for you and the dynamic client never does. These exercises need no cluster: they are pure conversion.▸
CRDs	crd1	The dynamic client hands you unstructured maps, but your code wants a real Go struct. runtime.DefaultUnstructuredConverter bridges the two — and it maps fields by their `json` tags, exactly like encoding/json. A tag that does not match the key in the unstructured object leaves that field zero. Read a custom resource's spec into a typed struct.▸
CRDs	crd2	Going the other way — typed struct to unstructured — is how you hand a Go value to the dynamic client. The catch: the dynamic client routes the request by apiVersion and kind, and those come from the embedded TypeMeta. Forget to set TypeMeta and the resulting object has no kind, so the client has no idea what endpoint to send it to. Convert a typed Widget into an unstructured object the dynamic client can send.▸
Informers Advanced · Controller Machinery 5 exercises
Informers	Chapter	Watch ended with an admission: a raw watch is correct only if you also implement relisting, reconnection and 410 recovery, and at that point you have written a `Reflector`. An informer is that `Reflector` plus a local cache plus a fan-out of event handlers — the layer every controller in Kubernetes is actually built on, including the ones in the control plane. The payoff is that a controller reconciling thousands of times a second never touches the API server to read. The cost is a lifecycle with several ordering rules, and almost every way of getting it wrong produces no error at all: the program compiles, runs, and blocks forever on a channel nobody will send to. This chapter is mostly about those orderings, and about the two rules — never mutate a cached object, never do I/O in a handler — that keep the shared machinery safe.▸
Informers	inf1	Informers replace polling: List once, Watch forever, keep a local cache you read for free. But the cache starts EMPTY — Start() only launches the machinery in the background. Read before the initial sync finishes and you get whatever happens to be there: usually nothing. Wait for the cache to sync before reading from the lister.▸
Informers	inf2	Informers PUSH: AddEventHandler registers callbacks for every Add, Update and Delete the watch sees. But registering a handler wires nothing up by itself — until the factory is STARTED no goroutine runs, no watch opens, and no handler ever fires. Forgetting Start() is the classic silent informer bug. Start the factory so the handler sees the pod appear.▸
Informers	inf3	Objects in an informer cache are indexed by KEY, and for namespaced objects the key is "namespace/name" — that's what MetaNamespaceKeyFunc produces and what every controller's workqueue carries around. Ask the store for a bare name and it politely finds nothing: no error, just exists=false. Look the pod up by its real cache key.▸
Informers	inf4	A factory built with no options watches the WHOLE cluster — kube-system, other tenants, everything. That's cache memory and API load you pay for objects you will never touch. WithNamespace scopes the underlying List+Watch so only your namespace ever crosses the wire. Scope the informer so the cache holds exactly this exercise's pods.▸
Informers	inf5	A lister can answer "everything" and "everything matching a label", but both walk the whole cache. When a controller needs "every pod for THIS owner" on every reconcile, that linear scan is the hot path. Indexers fix it: you register a function that maps an object to the keys it should file under, and the cache maintains a reverse map you can hit directly. The catch: an index is only as good as the key its function returns. Return the wrong string and ByIndex finds nothing — no error, just an empty slice. Index the pods by their tier label.▸
Workqueue Advanced · Controller Machinery 3 exercises
Workqueue	Chapter	Informers ended with a rule: handlers enqueue a key and return. This is the queue on the other end of that rule, and it is the piece that decides how a controller behaves when things go wrong — which, in a distributed system, is most of the time. The workqueue is deliberately more than a channel. It deduplicates, so a hot object updated ten times before anyone looks costs one reconcile. It guarantees single-flight per key, so no two workers ever reconcile the same object concurrently. And it carries a per-key retry history, so a wedged resource backs off without slowing down anything else. Those last two are tracked separately and released by two different calls, `Done` and `Forget`. Confusing them is the defining bug of this topic: nothing errors, and your controller just gets mysteriously slower, or silently stops reconciling one object forever.▸
Workqueue	wq1	A rate-limiting queue tracks how many times each key has been requeued, so repeated failures back off. Two different calls "release" a key, and they are NOT the same: Done — "I stopped processing this item" (from Get/Done). Says nothing about retries. Forget — "clear this key's retry history" so its next failure starts from the short base delay again, not a long one. A successful reconcile must reset the backoff so the next failure starts fresh.▸
Workqueue	wq2	Why do controllers back off exponentially? Because a resource that fails once often keeps failing, and hammering the API server helps no one. The exponential rate limiter answers When(key) with a delay that DOUBLES on every failure — base, 2·base, 4·base … capped at a max. Pick the rate limiter whose delays actually grow, and watch them.▸
Workqueue	wq3	A poison item — one that fails no matter what — must not requeue forever. Every real controller has a handleErr that caps retries: requeue with backoff while NumRequeues is under a limit, and once it hits the limit, Forget the key and drop it so it stops consuming the queue. Implement the retry cap so a permanently failing key is eventually dropped.▸
Controllers Advanced · Controller Machinery 3 exercises
Controllers	Chapter	Everything so far has been a part. Informers cached the world, workqueue handled failure, watch delivered changes. This topic bolts them together into the thing they were always for: a process that observes desired state, compares it with actual state, and acts — forever, idempotently, and without ever assuming it saw every event. The skeleton is small and always the same: informer → key → queue → worker. What makes it correct is not the wiring but two commitments. Enqueue a key, never an object, so the worker acts on what is true now rather than on a snapshot. And run exactly one instance that acts, however many replicas exist, which is what leader election is for — and where three duration knobs have to satisfy inequalities that the library will refuse to build past.▸
Controllers	ctrl1	The workqueue is the beating heart of every controller. It has a contract: Get hands you an item and marks it PROCESSING; you MUST call Done when finished. If the same key is Added while you're processing it, the queue holds it back — and re-delivers only after your Done. Skip Done and that key is stuck forever: Get never returns it again. Honor the Get/Done contract so the requeued item comes back.▸
Controllers	ctrl2	This is a whole controller in one file: informer → workqueue → worker. The glue is the KEY. Handlers push "namespace/name" strings (never whole objects — the cache dedupes by key), and the worker splits the key back apart to act. Push the wrong string and the worker reconciles a pod that doesn't exist. Enqueue proper cache keys so the worker can find the pod.▸
Controllers	ctrl3	Run two replicas of a controller and they fight — unless exactly one leads. Leader election settles it with a Lease object. The timings form a contract the library ENFORCES: LeaseDuration (how long a lost leader's lease lingers) must be longer than RenewDeadline (how long the leader keeps trying to renew), which must be longer than RetryPeriod. Fix the timing contract so the candidate can run for office.▸
Server-Side Apply Advanced · Advanced Writes 2 exercises
Server-Side Apply	Chapter	Every write so far has been imperative in shape: read the object, mutate it, send it back, and hope nobody else was writing at the same time. That works until two controllers care about one object — and then it fails the worst way Kubernetes fails, which is silently. Each overwrites the other every few seconds, and the only symptom is a resource that flaps. Server-Side Apply reverses the flow. You send only the fields you intend to own, the API server merges them into whatever is there, and it records in `metadata.managedFields` which manager set which field. That ledger is the whole feature. It makes multi-writer objects safe, it turns a silent fight into a 409 that names your opponent, and it is the one merge mechanism that works on CRDs — because it reads list-merge semantics from the OpenAPI schema rather than from Go struct tags.▸
Server-Side Apply	ssa1	Server-Side Apply lets you declare the state you want and let the apiserver merge it — but it tracks WHO owns each field, so every apply must name a field manager. Leave it empty and the apiserver rejects the request: it has nowhere to record ownership. Apply a ConfigMap with a field manager.▸
Server-Side Apply	ssa2	Two managers, one field: when a second manager applies a value to a field the first already owns, the apiserver returns a conflict — that is SSA protecting you from silent clobbering. To deliberately take ownership, you set Force. It resolves the conflict and moves the field into your name. Take over a field a different manager owns.▸
Subresources Advanced · Advanced Writes 2 exercises
Subresources	Chapter	An object looks like one document, and for writing purposes it often is not. Some fields live behind their own URL, with their own type and their own RBAC verb — and once a field lives there, the main endpoint stops accepting writes to it. Not with an error: it takes your request, returns 200, and silently drops that half of the object. That silence is the entire hazard of this topic. It is also, once you see why, the mechanism that makes controllers work at all: it is what keeps a user editing `spec` and a controller writing `status` from clobbering each other, and what makes `metadata.generation` mean something.▸
Subresources	sub1	Replicas live behind the /scale subresource, not the main object. That is why `kubectl scale` and an HPA can resize a Deployment without touching its spec: they GET the Scale, set Scale.Spec.Replicas — the DESIRED count — and UpdateScale. Scale.Status.Replicas is the observed count and is read-only on write; setting it does nothing. Resize a Deployment through its scale subresource.▸
Subresources	sub2	When a resource has the status subresource enabled, status becomes a separate write path. The main endpoint IGNORES any change to status, and only UpdateStatus (the /status subresource) can write it. This split is what stops a controller writing status and a user writing spec from clobbering each other. A custom resource is perfect for seeing this cleanly — nothing else reconciles it. Write the Widget's status through the right endpoint.▸
Finalizers Advanced · Advanced Writes 2 exercises
Finalizers	Chapter	An operator that allocates something outside the cluster — a cloud load balancer, a database, a DNS record — has a problem at delete time. The moment a user deletes the Kubernetes object, the record of what to clean up disappears with it, and the external resource leaks with nothing left to point at it. A finalizer is the hook that fixes that, and its mechanism is almost absurdly simple: `metadata.finalizers` is a list of opaque strings, and the API server's only rule is if the list is non-empty, do not remove the object. Delete then stops meaning "delete" and starts meaning "mark terminating". The object stays readable while controllers do their cleanup, and it vanishes when the last string is removed. The same simplicity is why objects get stuck terminating forever, which is one of the most common Kubernetes support questions there is. Nothing in the cluster will ever remove a finalizer whose controller no longer exists.▸
Finalizers	fin1	A finalizer is how an operator says "wait — I have cleanup to do before this object goes away." While a finalizer is present, a Delete does NOT remove the object: the apiserver just stamps it with a DeletionTimestamp and leaves it, terminating, until the finalizer is cleared. Protect an object from deletion with a finalizer.▸
Finalizers	fin2	The other half of the finalizer contract: once your cleanup is done, you remove your finalizer, and only then does the apiserver actually delete the object. A terminating object with a non-empty finalizer list lingers forever — clearing the list is what lets it go. Release a terminating object by removing its finalizer.▸
Webhooks Advanced · Webhooks & Testing 2 exercises
Webhooks	Chapter	Admission control looks like the most intimidating extension point in Kubernetes — TLS certificates, CA bundles, a `ValidatingWebhookConfiguration` nobody enjoys writing. But the part you actually write is one pure function: `AdmissionReview` in, `AdmissionResponse` out. No cluster, no state, fully unit testable, which is exactly how these exercises treat it. The function is easy. What is not easy is that both of its failure modes are quiet in opposite directions. A validating handler that forgets to set `Allowed` denies everything, because `false` is the zero value. A mutating handler that forgets `PatchType` mutates nothing, because a nil pointer means "no patch here" and the server drops your work without a word. And around the function, `failurePolicy: Fail` means a crashed webhook can stop a cluster from scheduling pods at all.▸
Webhooks	wh1	A validating webhook is a plain function: apiserver sends an AdmissionReview describing the object under review, your handler returns an AdmissionResponse that allows or denies it. Two rules that trip everyone up: you MUST echo the request UID back, and Allowed defaults to false — so an "allow" path has to set it explicitly. This handler must deny any pod that lacks a "team" label. Test it directly — no cluster, no TLS, just the function.▸
Webhooks	wh2	A mutating webhook changes the object by returning a JSON patch. The gotcha is invisible: the AdmissionResponse must announce its PatchType, or the apiserver silently ignores the patch bytes — no error, the mutation just never happens. A JSON patch always travels with PatchType = JSONPatch. Build a response that injects a default label, and declare its patch type.▸
Testing Advanced · Webhooks & Testing 3 exercises
Testing	Chapter	A controller's interesting behaviour is almost entirely about failure: the conflict retry, the `IsNotFound` branch, the backoff when the API server is unhappy. None of that is reachable against a healthy cluster, and all of it is where the bugs are. The fake clientset exists to make those paths ordinary unit tests — fast, deterministic, and runnable anywhere `go` runs. Three pieces make that work. The fake clientset implements the same interface as the real one, over an in-memory object tracker. Reactors let you intercept any call and return whatever error you like. And `Actions()` records every call the fake received, turning "what did my controller actually do?" into an assertion. The important caveat runs through all of it: the fake is a good imitation, not an API server. Knowing precisely where the imitation stops is what keeps these tests from giving false confidence.▸
Testing	test1	You do not need a cluster to test client-go code — the fake clientset implements the same typed interface backed by an in-memory object tracker. Seed it with objects and every Get/List/Update works against that tracker. The one rule: the tracker keys objects by namespace AND name, exactly as seeded. Query the wrong namespace and you get an ordinary NotFound.▸
Testing	test2	Real code must survive API failures — but you cannot make a live server return "etcd unavailable" on demand. Reactors can: a reactor intercepts matching calls on the fake client and returns whatever you want. That is how you test the sad path. A reactor matches on (verb, resource). PrependReactor puts it at the front of the chain so it fires first, and returning handled=true short-circuits the default tracker.▸
Testing	test3	The fake client records every call it receives. Actions() returns them in order, and each action knows its verb and resource (Matches) and — for writes — carries the object that was sent. This is how you assert that a controller did the right WRITE, not just that it did not error. Inspect the recorded action to prove what was created.▸
Operator Advanced · Operator 3 exercises
Operator	Chapter	Controllers had you wire the skeleton by hand: factory, informers, key function, workqueue, workers, rate limiter, leader election. Every real operator needs all of it and none of it is where the value is. controller-runtime owns that machinery so you write one method — `Reconcile(ctx, req) (ctrl.Result, error)` — and get the rest from a manager. What survives the abstraction is the thinking. A reconcile still receives a key, not an event, and still has to read current state and make it correct. Ownership references still route child events back to the parent, so they stop being bookkeeping and become the routing table. And the helper that looks most convenient, `CreateOrUpdate`, has a trap that makes creation look perfect while updates silently never happen.▸
Operator	op1	Your first controller-runtime Reconciler. The manager owns all the machinery you built by hand in ctrl2 (informer → queue → worker); you only write Reconcile. The contract: a Request names one object, you return a Result plus error. NotFound is NORMAL — the object was deleted and the cache already forgot it. Returning an error there makes the manager retry a reconcile that can never succeed. Fetch the object the request names, and treat NotFound as "done".▸
Operator	op2	Owns() is how a controller notices its children. It maps an event on a child object back to a reconcile request for the OWNER — by reading the child's controller ownerRef. No ownerRef, no mapping: delete the child and nobody hears it. SetControllerReference is what stamps that ref. Stamp the parent's controller reference on the child before creating it.▸
Operator	op3	controllerutil.CreateOrUpdate is the reconciler's read-modify-write: it GETS the live object into your struct, runs your mutate() to lay desired state over it, then Creates or Updates only if something changed. Desired state set OUTSIDE mutate() is wiped by that Get — creation works once, but the update path becomes a no-op and drift is never repaired. Move the desired state (and the ownerRef) into the mutate function.▸