When a person starts limping, a good clinician does not guess; they ask where it hurts, when it started and what changed. Operations teams working on container clusters need the same habit, and the daily craft of KubernetesOps is mostly that: reading symptoms, narrowing causes and choosing the smallest fix. Let’s chat through four common “limps” in a cluster, the likely causes behind each, and what an administrator can try, always from the point of view of a hospital operations office rather than a clinic. None of it is health guidance; the subject is keeping systems upright.
Symptom one: the maintenance window stalls on a single machine
The team starts draining a worker machine so it can be updated, and the command just sits there. Nothing is wrong with the machine. The cause, very often, is a disruption budget. These budgets declare how many copies of an application must stay available, and if one is set to allow zero interruptions, a drain cannot move that application anywhere.
A reference entry from xOps School, the Ops Atlas page on this discipline, lists zero-disruption budgets among the classic pitfalls, and its thirty-day plan begins by finding and fixing them. The cure is to relax the budget to something honest, say one copy allowed to move at a time, and then retry. What it advises against is yanking the stuck application out by force, since that sidesteps the budget entirely. Force deletion is a last resort, and the owner of the application should agree to it first.
Hospital version: a ward that insists every bed must stay occupied during a refit. Nobody can paint, so the refit never starts. The fix is a conversation about how many beds can move at once, not a crowbar.
Symptom two: one failure spreads like a rumor
A database blips for ten seconds, and suddenly dozens of healthy application pieces restart in a loop. This is the signature of a liveness probe that checks too much. A liveness probe is a question the cluster asks: “are you alive, or should I restart you?” If that question includes the state of a database or another service, then the outage of that service makes everything look dead, and the restarts pile on top of the original trouble.
According to the Atlas, liveness should be process-local. It should only ask whether the process itself is stuck. Questions about dependencies belong in a different kind of check, one that merely pauses traffic rather than ordering a restart. The comparison with a hospital is straightforward: a fire alarm should detect fire, not the fact that a neighboring building has lost power.
Symptom three: new deployments are refused with odd errors
Suddenly nothing can be created. Error messages mention a webhook that timed out. Admission webhooks are rules that inspect each request before it is accepted, and by default the setting called failurePolicy is Fail, which means that if the rule’s server is unreachable, the request is rejected. If the webhook’s own server runs on the very cluster it guards, you have built a lock that needs the door open to be unlocked.
The Atlas recommends keeping webhooks away from the kube-system namespace and away from the networking plugin. That way the parts the cluster needs in order to heal itself are never held hostage by a policy server. A hospital would not route the generator controls through the badge-reader network for similar reasons; essential services need a path that does not depend on the checks.
Symptom four: applications get evicted and “protection” did nothing
A machine runs low on memory and workloads vanish, even though a disruption budget was in place. Here the pitfall is a misunderstanding. The Atlas states that eviction triggered by node pressure ignores those budgets, and preemption respects them only on a best-effort basis. Budgets protect against voluntary actions like drains; they are not a shield against a machine running out of room.
What decides who goes first? The entry gives the ordering: pods using more than they requested are evicted first, and ties are then settled by priority. That makes the requests an application declares a contract, because an honest request is both a promise to the scheduler and a place in the eviction queue. Declaring a tiny request for a hungry workload buys a smooth placement today and a first-in-line seat for eviction tomorrow.
If you enjoy plain talk on keeping complex operations steady, our article on building resilient infrastructure for modern IT operations pairs well with this one.
Symptom five: latency creeps up and nobody changed anything
Often it did change: demand rose, or the share of machine time handed to applications drifted away from what they truly use. Two habits help. First, check how much capacity has been requested compared with what is used; the Atlas calls this the usage-to-request ratio and recommends tracking it. Second, remember that autoscaling depends on requests. The horizontal pod autoscaler can only act on CPU utilization if the workload states a CPU request, because utilization is measured against it.
The entry also includes a worked example. With four replicas running at 90% against a 60% target, the formula asks for six. At 64%, it does nothing, because a tolerance of 0.1 keeps small wobbles from causing churn. A scale-down only happens after a 300-second stabilization period. Those details are why a team that tunes numbers by hand without understanding the formula often ends up chasing its own tail.
The same entry mentions a debate worth knowing about: whether to set CPU limits at all. A 2020 post from an engineering team found removing them reduced latency, and a kernel bug was fixed in version 4.19. The Atlas treats it as an open discussion, not a rule, so test on your own workloads.
A tiny triage checklist
- Is the problem local to one machine, one application or the whole cluster?
- What changed in the last day: a version, an add-on, a budget, a rule?
- Do requests match real usage for the affected workload?
- Is a safety rule, such as a budget or webhook, causing the block instead of the failure?
- When did a restore last succeed, in case the answer is to rebuild?
For a lighter read on keeping automated things under control, take a look at a calm piece on running smart machines sensibly, which complements this checklist nicely.
The friendly takeaway
Almost every cluster limp traces back to one of three roots: a rule that is too strict, a number that is untrue, or a rehearsal that never happened. Diagnose in that order, alter a single variable per attempt and jot down what you saw. That is the whole craft, and it is far more about patience than genius.
