What follows is an imagined account. The team, the warehouse and the dates are made up, but the failure modes come straight from documented Kubernetes behaviour. It follows a robotics operations group whose fleet-coordination software runs in a cluster, and it shows how a handful of ordinary KubernetesOps habits decide whether a Tuesday shift is calm or chaotic.
The setting
A mid-sized distribution centre runs automated guided vehicles that pick up totes and move them between stations. A coordination service assigns routes, a telemetry service gathers battery and position data, and a small dashboard shows operators the state of the floor. All three run as pods on a cluster of modest size. The operations group, three people on rotating on-call, also maintains the cluster itself. They think of it as part of the plant: if it stops, the robots stop.
Monday: a harmless-looking change
The group adds an admission rule, a webhook that checks every new pod for a required label so cost reports can be grouped by cell. Reasonable goal, modest code. What nobody remembers is that the Atlas lists the default behaviour of such webhooks as failing closed: if the webhook service cannot be reached, requests to create objects are rejected. Nobody has excluded the system namespace or the pod-network components from its scope.
Tuesday: the shift change
During the afternoon, the webhook’s own pod is evicted when its node runs short of memory. Pods are rescheduled, except that the networking component’s replacement pod cannot be created, because the very webhook that would approve it is down. A routine event becomes a circular dependency. Telemetry stalls, route assignments slow and the dashboard goes blank. The on-call engineer sees the symptom, not the cause, for almost twenty minutes.
The fix, once found, is plain. The team removes the webhook’s reach over system components and sets it to tolerate its own absence for non-critical namespaces. The lesson they write in the review matches the Atlas guidance: keep webhooks out of kube-system and away from the CNI, and treat the failure policy as a deliberate decision, not a default to inherit.
Wednesday: the probe that was too curious
While reading logs, they notice a second problem hiding in plain sight. The coordination service’s liveness probe calls the database to check that it is reachable. When the database had a short hiccup the week before, all replicas restarted together, producing a brief fleet-wide pause. The Atlas warns about this pattern: liveness probes should be process-local, because dependency checks can cause cascading failures. They move the database check into a readiness signal and leave liveness checking only the process.
Thursday: making the numbers visible
The group now wants a view of cluster health that operators can read without opening a terminal. They borrow from the Atlas’s list of indicators: how far behind they are on versions, days left in support, pods stuck unscheduled, restarts, API response time and whether a restore has been proven. It is the same mindset factory managers apply when they track throughput and downtime, as described in the piece on key performance indicators for robotic operations. They add a small panel to the existing dashboard showing the last restore test date in plain numbers, and it turns amber after ninety days.
Friday: the upgrade conversation
With things calmer, the team reviews the calendar. The cluster is two minors behind. The Atlas aims for no more than one behind in production, so there is work to do, and the sequence is not negotiable: one minor at a time, API server first, then the scheduler and controllers, then each node agent after its drain. They test a hop on a copy of the environment first. Because the fleet pauses whenever coordination pods move, they agree to run drains in the quiet window between shifts. A disruption budget for the coordination service keeps at least one replica alive at all times, and, learning from an earlier near-miss, they check it does not demand zero disruption, since that would block every drain.
A note on the two numbers that mattered most
When the imaginary team compared the week’s events against their indicators, two of them stood out. Restart rate would have flagged the probe problem days earlier, had anyone been watching it as a trend rather than as a pile of log lines. Restore readiness, the date a backup was last replayed, showed that nobody had practised recovery since the cluster was first built. In a plant where minutes of downtime have a price, the group decided that a drill every quarter was cheaper than learning the procedure during an outage. They also wrote the steps on a single page and stored it where the on-call phone could reach it without the cluster being up.
What the review concluded
The imagined post-incident document is short. Failure policy on admission webhooks needs an owner. Liveness means process-local. Restore dates belong on a dashboard. And the team promised to take their network-component inventory seriously, because the part of the stack that robots depend on most is the part that humans look at least.
The Atlas also reminds readers, in its KubernetesOps entry, that none of this is a product you can buy under one name; it is a collection of habits. For readers connecting the cluster to the physical floor, the overview of automating warehouse movement with guided vehicles is a useful companion.
KubernetesOps takeaways for a robotics operations team
- Audit every admission webhook for scope and failure policy.
- Keep liveness checks local to the process.
- Place a restore-date indicator where operators will see it.
- Plan drains around shift patterns and keep budgets realistic.
The robots did their job all week. The quiet achievement was that the cluster around them stopped being the interesting part of the story.