A robot cell that uses a learned model for vision or grasp selection has two lives. The first is the commissioning week, when everything is measured and everyone is watching. The second is the next eighteen months, when nobody is. Most operational surprises happen in the second life, and many trace back to beliefs that were reasonable on day one. Below I put six of those beliefs up against the evidence gathered by the xOps School Atlas reference for MLOps, a footnoted summary of how teams operate learned models. The scenarios are illustrative, not reports of real installations.
Myth 1: “The vision model is installed once, like a camera bracket”
What the evidence says. A learned model is an artefact built from data, and the data describes a moment in time. The Atlas describes the discipline as the continuing care of a deployed model: watch for skew and drift, and retrain under rules rather than on impulse. On a factory floor, drift arrives in boring forms: a supplier changes the surface finish of a part, the lighting in a bay is altered, a new product variant joins the line. None of these breaks a bracket. All of them can erode a model’s accuracy without raising a fault code.
The operational answer. Treat the model like a consumable with a review date, and monitor on three levels: is the service responding within its time budget, do incoming images resemble the ones it learned from, and are the results still good? That last one is awkward. Ground truth may arrive late, for instance when a downstream inspection finds a defect hours after a pick. The Atlas suggests watching stand-in signals such as the spread of predictions while you wait.
Myth 2: “If it works in simulation, the floor will match”
What the evidence says. The reference calls the mismatch between how a model is trained and how it is served skew, and it lists concrete controls: reuse the same preparation code on both paths, log the inputs that the live system actually saw and replay them through the training pipeline to confirm agreement. Simulated images are cleaner than real ones. Real cameras add noise, glare and calibration drift. A model trained on one and served on the other is an exact case of mismatch.
The operational answer. Capture real frames at the cell, store them with the model version that processed them and replay them offline. Where the Atlas expects no disagreement between paths, accept that a sim-to-real gap is a measured quantity you track, not a rumour you assume away. If you are weighing the sensing side, our explainer on the way sensors are integrated into industrial robots helps explain where such differences start.
Myth 3: “A model that beats the old one overall can go straight to the fleet”
What the evidence says. Overall scores hide segments. One lab in the Atlas has a challenger up 0.009 in aggregate yet down 0.029 for a single group; the policy turned it away, and it only passed after the group’s result was repaired. A parallel exists on the floor: a grasp model that is better on average but worse on shiny parts will cost real money on the line that handles shiny parts.
The operational answer. Define slices before the test: part families, lighting conditions, shifts, camera positions. Compare on a frozen, time-split test set. Then release progressively. The Atlas lists shadow mode, where the new model’s outputs are logged but only the incumbent’s are acted on, then a canary with a small share of traffic, then an online experiment. For a robot fleet, a canary might be a single cell on the quietest shift.
Myth 4: “A small tweak to one input is safe”
What the evidence says. The Atlas cites the acronym CACE, “changing anything changes everything”, from the hidden-technical-debt literature. Replacing a camera, shifting a crop window or changing a normalisation constant alters the distribution the model sees, and the effects interact.
The operational answer. Route even small input changes through the same gate as a model change. Hold one definition per input and one owner per definition. The Atlas also flags undeclared consumers: if a scheduling tool reads the confidence output from the vision model, a change in calibration can quietly alter its decisions. Keep a list of everything that reads model output.
Myth 5: “Nightly retraining and automatic promotion keeps the fleet fresh”
What the evidence says. The reference lists automating everything as an anti-pattern, noting that a retrain-and-auto-promote cycle can cause outages through feedback loops. In robotics, loops are easy to build by accident: if the model decides which parts to pick, and the picked parts are what new training data is made of, the model can reinforce its own blind spots.
The operational answer. Automation level is a decision, not a default. Retrain on a written policy, such as a schedule, a volume of new labelled data or a sustained degradation, and require a candidate to pass the gate before promotion. The Atlas adds a modest way to choose the cadence: train to a past date, measure how fast quality decays week by week, and set the retraining rhythm from that curve.
Myth 6: “Falling back to the old logic means we failed”
What the evidence says. The Atlas is insistent that going back means re-pointing an alias or a share of traffic, with no retraining involved, and that a non-ML fallback should be kept and rehearsed on a regular rhythm, such as quarterly. In an industrial setting the fallback can be rule-based picking, a fixed trajectory or a hand-off to a human.
The operational answer. Define the trigger, such as a rise in rejected picks, and the person who authorises the switch. Practise it. A rehearsed fallback turns a stoppage into a scheduled-looking event. The KPI view matters here too; the factory-level measures discussed in the robotic performance indicators every factory manager should track give you the production-side signals that tell you when to pull the lever.
A worked example from a fictional packing line
To see the myths interact, imagine a packing line whose vision model identifies box orientation. In spring a new carton supplier changes the printed pattern. The model’s pick success dips by a small amount. Nobody notices for three weeks because the dips are within shift-to-shift noise. A monitor that compares incoming image statistics against the training snapshot flags the new pattern on day two. The engineer’s first move is not to retrain. She replays logged frames through the training-side code and confirms the two paths agree, so skew is ruled out. She then checks the supplier change, so an upstream defect is ruled out, and concludes that this is real input drift of a benign kind. Labels are available from the operators’ corrections, so retraining on fresh frames is the right response. The candidate goes through the gate, passes on all three named slices, runs in shadow for a shift and goes to one cell as a canary before the rest follow. If it had misbehaved, the alias would have moved back in minutes.
The scenario takes roughly a week in the telling and a few hours of real effort, because the records and the drills existed before the dip. That is the economic argument for the whole discipline in one paragraph.
A compact evidence table
- Drift: distinguish pipeline skew, upstream defects and genuine change before retraining.
- Thresholds: calibrate on your own history; the widely repeated cut-offs for one popular drift score have no academic footing.
- Windows: compare like with like, not a holiday shutdown week with a full production week.
- Embedded serving: the Atlas lists batch, online, streaming and embedded serving as in scope, so a model on a robot controller is still in the discipline’s territory.
Where the myths come from
Every one of these beliefs is a sensible extension of how physical equipment behaves. A bracket stays put, a gearbox wears predictably, a calibration holds for months. A learned model behaves differently: it can degrade without any physical change and improve on paper while getting worse where it matters. Robotics teams that internalise that difference tend to adopt the habits the Atlas describes without much resistance, because they already think in terms of maintenance intervals and safe states.
First moves for a robotics team
Start with one cell and one model. Record each training run’s code, data fingerprint, parameters and environment. Store a time-split test set and name three slices. Write the gate in plain language. Add the three monitoring levels with named owners. Rehearse the fallback once before you need it. These steps align with the Atlas’s own modest starting advice: a lone repeatable pipeline plus a registry with provenance and an exit route is a better bet than an elaborate platform left to gather dust. The rest can follow when evidence shows where the real pain is.