Here is a problem that makes robot operators grumble: a work cell runs happily for days, then stalls for a few seconds at odd moments, or stops accepting new jobs after a long weekend. The arm and the sensors check out fine, so everyone suspects the software. Quite often the culprit is the ordinary Linux computer that coordinates the cell, and the fix lives in SysOps, the practice of keeping that machine in good order. Below are four common symptoms, what usually causes them and how to fix each one without drama.
Symptom 1: the cell refuses new work after a few days
What it looks like: the controller software logs errors such as “too many open files” or fails to accept new connections, and a restart cures it for a while.
Likely cause: a limit on open file descriptors. The Ops Atlas notes that services started by systemd get a soft limit of 1024 open files by default, with a hard ceiling far above that. A program that opens sockets to many sensors and never closes some of them hits the soft ceiling eventually.
Fix: raise the limit in the service definition, not globally, using the supported directive; the Atlas warns that one older directive is marked “do not use” in systemd’s documentation, so check the manual for your version. Then find out why descriptors leak, because a higher limit only delays the stall. Before you restart anything, record the count of open files per process so you have evidence.
Symptom 2: the machine slows to a crawl on Monday mornings
What it looks like: response times stretch, logs are late and the operator interface freezes briefly.
Likely cause: a scheduled job colliding with production. Backups, log compression or a database clean-up that starts at a fixed minute can saturate the disk the controller depends on. The classic scheduler fires everything at the same instant, while timers can add a randomised delay so jobs do not all start together.
Fix: move heavy jobs to quiet shifts, give them a lower priority and use timers with a randomised delay. Also check that jobs finished: the Atlas treats scheduled-job success as a core measure. For help picking which numbers to watch on the factory floor, a rundown of the key robotic performance indicators managers track is a handy companion.
Symptom 3: timestamps jump and the cell logs make no sense
What it looks like: events appear out of order, durations come out negative or a safety log shows two entries with the same second.
Likely cause: clock correction. When a time service steps the clock backwards or a leap second is added, software that assumes time only moves forward can misbehave. The Atlas recounts a leap-second incident at a large network operator where a negative duration caused a panic, with a small share of DNS queries failing before the settings were changed to spread the correction over roughly twelve seconds.
Fix: configure the time service to slew instead of step for leap seconds, and allow stepping only during the first few updates after startup. Then test it: change the clock in a lab and watch how the controller logs behave. A cell that records data for later quality review needs monotonic, trustworthy time.
Symptom 4: the logging fills the disk and everything stops
What it looks like: writes fail, the database refuses to start or the operator screen shows blank history.
Likely cause: unbounded growth of logs, recordings or temporary files. Robot cells generate steady data, and a small embedded disk fills sooner than expected.
Fix: calculate the runway. Take the free space, subtract the slice the system reserves, and divide by the daily growth. If the answer is under two weeks, act: cap the journal’s size, rotate application logs, move recordings to bulk storage or enlarge the volume. Set the journal’s storage mode explicitly. Be aware that a rotation method which truncates a file in place can lose a very small amount of data, which matters if those logs are evidence.
A bonus: when the kernel kills something important
If the controller occasionally disappears with no error of its own, check whether the kernel ended it for lack of memory. A tempting trick is to give the process an extreme score that makes it unkillable. The Atlas says that merely moves the kill: another process, perhaps the watchdog or the logger, is chosen instead. Size memory properly, limit the greedy neighbours and fix the leak.
Reading the sensors’ side of the story
Sensor drivers and polling loops interact with host health in subtle ways, since a slow disk or a busy queue delays every reading. For background on how those feeds are assembled, see the explanation of how sensors are wired into industrial robots, then trace one reading from device to log on your own machine.
Why these faults hide so well
Cell stalls are hard to pin down because each cause looks innocent in isolation. A descriptor leak is invisible until the day it reaches the ceiling. A backup job runs perfectly well; it just happens to run at the wrong moment. A clock correction takes a fraction of a second. Because nothing is broken for long, nobody files a ticket, and the pattern only appears when somebody lines up the timestamps. The remedy is a habit rather than a tool: keep a short incident note for every stall, however brief, and review the notes monthly. Patterns that were invisible one event at a time become obvious across ten, and the notes make an excellent starting point for the alerts and baselines described above.
A quick routine for your next stall
- Do not restart yet. Save the journal, the process list and the open-file counts.
- Check disk space, memory available and whether any scheduled job ran at that time.
- Compare timestamps around the stall for clock steps.
- Apply one change at a time and write each down.
- Add an alert for the symptom so you hear about it before the operators do.
Keep it from coming back
Stalls are cheapest to fix when each one leaves a record. A one-page host sheet listing jobs, limits, log retention and the restore procedure makes the second occurrence a five-minute job. The Atlas describes this progression as moving from ad hoc to documented to measured, and its own entry on SysOps for Linux hosts is a good place to look up each setting named above. Put the routine on a laminated card next to the cell terminal and the next stall will feel a lot less mysterious.