The restore bug that looked like a success
One background service quietly died every time we restored a desktop. Our checks said everything was fine.
October 28, 2026
Drums gives each training attempt a desktop that can be saved and restored in about a second. Under the hood we use CRIU, the open-source tool that freezes running programs, writes them to disk and brings them back. When many desktops restored at the same time, a few percent of restores failed with a confusing error: Unable to detach: No such process.
What was really happening
The culprit was rtkit-daemon, a small Linux service that helps audio and video programs get priority. It sets a rule on itself: it may never write to a file. That rule is normal. The problem was the order of things during a restore.
- CRIU put the "never write a file" rule back on the service.
- Then CRIU kept writing its own restore log from inside that service, for a few more steps.
- Each of those writes broke the rule, so the system lined up a signal that means "stop now".
- The moment CRIU let the service run again, the signal fired and the service died.
If the service died while CRIU was still finishing, the restore failed with the confusing error. If it died a moment later, the restore reported success, and the desktop simply came back without that service. That second case happened on every restore, including the ones that looked perfect.
Why it mattered
During our first full training run, desktops kept dropping out. Each failed restore took a desktop out of the pool, until none were left and training continued with no new data. Our state checks did not catch it, because the files and screen were exactly right. Only a running program was missing.
The fix
We moved one step: the "never write a file" rule is now applied at the very end, after CRIU has written its last log line and just before the programs start running again. If applying it fails, the restore fails loudly instead of handing back a broken desktop.
| Before | After | |
|---|---|---|
| Failed restores, 16 desktops restoring at once | 3.6% to 6.7% | 0 of 1,700 |
| Service still alive after a "successful" restore | 0 of 10 | 10 of 10 |
| New automated test (3 threads, the same rule) | fails 10 of 10 | passes 10 of 10 |
| Desktop failures in our next training run | pool emptied after 7 steps | 0 |
What we changed in how we test
Checking that files and pixels match is not enough. Every restore in Drums now also checks that the programs that were running before are still running a few seconds after. That check is what showed the old build losing the service on 10 out of 10 restores while every other check passed.
What is next
The fix and its test are part of our own CRIU build. Other process limits could cause similar trouble; we have not seen it, and the new liveness check would catch it if they did.
Measured on AWS (c8i.4xlarge, Linux 7.0) with our CRIU build based on version 4.3. Details and raw data are on bench.drums.run.