What we learned testing Drums against virtual machines
Our first head-to-head benchmark. Some of it went our way, some of it did not.
October 28, 2026
AI agents that use computers are trained by letting them try tasks on real desktops, thousands of times. Today each attempt usually gets its own virtual machine. Drums takes a different approach: many attempts share one running operating system, and each attempt keeps its own private files, screen and clipboard. We wanted to know if that is actually better, so we tested it against the setups people use today.
What we tested
One cloud server, real desktop tasks from the OSWorld benchmark, and the same AI agent (Claude Opus 5.5) for every setup. We compared Drums with tuned virtual machines (QEMU/KVM), Firecracker microVMs and a desktop container, and also measured OSWorld's own Docker setup, an OSGym-style setup and Cua.
The main test: take a task halfway done, make 8 independent copies of that exact moment, and let the agent finish each copy. Which setup produces the most correct finishes per dollar?
Where Drums wins
- Memory. Each copy of a desktop needs about 270 MB, versus about 2.5 GB for a virtual machine: roughly 9 times less.
- How many fit on one server. We ran 1,024 desktops on one machine with memory to spare. The virtual machine setups topped out around 128.
- Making copies. 8 copies of a running desktop are ready in about 1 second, 3 to 4 times faster than Firecracker.
- Resetting. A desktop resets in about 1.3 seconds, versus 16 to 95 seconds for the open setups people commonly use today.
- Isolation. Zero leaks between copies, in every test.
- Fixes to open source. We fixed bugs in CRIU, the open-source tool that saves and restores running programs: lost file locks, a background service that was silently killed on every restore, and saving a desktop dropping from about 2 minutes to a few seconds.
Where Drums falls short
- Some apps cannot be saved yet: Chrome, VS Code and Thunderbird.
- Tuned virtual machines save a running desktop faster (about 1 second versus 4).
- Firecracker resets faster (0.7 seconds versus 1.3).
- With today's expensive models, Drums' speed and memory savings are a small part of the total cost.
A first training run
We also ran our first full training run with Drums supplying every desktop: a 9-billion-parameter computer-use model, trained with reinforcement learning, with no desktop failures. It was too short to show the model improving. It shows the pipeline works end to end.
Cost per correct result: not there yet
| Drums compared with | Correct finishes per dollar |
|---|---|
| Tuned virtual machines | 20% fewer |
| Firecracker microVMs | 17% fewer |
| Desktop container | About the same |
On this measure, Drums came in behind the virtual machines for now, and the reason is specific. Drums could not save the state of a few apps: Chrome, VS Code, some system-settings tasks and Thunderbird. That broke 14 of the 80 test cases. On the cases where every setup worked, Drums tied the virtual machines.
We also learned that the AI model is about 95% of the total cost. A faster environment barely changes the bill when the model is that expensive.
What comes next
- Fix the apps Drums cannot save yet, then rerun the same test.
- Test the cases where memory and density decide the bill: many desktops per server, and cheaper AI models.
The fine print
One server and 80 paired test cases. We call a difference real only when its 95% confidence interval rules out a tie. Every failure is counted against the setup that had it. Cua was run with its open-source parts only, because its fast-restore server has a license we cannot use. OSGym's code is not public, so we tested our own version built from its paper. The full numbers and raw data are on bench.drums.run.