apt history first: when "nothing changed" but everything broke

A node that had served the same model for weeks started failing. Nothing in the config had changed. Nothing in the model had changed. That phrase, “nothing changed,” is the tell.

The node was the vision worker: a stack of older AMD cards running a 30B vision-language model on Vulkan. It’s not glamorous hardware, but it had been rock-solid. Describe an image, summarize a screenshot, go back to sleep. Then one morning it stopped loading the model cleanly. Same GGUF. Same launch flags. Same box that had answered every request for weeks. By every measure I could point a command at, it was the machine it had always been.

So I did what fifteen years of debugging trains you to do: I assumed the problem was mine.

The wrong rabbit holes

When an inference node breaks, the suspects line up in a familiar order, and I walked straight down the list.

First theory: the model file. A bad copy off the NFS share, a truncated download, a flipped bit. I re-hashed it. Clean. It matched the copy on two other nodes byte for byte.

Second theory: memory pressure. Vision models are heavier than their parameter count suggests, and the obvious story is that something nudged the KV cache or the working set over a line it used to sit under. I trimmed context, dropped batch size, watched VRAM with everything else killed. The ceiling hadn’t moved. It still wouldn’t load.

Third theory: the placement and the loader. Maybe mmap was faulting on a weird page, maybe the tensor split across the cards had drifted, maybe a layer was landing on the wrong device. So I went spelunking through launch flags I hadn’t touched in a month, turning mmap off, pinning the split by hand, reordering devices. Each one ate a few hours, because each one was plausible enough to feel like progress.

Every one of those is a real failure mode. Every one of them was wrong here. By the time I had swapped models, rewritten launch flags, and burned through every parameter I could think of, I was just sitting there with nothing left to try. And that is when the actual question finally surfaced, the one I should have asked in the first ten minutes: did something get upgraded?

The one log that mattered

That hunch was the only thing that turned me away from my own stack and toward the system underneath it. The answer lives at /var/log/apt/history.log*.

Overnight, unattended-upgrades had run. It does that, quietly, on a timer, exactly as Ubuntu ships it. And in that run it had bumped libvulkan1 from the stock 1.3.204 that the stack was validated against to a 1.4 release candidate. The vision model loads through Vulkan. A new loader ABI underneath a workload that was, by every other measure, identical was the entire story.

When I pulled the log back and saw the overnight entry, the reaction was about what you would expect after three days of suspecting my own work: oh, you have got to be kidding me. Are you serious? No human had touched that box. That is precisely why “nothing changed” felt true, and precisely why it was a lie. Something with root and a schedule had reached in and swapped a load-bearing library out from under a validated stack, and it left no fingerprint anywhere I’d thought to look, because I was looking at my layer, and the change was a layer down.

The fix for the node itself was thirty seconds: pin libvulkan1 back to the known-good version, restart, watch it serve. The real fix was deciding it would never happen again.

The rule

We disabled unattended-upgrades on every validated inference host and put an apt-mark hold on the whole driver stack: Vulkan, CUDA, cuDNN, the kernel, the lot. Full disable plus a hold, not a softer “download but don’t install,” because the only state I want on a serving node is the one I validated. Manual apt upgrade still works when I deliberately want to patch something; nothing moves on its own. One hardening script, run once per box at onboard, zero ongoing maintenance. Nothing in the fleet drifts silently anymore.

But the durable lesson isn’t a config flag, it’s a debugging order of operations:

When a validated-stack node starts failing on work it has done cleanly for weeks, read /var/log/apt/history.log* for unattended-upgrades activity in the last seven days before you touch mmap, KV pressure, tensor split, or anything else in your own stack. It’s one command. It’s free. And it sits above every other theory on the list, because “nothing changed” is almost never true. It just means nothing you changed. On a machine with a package manager and a clock, that’s a much smaller claim than it sounds.

The reason it earns the top slot isn’t that silent upgrades are common. It’s that the cost of checking is a single cat and the cost of not checking is three days of suspecting your own work. Cheap test, expensive miss. That’s the whole heuristic. Check the log first.