The coordinator didn't need our biggest CPU

When we first built the cluster, the instinct was obvious. We were splitting large mixture-of-experts models across a set of machines using the llama.cpp RPC pattern. One node acted as the coordinator, handling tokenization, sampling, and orchestration. Every worker node talked to it.

The serial nature of that work pointed a finger at the CPU. If the coordinator choked, the whole cluster stalled, regardless of how much GPU power sat behind it. We assumed the coordinator would be the ceiling. We needed a machine with maximum core count to keep the serial pipeline moving.

So we built the coordinator box around an 18-core processor. We chose it for its core count. We thought we were future-proofing the cluster against CPU saturation.

It was a reasonable guess. It was also wrong.

The measurement

The belief held until April 2026. That was when we started a systematic bottleneck hunt. We ran sustained inference on 120B-class and 122B-class MoE models. We watched the coordinator.

The numbers were not what we expected. The coordinator sat at roughly 20 percent of a single core. Nonvoluntary context switches hovered at 0 to 3 per second.

It was not CPU-bound. It was not even close.

We had planned a stage-variation benchmark sweep based on the theory that the CPU was the limit. We cancelled tier one of that sweep immediately. The data disproved the premise. The 18-core machine was overkill. The serial work was light enough that a modest processor could handle it without breaking a sweat.

What the wire said

If the CPU was not the bottleneck, what was?

We looked at the network traffic. In this RPC pattern, the per-token payload is small. We are sending hidden-state hand-offs at layer boundaries. The expert weights never cross the wire.

The observed traffic was about 300 KB per token. The peak bandwidth was roughly 22 Mbps. That is about 0.2 percent of a 10-gigabit link’s line rate.

The network was nowhere near bandwidth-saturated. A 0.2 percent utilization figure usually means the link is idle. But bandwidth is not the only metric that matters. Latency is the other half of the equation.

The topology is hub-and-spoke. The coordinator holds every worker connection. The workers never talk to each other. The token loop is serial. The coordinator sits on every single hop of that loop.

Every token has to travel from the coordinator to the workers and back. The coordinator is the bottleneck not because it has to do too much work, but because it has to wait for the network to deliver the data. The serial nature of the loop means that even a tiny delay on the wire compounds with every token.

The lever that worked

We had a clear path to fix the latency issue. We did not need more CPU cores. We needed a faster link.

We upgraded only the coordinator’s network leg from 2.5-gigabit to 10-gigabit. Every other node in the cluster remained unchanged. We kept the 18-core CPU. We kept the same worker nodes.

The result was immediate.

On short generations, measured at 128 tokens, cluster throughput lifted by 22 percent. We hit 30.4 tok/s on the 120B-class reference model. On long generations, measured at 512 tokens, throughput improved by 16 percent. We hit 25.6 tok/s.

The speedup was hiding in the network leg. The CPU was sitting idle, watching the network do the work.

The rule

We changed our placement strategy. We no longer choose the coordinator based on core count. We position it by its network leg. The coordinator gets the fastest, lowest-latency link first. The cores it needs are modest.

This is a specific lesson for the llama.cpp RPC layer-split decode on MoE models. It does not mean CPU never matters for inference. It means that in this specific architecture, the serial coordination overhead is light, and the network latency is the dominant factor.

Measure the bottleneck before you buy for it.

The serial-work intuition points at the CPU. It is a natural assumption. But the wire clock disagrees, and the wire clock is usually right.

There is a cheap check that settles this argument in a minute. Watch the coordinator CPU during a sustained multi-node run. If it is sitting at 20 percent of one core, you have your answer. Stop looking at the CPU. Look at the network.

Low bandwidth does not mean the network is innocent. A link can idle at 0.2 percent utilization and still set your pace if every token serializes through it. Bandwidth and latency are different suspects. You have to interrogate both.

We spent months buying cores to solve a problem that was solved by a cable upgrade. The lesson is simple. Do not assume the serial work is heavy just because it is serial. Measure it. If the CPU is not the limit, the limit is elsewhere.