Before the 10-gig upgrade, dense models were a trap: the sync tax scales with hidden_dim. The numbers, and the architecture rule we still design around.

One host's throughput cratered on every large model. We swapped NICs, cables, drivers, and reproduced it on all of them. The defect was somewhere we weren't looking.

A validated cluster node started crashing on a workload it had served cleanly for weeks. Three days down the wrong rabbit hole (mmap theory, KV pressure, tensor-split) before checking the one log that actually mattered.