War stories and research notes from a local-first homelab AI cluster.https://robinsonlabs.ai/en-ushttps://robinsonlabs.ai/writing/rpc-coordinator-biggest-cpu/https://robinsonlabs.ai/writing/rpc-coordinator-biggest-cpu/We placed our cluster coordinator for core count and waited for tokenization to become the bottleneck. Then we measured it: 20 percent of one core. The speedup was hiding in the network leg.Sat, 04 Jul 2026 00:00:00 GMTWar story · ResearchStephen Robinsonhttps://robinsonlabs.ai/writing/122b-moe-beat-dense/https://robinsonlabs.ai/writing/122b-moe-beat-dense/Before the 10-gig upgrade, dense models were a trap: the sync tax scales with hidden_dim. The numbers, and the architecture rule we still design around.Fri, 03 Jul 2026 00:00:00 GMTWar story · ResearchStephen Robinsonhttps://robinsonlabs.ai/writing/tx-cliff-not-the-nic/https://robinsonlabs.ai/writing/tx-cliff-not-the-nic/One host's throughput cratered on every large model. We swapped NICs, cables, drivers, and reproduced it on all of them. The defect was somewhere we weren't looking.Fri, 03 Jul 2026 00:00:00 GMTWar story · OpsStephen Robinsonhttps://robinsonlabs.ai/writing/apt-history-first/https://robinsonlabs.ai/writing/apt-history-first/A validated cluster node started crashing on a workload it had served cleanly for weeks. Three days down the wrong rabbit hole (mmap theory, KV pressure, tensor-split) before checking the one log that actually mattered.Sun, 21 Jun 2026 00:00:00 GMTWar story · OpsStephen Robinson