The infrastructure conversation of 2023 has been dominated by one question: how do you scale LLM inference? The answer reveals a deeper truth about scaling itself—horizontal and vertical are not interchangeable strategies; they solve fundamentally different problems.
The stateless illusion. Web scaling mythology rests on the premise that any request can be routed to any server. This works for HTTP requests, cache lookups, and database reads because each request carries its own context. LLM inference breaks this model. A model loaded into GPU memory occupies 10-80GB depending on parameters. You cannot simply add more servers and expect linear throughput—each server must load the model, warm its KV cache, and maintain context for active sessions.
Batching as the new horizontal. The solution is not more servers but smarter batching. A single GPU serving 50 concurrent requests with dynamic batching achieves higher throughput than 50 GPUs each serving one request sequentially. This inverts the traditional scaling model: instead of distributing requests across more machines, you concentrate them onto fewer, more fully utilized machines.
Implications for procurement. Government cloud procurement frameworks—whether through TED in the EU or SAM.gov in the US—still specify compute requirements in terms of vCPU count and RAM. GPU capacity is an afterthought, if it appears at all. As AI workloads become central to public-sector digital transformation, procurement specifications must evolve to account for GPU memory, interconnect bandwidth, and inference latency SLAs.
We have been designing GPU-constrained infrastructure for clients since before the current hype cycle. The principles are the same whether you are scaling a ClickHouse cluster or an inference pipeline: understand your vertical bottleneck first, then scale horizontally around it.