It was only eighteen months ago that "deploying an LLM" meant wrapping a Hugging Face checkpoint in a Flask API, shoving it onto a single A100, and praying the OOM (Out of Memory) daemon didn't kill your process the moment a second user hit the endpoint.
In the early days of the generative AI gold rush, we treated Large Language Models like giant, fragile monoliths. You provisioned a GPU, loaded the weights, and that was your "server." If you needed more throughput, you bought another $30,000 card. It was inefficient, insanely expensive, and architecturally primitive.
Fast forward to today, and the landscape is unrecognizable. We have moved from static, rigid endpoints to dynamic, multi-tenant inference graphs. We are no longer just "serving a model"; we are orchestrating distributed systems that decouple compute from memory, separate prefill from decoding, and route requests through complex meshes of Mixture-of-Experts (MoE) and specialized adapters.
This is the story of the architectural evolution of LLM serving—a transition from "dumb" hardware allocation to the sophisticated, software-defined inference stacks that power the modern AI economy.
The Era of the Monolith: Why Static Endpoints Failed
In the beginning, LLM serving was essentially a CRUD problem with a very heavy payload. The architecture was simple: One Model per GPU (or Pod).
This "Static Endpoint" model suffered from three fatal flaws:
- VRAM Fragmentation: LLMs are memory-hungry. A 70B parameter model in FP16 takes up ~140GB of VRAM. If your GPU only has 80GB, you’re forced into multi-GPU setups (Tensor Parallelism), but even then, most of that memory is "dead air" until a request actually arrives.
- The KV Cache Crisis: As the model generates text, it stores "Key-Value" pairs of previous tokens to avoid re-calculating them. This KV cache grows linearly with sequence length. In static setups, we had to pre-allocate a fixed "max sequence length" buffer. If a user sent a 10-token prompt, 95% of the reserved memory sat idle.
- The Compute-Memory Mismatch: LLM inference happens in two phases: Prefill (processing the input prompt) and Decode (generating the output tokens). Prefill is compute-bound (it likes high FLOPS); Decode is memory-bandwidth bound (it likes fast HBM). Running them on the same hardware is fundamentally inefficient.
The industry realized quickly that if we wanted to scale to millions of users, we couldn't just keep throwing H100s at the problem. We needed a rewrite.
Phase 2: The Throughput Revolution (Continuous Batching & PagedAttention)
The first major breakthrough came from the realization that we were treating GPU memory all wrong. In 2023, researchers at UC Berkeley released vLLM, introducing a concept called PagedAttention.
Before PagedAttention, the KV cache was stored in contiguous memory. If you didn't have a perfectly sized block of memory for a new request, you couldn't process it—even if you had plenty of "holes" of free memory scattered around. This is classic "external fragmentation."
The Technical Substance: PagedAttention
PagedAttention borrowed a 50-year-old idea from operating systems: Virtual Memory. Instead of contiguous blocks, it breaks the KV cache into small "pages." These pages can be mapped to non-contiguous physical memory.
Why this mattered:
- Near-zero waste: Memory is allocated only as needed.
- Continuous Batching: Instead of waiting for an entire "batch" of requests to finish, the engine can inject new requests into the running batch the millisecond a previous request finishes a token.
# Conceptual look at how Continuous Batching manages the iteration loop
while True:
# New requests can be added even while others are generating
active_requests = scheduler.get_next_batch()
# Perform one step of inference for all active sequences
logits = model.forward(active_requests, kv_cache_manager)
# Sample next tokens and update the "pages" in memory
next_tokens = sampler(logits)
kv_cache_manager.update(next_tokens)
# If a sequence hits EOS, its memory pages are instantly freed
scheduler.free_finished_sequences()
This moved us from "Static Endpoints" to "Dynamic Batching Endpoints." Throughput increased by 10x-20x overnight, but the architecture was still tethered to the idea of a single, unified model instance.
Phase 3: The Disaggregation of Inference (PD Separation)
As we pushed the limits of vLLM and Hugging Face’s TGI (Text Generation Inference), a new bottleneck emerged. Remember the Prefill vs. Decode problem?
When a user submits a 2,000-word document and asks for a summary, the Prefill stage is massive. It saturates the GPU's compute cores. Meanwhile, other users who are already receiving their summaries (the Decode stage) are forced to wait. This leads to high Jitter and spikes in Time Per Output Token (TPOT).
The Rise of PD Separation
The most sophisticated engineering teams (Google, OpenAI, and specialized startups like Fireworks.ai or DeepL) began disaggregating the architecture. They split the inference cluster into two distinct pools:
- Prefill Nodes: High-compute nodes optimized for massive parallel processing of prompts.
- Decode Nodes: High-bandwidth nodes optimized for fast token generation.
By using a high-speed interconnect (like InfiniBand or RoCE), the "KV Cache" generated by the Prefill node is "handed off" to a Decode node.
The Infrastructure Shift: This turned LLM serving into a distributed state transfer problem. It's no longer just an API; it’s a network where multi-gigabyte KV caches are flying between nodes at sub-millisecond speeds. This allows for massive multi-tenancy because you can scale your "Decode" fleet independently of your "Prefill" fleet based on the specific traffic patterns of your users.
Phase 4: Multi-Tenant Inference Graphs and the MoE Explosion
This is where we are right now. The release of models like Mixtape-8x7B, Grok-1, and Llama-3 (and its rumored MoE variants) changed the game again.
We are no longer serving a "model." We are serving a Graph.
Mixture of Experts (MoE) and Dynamic Routing
In an MoE model, only a fraction of the total parameters (the "experts") are active for any given token. For example, in a model with 8 experts, a router chooses the top 2 for each token.
From a serving perspective, this is a nightmare and an opportunity. If you load all experts onto one GPU, you’re back to the VRAM wall. But if you're building a multi-tenant inference graph, you can distribute experts across a cluster.
The Multi-Tenant Architecture: In a modern enterprise environment, you don't just have one model. You have:
- A base model (e.g., Llama-3 70B).
- 50 different LoRA adapters (fine-tuned versions for coding, legal, creative writing, etc.).
- A Guardrail model to check for safety.
- A Router to decide which adapter or expert to use.
The "LoRA Exchange" (LoRAX) Pattern
Multi-tenancy used to mean "User A gets GPU 1, User B gets GPU 2." Now, multi-tenancy happens at the layer level.
With technologies like LoRAX, the base model weights stay fixed in GPU memory. When a request comes in for "Customer A's Legal Assistant," the system dynamically swaps in the tiny LoRA adapter weights (only a few megabytes) at runtime.
This is True Multi-Tenancy: You can serve thousands of custom, fine-tuned models on a single cluster of GPUs because you're only swapping the "delta" between models, not the models themselves.
The Technical Deep Dive: Inside the Inference Graph
Let’s look at what actually happens when you hit a modern LLM endpoint today. It's not a linear path; it's a traversal of an Inference Graph.
1. The Global Router & Request Steering
The request hits a global load balancer that isn't just looking at CPU/GPU load. It's looking at KV Cache locality.
- Engineering Curiosity: If a user is having a long conversation, the system tries to route them to the same physical node where their KV cache is already resident to avoid the "State Transfer Tax."
2. Speculative Decoding (The Parallel Hack)
To make tokens appear faster, we now use Speculative Decoding. The system runs a tiny "draft" model (e.g., a 100M parameter model) alongside the 70B giant. The draft model predicts the next 5 tokens very cheaply. The 70B model then checks all 5 tokens in a single parallel pass (a Prefill-style operation).
- If the draft was right, we just generated 5 tokens in the time it usually takes to generate 1.
- If it was wrong, we discard and go back to the giant.
This requires the inference engine to manage two models in sync—a complex dance of memory and compute timing.
3. The Execution Graph
In agentic workflows, the "inference" might involve multiple steps.
- Step A: A classifier model determines the intent.
- Step B: A retrieval step (RAG) pulls context from a vector DB.
- Step C: The LLM generates a draft.
- Step D: A critic model reviews the draft.
In a Static Endpoint world, this involves 4 separate API round-trips over the internet. In a Dynamic Inference Graph, this entire sequence is "fused" into the serving layer. The data never leaves the high-speed backend network until the final answer is ready.
Why the Hype is Actually Understated
You’ve likely heard the hype about "Custom Silicon" (TPUs, Groq's LPU, AWS Inferentia). The media focuses on the chips, but the software architecture described above is the real "moat."
The hype around companies like Groq or Together AI isn't just about raw speed; it's about their ability to handle SRAM-resident models and software-defined interconnects. When you can move data at 500GB/s between chips, your "model" can be spread across 256 GPUs, and to the user, it feels like a single, lightning-fast brain.
The Substance: Hardware-Aware Orchestration
We are moving away from Kubernetes' generic "Resource Limits" (CPU/RAM) and toward Hardware-Aware Orchestration.
Modern LLM schedulers need to know:
- The topology of the NVLink (How are these GPUs connected?).
- The HBM (High Bandwidth Memory) pressure across the cluster.
- The Compute-to-IO ratio of the incoming request.
If you send a request with a 100,000-token context, the scheduler shouldn't send it to a busy node—it should trigger a "distributed prefill" across four nodes to minimize latency. This is a level of infrastructure complexity that was previously reserved for High-Performance Computing (HPC) labs, now being deployed for every "Hello World" chatbot.
Building the Future: The Multi-Tenant Stack
If you were to build a premium LLM serving stack today, it would look like this:
- Orchestration Layer: A system like Ray or Kubernetes with specialized operators to manage GPU clusters.
- Inference Engine: A core that supports PagedAttention and Continuous Batching (vLLM, TensorRT-LLM).
- The Adapter Layer: Using LoRAX or S-LoRA to allow thousands of concurrent fine-tuned models.
- The Cache Layer: A distributed KV cache (like RadixCache) that allows sharing prefixes across different users (e.g., if 1,000 users are all chatting about the same 50-page PDF, you only store that PDF's KV cache once).
- The Graph Router: A logic layer that handles speculative decoding and multi-step agentic flows.
A Glimpse at the Logic (Conceptual Code)
# A modern, graph-based inference request
graph_request = {
"workflow": [
{"node": "classifier", "model": "distilbert-fast"},
{"node": "router", "logic": "if class='code' use 'llama-3-coder'"},
{"node": "generator", "model": "llama-3-70b", "adapter": "customer_a_style"},
{"node": "validator", "model": "guardrail-v2"}
],
"speculative_decoding": True,
"max_latency_ms": 200
}
# The serving engine doesn't just 'run' the model.
# It compiles this graph into a set of GPU kernels and network transfers.
engine.execute_graph(graph_request)
The Path Ahead: Inference as the New Operating System
We are witnessing the "Kernelization" of LLMs. In the same way an Operating System manages hardware resources for multiple processes, the inference stack is becoming an OS for weights and tokens.
The "Static Endpoint" is a relic. It was a bridge from the old world of REST APIs to the new world of cognitive compute. The future belongs to the Inference Graph—a fluid, stateful, and highly distributed architecture that treats VRAM as a global pool, compute as a tiered service, and intelligence as a routed commodity.
For engineers, this shift is exhilarating. We’ve moved past the "black box" phase. We are now in the era of optimization, where the difference between a mediocre implementation and a world-class one is measured not just in accuracy, but in tokens per second per dollar.
The monolith has been shattered. The graph has risen. And the most interesting part? We’re still in Version 1.0.