The year is 2024, and the “GPU-poor” vs. “GPU-rich” divide is no longer just about who owns the most H100s. It’s about who can actually use them.
In the early days of the LLM gold rush—meaning about 18 months ago—the strategy was simple: throw a model onto a GPU, wrap it in a Flask API, and hope the OOM (Out of Memory) errors didn’t happen too often. But as we transition from experimental toys to massive, multi-tenant production environments, that strategy is hitting a wall. A very expensive, silicon-based wall.
At scale, the bottleneck isn’t just raw TFLOPS; it’s VRAM orchestration. When you are running a cluster serving thousands of concurrent users across hundreds of different LoRA adapters and model versions, you aren’t just a software engineer anymore. You are a logistics manager for a high-speed, volatile, and incredibly fragmented memory landscape.
Today, we’re diving deep into the architecture of modern LLM inference engines. We’ll explore why traditional Kubernetes scheduling fails LLMs, how PagedAttention rewritten the rules of memory management, and how to build a hardware-aware orchestrator that treats VRAM fragmentation not as a bug, but as a solvable scheduling problem.
The Physics of the Problem: Why VRAM is Dying a Death of a Thousand Cuts
To understand why we need hardware-aware orchestration, we have to look at what’s actually happening inside the H100’s HBM3 memory.
In a standard inference request, VRAM is consumed by three primary entities:
- Model Weights: Static, predictable, and massive (e.g., ~140GB for a FP16 Llama-3 70B).
- Intermediate Tensors: Activation buffers used during the forward pass.
- The KV Cache: The “memory” of the conversation.
The KV Cache is the silent killer. For every token the model generates, it needs to store the Key and Value vectors for all previous tokens in its self-attention layers to avoid recomputing them. As the sequence length grows, the KV cache balloons.
In a traditional memory allocation scheme (like CudaMalloc), memory is allocated contiguously. If a user starts a conversation that eventually grows to 4,000 tokens, the system has to reserve a massive chunk of contiguous VRAM upfront, or risk failing later. Because we don’t know how long a user’s response will be, we over-provision.
The Fragmentation Tax
This lead to two types of waste that crippled early LLM clusters:
- Internal Fragmentation: Reserving space for 4096 tokens when the user only generates 10. The remaining 99% of that reserved VRAM sits idle, but unusable by other requests.
- External Fragmentation: Small “holes” of free memory scattered across the VRAM that are too small to fit a new request, even if the total free memory is sufficient.
In a multi-tenant environment where you’re trying to pack as many requests as possible onto a single node to justify the $30,000 price tag of the GPU, this inefficiency is a death sentence for margins.
Enter PagedAttention: The Virtual Memory Moment for AI
The breakthrough came with the realization that we could treat GPU memory like an Operating System treats RAM. This is the core innovation behind vLLM and its PagedAttention algorithm.
Instead of demanding a contiguous block of VRAM for the KV cache, PagedAttention breaks the cache into small, fixed-size blocks. These blocks can be scattered anywhere in the VRAM. A lookup table (a “Page Table”) maps logical tokens to physical blocks.
# Conceptualizing the Page Table Mapping
logical_kv_cache = {
"request_id_alpha": [physical_block_7, physical_block_102, physical_block_15],
"request_id_beta": [physical_block_2, physical_block_99]
}By allowing non-contiguous allocation, we effectively eliminate external fragmentation. More importantly, we can perform Copy-on-Write (CoW) for parallel sampling. If a user asks for five different endings to a story, they can all share the same KV cache for the prompt, only branching off into new blocks when they start generating different tokens.
But here is the catch: PagedAttention solves the problem inside a single inference engine. It doesn’t solve the problem of how you manage a cluster of 500 nodes, each running multiple engines, serving different models, with varying degrees of VRAM fragmentation.
Orchestration Beyond the Node: The “Hardware-Aware” Layer
Traditional orchestrators like Kubernetes see a GPU as a monolithic integer resource: nvidia.com/gpu: 1.
This is fundamentally insufficient for LLM inference. To the K8s scheduler, a GPU with 79GB of weights loaded and 1GB of KV cache available looks exactly the same as a GPU with 0GB of weights loaded and 80GB of KV cache available. Both are “used.”
A Hardware-Aware Orchestrator must be “VRAM-literate.” It needs to peek inside the engine and understand the Memory Pressure State.
1. Topology-Aware Bin Packing
When you’re dealing with models that require Multi-GPU setups (Tensor Parallelism), the “where” matters as much as the “how much.”
If you split a Llama-3 405B model across 8 GPUs, those GPUs must be on the same node or connected via an ultra-low-latency fabric like NVLink. An orchestrator that schedules 4 GPUs on Node A and 4 GPUs on Node B over a standard 10GbE network will result in inference speeds measured in seconds per token rather than tokens per second.
Your scheduler needs to understand the NVLink topology graph:
- Level 0: On-die communication.
- Level 1: NVLink (intra-node).
- Level 2: InfiniBand/RoCE (inter-node RDMA).
A hardware-aware scheduler will prioritize “affinity groups,” ensuring that high-bandwidth-dependent tasks are never split across a slow PCIe bus or a standard top-of-rack switch.
2. The LoRA Swapping Problem
In a multi-tenant cluster, users aren’t just hitting base models. They are hitting fine-tuned versions (LoRA adapters).
A naive orchestrator might spin up a new inference engine for every LoRA. This is a VRAM disaster. Instead, modern architectures use Multi-LoRA serving. The base model stays in VRAM, and the tiny LoRA weights (the “adapters”) are swapped in and out of the GPU’s memory dynamically.
The Engineering Challenge: LoRA weights are small (~100MB), but swapping them from CPU RAM to GPU VRAM takes time.
A hardware-aware orchestrator implements LoRA-LRU (Least Recently Used) Caching. It tracks which adapters are already “warm” on which nodes. If a request comes in for finance-llama-7b, the scheduler routes it to the node that already has those weights in its local VRAM cache, preventing a costly PCIe transfer.
Deep Dive: The Anatomy of a High-Scale Inference Request
Let’s trace a request through a hardware-aware, multi-tenant stack.
Step 1: The Global Load Balancer (The Traffic Cop)
The request hits the entry point. The balancer doesn’t just check for “round-robin” availability. It queries a Global State Store (like Redis or a specialized Control Plane) to find:
- Which nodes are running the required base model?
- Which nodes have the LoRA adapter cached?
- What is the current KV Cache Fill Rate across the cluster?
Step 2: The Continuous Batching Engine
Once the request reaches the node, it enters a Continuous Batching queue. Unlike traditional batching (where you wait for 16 requests to arrive and run them together), continuous batching allows new requests to join the batch as soon as others finish.
This is where PagedAttention shines. The engine allocates “virtual blocks” for the new request. If the VRAM is too fragmented, the engine doesn’t crash; it triggers a Preemption Signal.
Step 3: The Preemption and Swap Mechanism
If the VRAM is 100% full, the orchestrator has to make a choice. It can:
- Recompute: Drop the KV cache of a low-priority request and recompute it later.
- Swap to CPU: Move the KV cache blocks from GPU HBM to system DDR5 memory.
This “Swapping” is the safety valve. System RAM is much larger and cheaper than VRAM. By treating CPU RAM as a “Level 3 Cache” for the KV blocks, we can sustain bursts of traffic that would otherwise cause an OOM.
Technical Implementation: Designing the Scheduler Logic
If we were to write a simplified version of a hardware-aware scheduler in a Go-based controller, it might look something like this:
type GPUNode struct {
ID string
TotalVRAM uint64
ReservedWeights uint64
ActiveKVCache uint64
CachedLoRAs []string
TopologyNeighbors []string // Nodes connected via InfiniBand
}
func (s *Scheduler) SelectNode(req InferenceRequest) (*GPUNode, error) {
// 1. Filter nodes by Base Model availability
candidates := s.findNodesWithModel(req.BaseModel)
// 2. Score nodes based on LoRA warmth
for _, node := range candidates {
if node.HasLoRA(req.LoRAID) {
node.Score += 50 // High priority for warm cache
}
}
// 3. Score nodes based on "Headroom" (VRAM - (Weights + KV Usage))
// We want to avoid nodes at >90% KV capacity to prevent swapping
for _, node := range candidates {
headroom := node.GetAvailableKVCache()
if headroom < req.ExpectedTokens * BYTES_PER_TOKEN {
node.Score -= 100 // Penalize heavily if risk of OOM
}
}
return s.pickHighestScore(candidates), nil
}The BYTES_PER_TOKEN calculation is non-trivial. For a 16-bit model, it’s roughly:
2 * num_layers * num_heads * head_dim * 2 (for both K and V).
For a Llama-2 7B, that’s about 0.5MB per token. For a 70B, it’s significantly more. A hardware-aware orchestrator must calculate this on the fly based on the model architecture.
The Networking Bottleneck: RDMA and the “Invisible” Latency
In a multi-tenant cluster, you aren’t just managing compute; you’re managing a distributed shared state.
When a request is preempted on Node A and resumed on Node B, we don’t want to lose the KV cache. Recomputing a 10,000-token prompt is expensive and slow. The solution is KV Cache Transfer.
Moving 5GB of KV cache across a standard 1Gbps or even 10Gbps management network takes seconds—an eternity in LLM time. This is where RDMA (Remote Direct Memory Access) over InfiniBand or RoCE v2 becomes mandatory.
RDMA allows Node B to reach into Node A’s memory and pull the KV blocks directly, bypassing the CPU and the OS stack. A hardware-aware orchestrator knows the “Network Distance” between nodes. It will always prefer to migrate a request between two nodes in the same “Infiniband Island” to keep migration latency under 100ms.
Why “Hype” Meets Reality: The Shift from Throughput to Goodput
In the AI community, people love to talk about “Tokens Per Second” (TPS). But in a multi-tenant, production environment, the only metric that matters is Goodput: the number of successful completions delivered per dollar spent, within a specific latency SLA.
Without hardware-aware orchestration, your TPS might look great on paper, but your Goodput will be abysmal because:
- Tail Latency (P99): Requests are getting stuck behind massive batch recomputes.
- Cold Starts: Users are waiting 10 seconds for a LoRA to load from an S3 bucket because the orchestrator didn’t pick a “warm” node.
- The “Thundering Herd”: A spike in traffic causes multiple GPUs to OOM simultaneously, triggering a cascade of restarts.
By implementing PagedAttention and a custom, VRAM-aware control plane, engineering teams are seeing 3x to 5x increases in hardware utilization. That’s the difference between a project that’s a cost-sink and one that is economically viable.
The Observability Gap: Monitoring the Un-Monitorable
How do you know if your orchestration is working? Standard Prometheus metrics for “GPU Utilization” are misleading. A GPU can be at 100% utilization while doing nothing but swapping memory blocks back and forth.
To manage a multi-tenant cluster effectively, you need specialized telemetry:
- KV Cache Fragmentation Index: The ratio of logically free vs. physically usable blocks.
- LoRA Cache Hit Rate: How often the scheduler successfully finds a warm node.
- Preemption Frequency: How often the engine is forced to pause a request due to VRAM exhaustion.
- PCIe vs. NVLink Bandwidth: Monitoring if the bottleneck is the compute or the data movement.
Tools like NVIDIA’s DCGM (Data Center GPU Manager) are the starting point, but they must be integrated into a high-frequency feedback loop that the orchestrator can use to make real-time scheduling decisions.
Engineering the Future of the Cluster
We are moving toward a future where the distinction between “Compute” and “Memory” is increasingly blurred. With the advent of CXL (Compute Express Link), we may soon see clusters where a GPU can access a shared pool of memory that doesn’t live on its own PCB.
But until then, we are playing a high-stakes game of VRAM Tetris.
Managing fragmented VRAM and PagedAttention at scale isn’t just an infrastructure task; it’s a fundamental rethinking of how we build distributed systems. It requires a stack that is aware of the physics of the hardware—the speed of the electrons across the NVLink bridge, the size of the blocks in the KV cache, and the layout of the weights in HBM.
The winners of the LLM era won’t just be the ones with the most GPUs. They will be the ones who can squeeze every single token out of the memory they have, through intelligent, hardware-aware orchestration.
If you’re building in this space, remember: The model is the brain, but the orchestrator is the nervous system. Don’t neglect the nerves.