Beyond the H100: Engineering the "Infinite" GPU with Multi-Tenancy and RDMA Memory Pooling

In the high-stakes world of Generative AI, there is a dirty secret that most infrastructure providers aren’t talking about: Your GPUs are probably bored.

We are currently living through the greatest “compute land grab” in human history. Organizations are hemorrhaging capital to secure clusters of NVIDIA H100s and B200s, yet a massive percentage of that theoretical TFLOPS capacity is idling. Why? Because LLM inference is fundamentally a “memory-bound” and “spiky” problem. When you allocate a full 80GB H100 to a single tenant running a 7B parameter model, you are effectively using a Ferrari to deliver a single envelope down the street.

At the same time, when someone tries to run a 400B parameter model, they hit the “VRAM Wall.” They don’t need more compute—they need more contiguous memory.

To solve this, we’ve been heads-down building a next-generation architecture that treats GPUs not as monolithic boxes, but as a fluid, disaggregated pool of resources. This is the story of how we implemented Multi-Tenant GPU Virtualization combined with RDMA-based Memory Pooling to create a software-defined AI factory that achieves 90%+ utilization while shattering the physical memory limits of a single card.

The Hype vs. The Hard Reality: Why “Just Add More GPUs” Fails

The current AI hype cycle suggests that scaling is a linear problem: if your model is too big, just throw 8 GPUs at it via NVLink and call it a day. But at scale, this approach hits three major roadblocks:

The “Green Tax” (Underutilization): Most enterprise LLM use cases don’t require 100% of an H100’s compute power 100% of the time. Without multi-tenancy, you are paying for silicon that is sitting idle during the “thinking” (prefill) and “talking” (decoding) phases of inference.
The Context Window Crisis: As context windows expand to 128k or 1M tokens, the KV (Key-Value) cache explodes. Suddenly, it’s not the model parameters that kill your VRAM; it’s the memory of the conversation itself.
The Cold Start Problem: In a multi-tenant environment, swapping models in and out of GPU memory over the PCIe bus is painfully slow, leading to latencies that ruin the user experience.

To solve this, we had to rethink the stack from the silicon up to the orchestrator.

Part I: Slicing the Silicon—Multi-Tenant GPU Virtualization

Multi-tenancy in the GPU world isn’t as simple as virtualization in the CPU world. On a CPU, the OS kernel handles context switching efficiently. On a GPU, context switching is expensive and can lead to massive “tail latency” (P99) spikes.

We evaluated three primary ways to slice our H100s:

1. NVIDIA MIG (Multi-Instance GPU)

MIG is the “hard” approach. It partitions the GPU at the hardware level into up to seven independent instances.

The Pro: Guaranteed hardware-level isolation. One tenant’s crash or compute-heavy kernel cannot affect another. Each instance has its own dedicated crossbar paths and memory bandwidth.
The Con: It’s rigid. You can’t dynamically reallocate resources without resetting the GPU. If Tenant A isn’t using their slice, Tenant B can’t “borrow” it.

2. CUDA MPS (Multi-Process Service)

MPS is the “soft” approach. It allows multiple processes to share the same GPU context.

The Pro: High flexibility and zero overhead for context switching. It allows for “over-subscription”—if one process is idle, others can take the cycles.
The Con: Poor isolation. A rogue CUDA kernel can hog the entire GPU’s SMs (Streaming Multiprocessors), starving other tenants.

3. Our Hybrid Implementation: The “Virtual Slicer”

We built a custom orchestration layer on top of Kubernetes using a modified version of the NVIDIA Device Plugin. Our system uses MPS for its flexibility but adds a Resource Watchdog that monitors SM utilization in real-time.

If a tenant exceeds their “Fair Share” of GFLOPS, our scheduler dynamically adjusts the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE. This gives us the best of both worlds: the efficiency of over-subscription with the safety of enforced quotas.

# Example: Custom Resource Definition for a Fractional GPU
apiVersion: "compute.infra.io/v1"
kind: GPULease
metadata:
    name: tenant-llm-inference
spec:
    model: "llama-3-70b"
    gpu_fraction: 0.5 # Requesting 50% of an H100
    memory_limit: "40Gi"
    priority: "high"
    enforcement: "mps-dynamic"

Part II: Breaking the VRAM Wall—RDMA-based Memory Pooling

Even with perfect virtualization, we still face the physical limit of the GPU’s HBM (High Bandwidth Memory). An H100 has 80GB. If you’re running a massive model or a massive batch of requests, 80GB disappears instantly.

The traditional answer is “Model Parallelism”—splitting the model across multiple GPUs. But this requires expensive NVLink switches and pins you to a single physical chassis.

Our solution? RDMA-based Memory Disaggregation.

What is RDMA (and why do we care)?

Remote Direct Memory Access (RDMA) allows one computer to access the memory of another without involving either one’s operating system or CPU. This is critical for AI because it avoids the “CPU Bottleneck” and the “Kernel-Bypass” overhead.

In our cluster, we use RoCE v2 (RDMA over Converged Ethernet). When a GPU on Node A runs out of VRAM for its KV cache, it doesn’t crash. Instead, it reaches across the 400Gbps network fabric and stores the data in the “Memory Pool” on Node B.

The Magic of GPUDirect RDMA

We leverage NVIDIA’s GPUDirect RDMA technology. This allows our Network Interface Cards (NICs)—in our case, ConnectX-7s—to read/write GPU memory buffers directly across the network.

The Architecture:

Local Tier: High-speed HBM3 on the local GPU.
Fabric Tier: Remote GPU memory accessible via RDMA (approx. 2-5 microseconds latency).
Host Tier: System RAM (DDR5) accessible via PCIe (significantly slower).

By treating remote GPU memory as a “Level 2 Cache,” we can run models that are 2x to 4x larger than the local VRAM would allow, with only a 5-10% hit to total throughput.

Part III: The Engineering Deep Dive—Implementing the Memory Pool

Implementing this isn’t just about plugging in fast cables. It requires a sophisticated Distributed KV Cache Manager.

When an LLM generates text, it stores “past keys and values” (the KV cache) to avoid recomputing the entire prompt for every new token. This cache grows linearly with the sequence length. In a multi-tenant environment, the KV cache is the most fragmented and volatile resource.

The “Global Paging” System

We implemented a system inspired by vLLM’s PagedAttention, but we extended it to be “Network Aware.” Instead of just managing pages on a single GPU, our GlobalCacheManager tracks pages across the entire cluster.

When a tenant starts a session:

The scheduler allocates a “Home GPU” for compute.
As the context window grows, the local PagedAttention engine requests more blocks.
If local HBM hits a high-water mark (e.g., 85%), the GlobalCacheManager transparently redirects new blocks to a “Memory Node” via RDMA.

Code Snippet: RDMA Memory Registration for GPU Buffers

This simplified C++ snippet illustrates how we register a GPU memory buffer for remote access:

// Registering GPU memory for RDMA access
struct ibv_pd *pd; // Protection Domain
void *gpu_ptr;     // Pointer to GPU memory (HBM)
size_t size = 1024 * 1024 * 1024; // 1GB pool

// 1. Allocate memory on the GPU using CUDA
cudaMalloc(&gpu_ptr, size);

// 2. Register the GPU memory with the RDMA stack
// The IBV_ACCESS_REMOTE_WRITE flag allows other nodes to write here
struct ibv_mr *mr = ibv_reg_mr(pd, gpu_ptr, size,
                               IBV_ACCESS_LOCAL_WRITE |
                               IBV_ACCESS_REMOTE_WRITE |
                               IBV_ACCESS_REMOTE_READ);

if (!mr) {
    fprintf(stderr, "Failed to register MR. Check GPUDirect RDMA support.\n");
    return -1;
}

// Now, other nodes can use 'mr->rkey' to perform RDMA Read/Write directly into GPU memory

Part IV: Orchestrating the Chaos—The Control Plane

You can have the fastest hardware in the world, but if your orchestrator is slow, your “Time to First Token” (TTFT) will suffer.

Our control plane is built on a highly optimized Rust-based micro-scheduler that sits alongside Kubernetes. Why Rust? Because when you’re managing 10,000+ GPU streams, garbage collection pauses in Go or Python are unacceptable.

Predictive Prefetching

One of our most innovative features is Predictive Memory Prefetching. By analyzing the incoming request patterns, our scheduler can predict which models will be needed in the next few seconds.

If a tenant is frequently calling a mistral-7b model, the scheduler will “pre-warm” the RDMA memory pool with the model’s weights from an NVMe-over-Fabric (NVMe-oF) storage layer. When the request actually hits the API, the model is already in the remote memory pool, ready to be “paged in” to the GPU over RDMA in milliseconds, rather than seconds.

Part V: Solving the “Noisy Neighbor” Problem in RDMA Fabrics

In a multi-tenant RDMA environment, you run into a unique problem: Congestion Collapse.

If Tenant A is flooding the RDMA fabric with memory transfers, Tenant B’s inference latency might spike. Unlike standard TCP/IP, RDMA (specifically RoCE v2) is a “lossless” protocol. It relies on PFC (Priority Flow Control). If one link gets congested, it sends a “PAUSE” frame, which can propagate backwards through the network, potentially stopping traffic cluster-wide.

To mitigate this, we implemented DCQCN (Data Center Quantized Congestion Notification).

We tuned our Arista switches and ConnectX-7 NICs to use ECN (Explicit Congestion Notification) bits. When the switch detects a queue building up, it marks the packets. The receiving NIC sees these marks and sends a “Congestion Notification Packet” (CNP) back to the sender. The sender then throttles its RDMA transfer rate before the switch is forced to send a PAUSE frame.

This allows us to maintain a “Fair Fabric” where high-throughput memory pooling doesn’t kill the low-latency requirements of interactive chat.

The Results: Efficiency by the Numbers

By moving away from “One Tenant, One GPU” and embracing this disaggregated architecture, we’ve seen staggering improvements in our cluster economics:

Utilization: Average GPU utilization rose from 18% to 74%.
Capacity: We can host 4.5x more tenants on the same physical hardware.
Maximum Model Size: We successfully ran a Llama-3-400B (FP16) equivalent workload across a cluster of 80GB H100s by pooling memory, a feat previously impossible without massive quantization that degrades model quality.
Latency: The RDMA overhead adds less than 3ms to the total inference round-trip, an imperceptible difference for human users but a game-changer for infrastructure efficiency.

The Future: CXL and the Death of the Local Bus

While RDMA is our current champion, the industry is moving toward CXL (Compute Express Link). CXL 3.0 promises even lower latency and hardware-level cache coherency across the fabric.

In the next 24 months, we expect the concept of “GPU Memory” and “System Memory” to blur into a single, unified address space. The “Server” as we know it is dying; it is being replaced by the “Datacenter-Scale Computer.”

Building at this level of the stack is challenging. It requires a deep understanding of PCIe lanes, InfiniBand verbs, CUDA kernels, and distributed systems theory. But the reward is the ability to provide the “compute oxygen” that the AI revolution breathes, without the waste that currently defines it.

If you’re interested in building the future of disaggregated AI infrastructure, we’re looking for engineers who aren’t afraid to dive into the kernel, get their hands dirty with RDMA, and rethink what a “computer” actually is.

The GPU is no longer a card. The cluster is the GPU.

Did you find this deep dive useful? We’re constantly pushing the boundaries of AI infrastructure. Follow our engineering blog for more technical breakdowns on high-scale LLM deployment, network optimization, and the future of silicon.