We’ve all seen the charts. Large Language Models (LLMs) are getting smarter, context windows are expanding to millions of tokens, and Retrieval-Augmented Generation (RAG) has become the gold standard for enterprise AI. But there is a dirty secret in the engineering trenches that no one likes to talk about: The Memory Wall is hitting a breaking point.
If you are building a production-grade RAG system for a Fortune 500 company, you aren’t dealing with a few thousand PDF snippets. You are dealing with terabytes of embeddings. We’re talking about billions of vectors, each representing a high-dimensional slice of corporate knowledge.
When you try to run a K-Nearest Neighbor (k-NN) search across 10 terabytes of vector data, the bottleneck isn’t your H100’s FLOPS. It’s not the complexity of your transformer. It’s the sheer physics of moving bits from a DRAM stick to a processor. In the world of high-performance vector databases, data movement is the tax that kills performance.
Today, we are diving deep into the architecture of Near-Memory Processing (NMP)—the engineering frontier where we stop bringing the data to the compute and start bringing the compute to the data.
The Math of the Bottleneck: Why Your Vector DB is Stalling
To understand why we need NMP, we have to look at the “Data Movement Tax.” Let’s do some quick back-of-the-envelope math.
Imagine a vector database containing 1 billion vectors. If we use OpenAI’s text-embedding-3-small model, each vector has 1,536 dimensions. Using FP32 (4 bytes per dimension), a single vector is ~6 KB.
- 1 Billion Vectors x 6 KB = 6 Terabytes of raw data.
In a standard von Neumann architecture, to perform a similarity search (like a Cosine Similarity or Euclidean Distance), the CPU or GPU must pull these vectors from DRAM into its local caches (L1/L2/L3 or VRAM) to perform the MAC (Multiply-Accumulate) operations.
The peak bandwidth of a high-end DDR5 memory channel is roughly 60 GB/s. To scan 6 TB of data just once:
- 6,000 GB / 60 GB/s = 100 seconds.
Even with multi-channel configurations, you are looking at multi-second latencies for a single exhaustive search. In an LLM inference pipeline, where the user expects a response in under 500ms, a 100-second (or even 2-second) vector lookup is an eternity.
Yes, we use Approximate Nearest Neighbor (ANN) algorithms like HNSW (Hierarchical Navigable Small Worlds) or IVFPQ (Inverted File Product Quantization) to skip the exhaustive scan. But at the terabyte scale, even these algorithms suffer from extreme cache thrashing and TLB (Translation Lookaside Buffer) misses. The pointers in an HNSW graph are scattered randomly across that 6TB address space, forcing the memory controller to jump between rows and columns constantly, destroying row-buffer locality.
What is Near-Memory Processing (NMP)?
Near-Memory Processing is a paradigm shift. Instead of treating memory as a “dumb” storage bin, we embed specialized logic (FPGAs, ASICs, or even small RISC-V cores) directly onto the memory module or within the memory controller’s logic layer.
In an NMP-equipped vector database, the host CPU doesn’t say “Give me vector #8,291,001.” Instead, it sends a high-level command: “Here is a query vector; tell me which of the 10 million vectors in your local DRAM bank are the top 10 most similar.”
By doing the heavy lifting—the dot products and distance calculations—right next to the memory cells, we achieve three things:
- Massive Internal Bandwidth: The bandwidth inside a memory chip is orders of magnitude higher than the bandwidth across the bus to the CPU.
- Reduced Latency: We eliminate the round-trip delay of the PCIe or memory bus.
- Energy Efficiency: Moving data across a PCB is the most energy-expensive part of computing. NMP cuts energy consumption by up to 80% for data-intensive tasks.
The Architecture: CXL and the “Logic Layer”
To build a terabyte-scale NMP system, we rely on two emerging technologies: CXL (Compute Express Link) and HBM3 (High Bandwidth Memory) with Integrated Logic Layers.
1. The CXL 3.0 Fabric
CXL is the game-changer. It’s an open standard interconnect built on top of PCIe Gen5/Gen6 that allows for cache-coherent memory sharing.
In a traditional setup, the CPU owns the RAM. In a CXL-enabled NMP setup, we can have a Memory Pooling architecture. We treat a massive rack of memory as a single, addressable fabric. An NMP accelerator sits on the CXL bus, acting as a “smart agent” that can traverse the memory fabric without waking up the main CPU.
2. The Logic Layer in HBM
HBM3 isn’t just stacked DRAM. It includes a “Base Logic Die” at the bottom of the stack. Companies like Samsung and SK Hynix are now experimenting with placing SIMD (Single Instruction, Multiple Data) engines directly into this logic die.
Imagine an HBM stack where the base die has 32 tiny ALU (Arithmetic Logic Units). When the vector DB starts a search, these 32 ALUs process 32 different vector dimensions in parallel, pulling data from the DRAM stacks directly above them through thousands of TSVs (Through-Silicon Vias).
Engineering the Software Stack: From Python to Bitstreams
You can’t just run a standard pip install faiss and expect it to work with NMP hardware. The software abstraction layer has to be completely reimagined.
The Problem with HNSW on NMP
HNSW is the most popular ANN algorithm, but it’s an NMP nightmare. It relies on a multi-layered graph where each node is a vector. Searching the graph involves “hopping” from one memory address to another based on the results of the previous distance calculation.
- The Issue: This is sequential logic. NMP thrives on parallel scans.
The Solution: Product Quantization (PQ) and NMP
Product Quantization is much better suited for NMP. PQ breaks a vector into sub-vectors and quantizes them into “codes.” To find the distance between a query and a quantized vector, you perform a series of lookups in a small table.
Here’s how we architect the NMP kernel for PQ-based vector search:
// A simplified conceptual NMP Kernel for Vector Distance
// This would be implemented in RTL (Verilog/HLS) on the NMP logic die.
void nmp_vector_search(
float* query_vector,
uint8_t* compressed_database,
float* lookup_table,
float* results_out
) {
// Each NMP processing element (PE) handles a chunk of the DB
#pragma HLS UNROLL factor=16
for (int i = 0; i < VECTORS_PER_BANK; i++) {
float distance = 0;
for (int m = 0; m < SUB_VECTOR_COUNT; m++) {
// Read the code from memory
uint8_t code = compressed_database[i * SUB_VECTOR_COUNT + m];
// Look up the pre-computed distance in the logic die's local SRAM cache
distance += lookup_table[m * 256 + code];
}
// Write result to local "Top-K" buffer if it's a candidate
update_top_k(i, distance);
}
}In this architecture, the Lookup Table stays in the logic die’s fast SRAM. The Compressed Database is streamed directly from the DRAM cells to the ALU. The CPU is never involved in the loop. It only receives the final top_k results.
Mitigating the “Cold Start” and Persistence Problems
One of the biggest hurdles in terabyte-scale vector databases is Indexing Time. Building an index for 10 billion vectors can take days on standard hardware.
With NMP, we can parallelize index construction. Because the NMP cores are integrated into the memory, they can perform the “k-means clustering” required for Product Quantization in parallel across all memory banks. Instead of one massive CPU trying to cluster 6TB of data, you have 128 NMP units each clustering their local 48GB slice of the data.
Persistence with NVMe over CXL
At the terabyte scale, you cannot lose your index if the power goes out. We are seeing a convergence of Storage-Class Memory (SCM) and NMP. By using technologies like Optane (RIP) or the newer CXL-based Persistent Memory, we can keep the vector database “Warm.”
When the system boots, the NMP controller performs a consistency check on the vector graph stored in persistent CXL memory, and within milliseconds, the database is ready for queries—no loading from SSD to RAM required.
The Hype vs. Reality: Where are we now?
If you listen to the AI hardware startups (Groq, Etched, Tenstorrent, etc.), they’ll tell you that the GPU is dead. That’s hype. The GPU is still the king of training and dense compute.
However, for Vector Databases and RAG, the GPU is actually overkill and inefficient. A vector search is “Compute-Bound” only for a split second; it is “Memory-Bound” for 99% of the operation.
The Reality:
- Samsung’s HBM-PIM: Samsung has already demonstrated functional prototypes of HBM-PIM that show a 2x performance gain in T5 (Text-to-Text Transfer Transformer) models.
- The CXL 3.1 Specification: This was recently finalized, adding features specifically for “Fabric-Attached Memory,” which is the foundation for rack-scale vector databases.
- Vector DB Native NMP: We are starting to see “Smart SSDs” and “Smart NICs” (DPUs) that can perform filtering and basic vector distance calculations before the data even hits the host’s RAM.
Designing for the “Long Tail” of Data
In an enterprise LLM context, you often have a “power law” distribution of data. 20% of your vectors (the most recent or most popular docs) are hit 80% of the time.
A high-performance architecture for this looks like a Tiered Vector Store:
- L1 (GPU VRAM): The most frequent 10,000 vectors.
- L2 (NMP-enabled CXL RAM): The “hot” 1-10 Terabytes.
- L3 (Smart NVMe SSDs): The “cold” Petabyte-scale archives.
By using NMP at the L2 layer, we eliminate the massive latency spike that occurs when a query misses the GPU cache and has to fall back to the CPU. We call this “Smoothing the Latency Tail.”
The Engineering Curiosity: How do we handle Dimensionality?
One of the most interesting challenges in NMP is the Dimensionality Explosion. As models like Gemini and GPT-4 move toward even larger embedding sizes (4096+), the “lookup table” for Product Quantization grows.
If the lookup table exceeds the size of the NMP unit’s local SRAM, we hit a mini-version of the memory wall inside the chip.
Engineers are solving this through Vector Sub-sampling and Binary Quantization. By converting a 1536-dimensional float vector into a 1536-bit string (where each bit is 1 or 0 based on whether the value is above/below a threshold), we can use Hamming Distance.
Hamming Distance is just a XOR followed by a POPCNT (Population Count). These are incredibly cheap at the hardware level. An NMP logic die can process billions of XOR/POPCNT operations per second with almost zero heat. This allows for a “first-pass” filter at incredible speeds, followed by a high-precision FP32 re-ranking of only the top 100 candidates.
Building the Future of Retrieval
We are moving away from the “CPU-centric” view of the world. In the era of terabyte-scale LLM inference, the memory is the processor.
Architecting for Near-Memory Processing requires a deep stack understanding:
- You need to understand the electrical constraints of CXL lanes.
- You need to design ANN algorithms that favor sequential scans over random-access pointers.
- You need to write custom kernels that can be synthesized into hardware logic.
For those of us building the infrastructure for the next generation of AI, the challenge is clear: The data is too heavy to move. So, we must teach the memory how to think.
The next time you query a RAG system and get a lightning-fast, highly accurate response from a corpus of a billion documents, remember—there’s a good chance that query never actually “hit” the CPU. It was resolved in the silent, buzzing circuits of an NMP-enabled memory bank, right where the data lives.
Are you working on scaling vector databases or implementing CXL-based memory pooling? Let’s talk in the comments. We’re particularly interested in how people are handling HNSW graph partitioning across non-uniform memory access (NUMA) domains in CXL fabrics.