Photons over Packets: How Google’s Jupiter v2 Killed the Spine-Leaf Architecture

Imagine you’re tasked with building a brain. Not a metaphorical one, but a physical, distributed system capable of training the world’s largest Large Language Models (LLMs). You quickly realize that your biggest enemy isn’t the compute power of the TPUs or GPUs—it’s the cables. Specifically, it’s the power consumed and the heat generated by moving petabits of data between those chips.

For two decades, the networking world has worshipped at the altar of the Clos topology (the ubiquitous Spine-Leaf architecture). It was the gold standard: predictable, scalable, and robust. But at Google’s scale, the Spine-Leaf architecture hit a wall—a literal physical wall of power consumption and cost.

Enter Jupiter v2.

This isn’t just a minor upgrade. It is a fundamental shift in how we think about data center fabrics. By replacing traditional electrical packet switching at the core with Optical Circuit Switching (OCS) and custom MEMS (Micro-Electro-Mechanical Systems), Google has effectively “killed” the spine-leaf model as we know it.

In this deep dive, we’re going to look under the hood of the Jupiter v2 fabric, explore the physics of the Apollo OCS, and understand why the future of the hyperscale data center isn’t electronic—it’s photonic.

The Ghost of Networks Past: Why Spine-Leaf Failed the AI Era

To understand why Jupiter v2 is such a big deal, we have to look at the limitations of what it replaced.

In a traditional Spine-Leaf (Clos) network, every leaf switch connects to every spine switch. If you want to scale, you add more spines or more tiers. This creates a “non-blocking” fabric where any node can talk to any other node at full line rate.

While elegant, this architecture has three massive flaws at Google-scale:

1. The O-E-O Bottleneck: Every time a signal moves from a fiber optic cable into a switch, it undergoes an Optical-to-Electronic-to-Optical (O-E-O) conversion. You take the light, turn it into electrons to let a silicon chip decide where it goes (packet switching), and then turn it back into light to send it to the next hop. This process is incredibly power-hungry and expensive.
2. Radix Limits: As you increase the number of ports (radix) on a switch to support more nodes, the complexity and power draw of the switching silicon grow exponentially.
3. Rigid Topologies: Once you cable up a Spine-Leaf network, that’s your topology. If your AI workload requires a specific ring or torus configuration for optimal “All-Reduce” operations, you’re stuck with whatever the physical wires dictate.

Google realized that in a world where AI training dominates the fabric, packet-by-packet routing at the core is overkill. Most of the data flowing through the core isn’t a random assortment of tiny web requests; it’s massive, long-lived streams of data between clusters of TPUs.

The Star of the Show: The Apollo Optical Circuit Switch

At the heart of Jupiter v2 lies a device that looks more like something out of a physics lab than a data center: the Apollo OCS.

Unlike a traditional switch that uses ASICs (Application-Specific Integrated Circuits) to read packet headers, Apollo doesn’t care about packets. In fact, it doesn’t even “see” the data. It works entirely in the photonic domain.

The Physics of MEMS

The Apollo switch uses a grid of 176 tiny mirrors mounted on micro-electro-mechanical systems (MEMS). When Google’s Software Defined Networking (SDN) controller wants to create a path between two points in the data center, it sends a command to the Apollo switch. The switch then physically tilts these mirrors using electrostatic actuators to reflect a beam of light from an input fiber directly into an output fiber.

Think about the implications of this:

• Zero Power for Data: Once the mirrors are tilted into position, they stay there. The data (the photons) passes through the switch at the speed of light without any electronic processing. The power required to maintain the circuit is nearly zero.
• Protocol Agnostic: Because it’s just mirrors, the OCS doesn’t care if you’re running 100G, 400G, 800G, or 1.6T Ethernet. You can upgrade the edge transceivers without ever touching the core switch.
• Ultra-Low Latency: There is no “buffer bloat” or “queueing delay” in an OCS. The latency is literally just the time it takes for light to travel the length of the fiber.

Wavelength Division Multiplexing (WDM) Integration

Google didn’t just stop at mirrors. They integrated WDM directly into the fabric. Instead of running one stream of data per fiber, they multiplex multiple wavelengths (colors) of light onto a single strand. By combining WDM with Apollo, Google can dynamically reconfigure not just where the bandwidth goes, but how much bandwidth is allocated to specific parts of the cluster.

From Spine-Leaf to “Direct-Connect” Topology

In Jupiter v1, Google used a standard Clos model. In Jupiter v2, they moved to a direct-connect architecture enabled by the OCS.

In this new world, the OCS acts as a massive, programmable “patch panel.” Instead of having layers of spine switches that constantly process packets, the leaf switches (which Google calls ToRs or Top-of-Rack switches) connect directly to the Apollo OCS.

The Dynamic Topology Advantage

This is where the “end of spine-leaf” narrative gets real. In a spine-leaf setup, the topology is static. In Jupiter v2, the topology is fluid.

If the SDN controller detects that a specific set of TPU racks are performing a heavy “All-to-All” communication pattern for a model like Gemini, it can reconfigure the Apollo mirrors in real-time to create a direct optical path between those specific racks.

This reduces the “hop count.” In a traditional network, a packet might go: TPU -> Leaf -> Spine 1 -> Spine 2 -> Leaf -> TPU. In Jupiter v2, it’s often just: TPU -> Leaf -> OCS -> Leaf -> TPU.

The result? Google reported a 40% reduction in power consumption and a 30% reduction in capital expenditure (CapEx) while increasing the total fabric capacity by a factor of 5.

The Brain: Orion SDN and Topology Engineering

You might be asking: “If the mirrors are physically moving, doesn’t that break the network?”

If you just tilted a mirror while data was flowing, you’d drop packets. This is where Orion, Google’s custom Software Defined Networking stack, comes into play. Orion is the orchestral conductor for the Jupiter v2 fabric.

The “Make-Before-Break” Transition

When Orion decides it needs to reconfigure the network topology (perhaps to route around a fiber cut or to optimize for a new ML job), it follows a sophisticated sequence:

1. Path Pre-computation: Orion calculates the new optimal topology.
2. Traffic Draining: Using BGP or a custom flow-control protocol, Orion gracefully moves traffic away from the circuits that are about to be reconfigured.
3. Mirror Tilting: The Apollo OCS tilts its MEMS mirrors. This takes about 10-100 milliseconds.
4. Signal Validation: The OCS uses a built-in laser monitoring system to ensure the mirrors are perfectly aligned and the optical signal is clean.
5. Traffic Injection: Orion begins routing traffic over the new physical path.

Code Snippet: Conceptual Topology Update

While we don’t have the internal Google C++ source code, we can model how an SDN controller like Orion might interface with an OCS via an API:

class OrionController:
    def reconfigure_fabric(self, workload_type):
        # 1. Determine optimal graph for the workload (e.g., Ring for All-Reduce)
        new_topology = self.compute_optimal_graph(workload_type)

        # 2. Identify circuits to be moved
        circuits_to_change = self.diff_topology(self.current_topology, new_topology)

        for circuit in circuits_to_change:
            # 3. Drain traffic (Shift flows to alternate paths)
            self.drain_traffic(circuit.id, graceful_timeout=50) # ms

            # 4. Command Apollo OCS to tilt MEMS mirrors
            apollo_switch.set_mirror_angle(circuit.input_port, circuit.new_output_port)

            # 5. Verify optical power levels
            if apollo_switch.get_optical_power(circuit.new_output_port) > THRESHOLD:
                self.update_routing_table(circuit.id, status="ACTIVE")
                print(f"Circuit {circuit.id} reconfigured successfully.")
            else:
                self.trigger_alarm("Optical Alignment Failed")

# The OCS handles the physical layer, while Orion handles the logical layer.

Scale and Performance: The Raw Numbers

To appreciate the engineering feat here, we have to look at the scale. A single Jupiter v2 deployment can support:

• Over 1.3 Petabits per second of aggregate bisection bandwidth.
• Thousands of nodes (TPUs/CPUs) connected with near-zero core latency.
• Inter-cluster connectivity that spans across entire data center buildings using the same OCS technology.

But the most impressive number isn’t the bandwidth—it’s the availability. Usually, adding mechanical parts (moving mirrors) to a network sounds like a reliability nightmare. However, Google found that the Apollo OCS is actually more reliable than traditional electrical switches. Why? Because the OCS is passive. It doesn’t have fans that fail, it doesn’t have complex ASICs that overheat, and it doesn’t suffer from the “soft errors” (bit flips) that plague high-density silicon.

Why the Industry is Obsessing Over This

The hype around Jupiter v2 and OCS isn’t just “Google being Google.” It addresses a looming crisis in the semiconductor industry: The End of Moore’s Law for Power.

While we can still cram more transistors onto a chip, we can’t easily get the heat out of them. A modern high-end network switch ASIC can pull upwards of 500W to 900W just for the chip itself. In a massive data center with thousands of these switches, the power bill for the network alone starts to rival the power bill for the compute.

By moving to OCS, Google has effectively bypassed the power-law of switching silicon. They’ve proven that for the “heavy lifting” of data center traffic, we don’t need to process every packet. We just need to point the light in the right direction.

The TPU v4 Connection

The Jupiter v2 fabric was the secret sauce that made the TPU v4 clusters so dominant. By using OCS, Google allowed users to define their own topology (e.g., a 3D Torus) via software. If a single TPU in a 4,096-node cluster fails, the OCS can simply “patch” it out of the network, allowing the training job to continue with minimal downtime. In a traditional wired spine-leaf network, that failed node would create a “hole” in the topology that is much harder to route around efficiently.

Engineering Curiosities: The Challenges They Overcame

It wasn’t all smooth sailing. Building Jupiter v2 required solving problems that traditional network engineers never have to deal with.

1. Dust is the Enemy: When you’re dealing with mirrors that are micrometers wide, a single speck of dust can block an entire 400G link. Google had to develop “ultra-clean” manufacturing and maintenance protocols for the Apollo units.
2. The Signal “Flash”: When mirrors move, the receiving transceivers briefly lose signal. Standard Ethernet transceivers are designed to panic when this happens (Loss of Signal / Link Down). Google had to modify the firmware on their transceivers and the “Orion” stack to ignore these millisecond-level outages during planned reconfigurations.
3. Precision Calibration: The mirrors don’t just have an “on/off” state. They have to be tilted at incredibly precise angles to ensure the light hits the center of the receiving fiber (which is only about 9 micrometers wide for single-mode fiber). This requires a continuous feedback loop using “pilot tones” to keep the mirrors aligned despite vibrations or thermal expansion.

The Verdict: Is Spine-Leaf Truly Dead?

In the context of the Hyperscale AI Data Center, the answer is a resounding yes.

The traditional Spine-Leaf model assumes that the cost of an electrical switch port is low enough to justify the “any-to-any” flexibility. But at the petabit scale, the “tax” of O-E-O conversion—in terms of dollars, watts, and nanoseconds—is simply too high.

Jupiter v2 represents a “de-layering” of the network. By moving the complexity to the Software Defined Networking layer and the physical layer to photons and mirrors, Google has created a fabric that is:

1. More flexible than wires.
2. Cheaper than silicon.
3. Faster than electricity.

As we move toward the era of 1.6T and 3.2T links, the “Apollo” approach will likely become the blueprint for any organization building AI infrastructure at scale. The spine-leaf architecture served us well during the rise of the cloud, but the AI era belongs to the photons.

Key Takeaways for the Modern Engineer:

• The Bottleneck has Shifted: It’s no longer about how fast your CPU is; it’s about the cost of moving data between those CPUs.
• Topology Matters: The ability to dynamically change your network shape to match your algorithm is a massive competitive advantage.
• Efficiency is Scalability: Power is the ultimate constraint in the modern data center. If you save 40% on power, you can deploy 40% more compute.

Jupiter v2 isn’t just a network; it’s a testament to the idea that sometimes, the best way to move forward is to stop processing and start reflecting. The future is bright, and it’s moving at the speed of light.