The ongoing tightness in DRAM supply has pushed AI giants to the point of dismantling old servers to recycle memory.
Nikhil Cherian, Senior Director of Supply Chain Infrastructure at Google, recently revealed that the AI industry has rapidly shifted from being compute-constrained to memory-constrained, with high-performance memory currently accounting for approximately 75% of the bill of materials cost for a single AI server.
According to Cherian, to break through the memory bottleneck, Google is making efforts on both the hardware and software fronts. On the hardware side, it has even established an internal recycling supply chain, dismantling decommissioned servers to recover usable DDR4 memory modules.
Cherian admitted that Google specifically designed hardware adapters to connect previous-generation DDR4 memory to new AI servers and is actively re-importing decommissioned servers to extract their DDR4 modules.
This means that new AI servers, which were originally intended to use DDR5, have had to downgrade to recycled DDR4 in some cases.
On the software side, Google continues to optimize underlying libraries, model architectures, and KV Cache compression technologies in an attempt to reduce memory consumption per unit of computing power.
This year, Google launched two TPU ASICs: the TPU v8t for training and the TPU v8i for inference. Each TPU v8i chip is equipped with 288GB of HBM3e, providing 8.6 TB/s of memory bandwidth, and integrates 384MB of on-chip SRAM to store active KV caches, reducing the need for queries to external system memory.
TPU v8i servers use Google's self-developed Axion processors, based on the Arm architecture, to replace traditional x86 host CPUs, relying on high-speed DDR5 memory architecture to handle host-level tasks such as data preprocessing.
Even so, Google has been forced to substitute recycled DDR4 for DDR5 in certain scenarios, highlighting the severe extent of the current DRAM supply shortage.