The AI Infrastructure Problem No One is Telling You About

It’s perfectly normal if you think that memory chips are boring.

For decades now, they’ve been cheap and plentiful. What’s more, they’ve followed a familiar supply cycle. When demand for chips slowed, prices fell. When demand picked up, manufacturers added supply and the market balanced itself out.

At least, that’s how it used to work. But that world no longer exists.

Because the AI boom hasn’t just increased demand for computing power. It has also rewired the global memory market around a new kind of buyer that doesn’t care about price, timing or traditional supply cycles.

That’s why I recommended scooping up shares of Micron Technology Inc (Nasdaq: MU) in my February 2024 issue of Strategic Fortunes, saying “memory plays an essential role in the newest wave of AI-based online tools…”

Since then, shares of Micron have soared 156% in under two years.

But a lesser known result of this shift is a severe memory shortage. Which might seem like an obscure industry problem, but consumers are already starting to feel the pinch.

What worries me the most is that this isn’t shaping up to be a temporary squeeze. It looks like it’s structural.

And if you haven’t heard much about it, that’s not an accident.

The Memory Squeeze

When we talk about AI infrastructure, the conversation usually revolves around GPUs.

A GPU, or graphics processing unit, is a specialized chip designed to perform many calculations at the same time, which makes it ideal for training and running AI models.

It’s a big reason that Nvidia has become the poster child of the AI boom.

But GPUs are only part of the equation. Every AI model also depends on massive amounts of memory to function.

Training large models requires high bandwidth memory, or HBM, stacked directly next to GPUs. Running these models at scale also depends on enormous pools of DRAM, the same type of memory used in laptops, phones and business servers.

In other words, AI isn’t just compute hungry. It’s memory hungry.

And AI’s hunger for memory has started to break the market.

Over the last year, memory manufacturers have shifted production aggressively toward HBM because it commands far higher margins than traditional DRAM.

This decision makes perfect sense from a business standpoint. Because hyperscalers like Microsoft, Google, Amazon and Meta are willing to sign long-term contracts and pay almost any price to secure a consistent supply of memory.

But the unintended consequence of this shift is that conventional DRAM production was deprioritized.

That means there is far less DRAM available for everyone else today.

Inventories that were once measured in months have now collapsed to just a few weeks of supply. In some segments, DRAM stockpiles are down roughly 80% from a year ago.

In other words, a commodity market that used to be flexible is now being squeezed by a handful of companies building massive AI data centers.

That’s why you can’t really compare today’s situation to past chip shortages.

In earlier cycles, shortages were usually caused by forecasting mistakes or short-term demand spikes. For example, consumer electronics companies might overorder or the economy might slow down. But in those cases, inventories would eventually flood back into the system and prices would drop.

That release valve doesn’t exist this time.

And that’s because the buyers driving demand today are the biggest tech companies in the world. They’re all operating on multi-year roadmaps. And now that the U.S. has effectively launched a Manhattan Project for AI, they’re treating AI capacity as strategic infrastructure.

Governments and corporations alike have decided that artificial intelligence is too important to leave to chance. It must be built, secured and scaled as quickly as possible.

That means speed has become far more important than cost. In other words, time is our biggest constraint today.

And memory sits right in the middle of that bottleneck.

New memory chip plants take years to build and bring fully online. HBM production is even more specialized, with tight integration between chipmakers, packaging technologies and GPU designs.

And even when new capacity comes online, the first customers in line will be the same hyperscalers that reshaped the market in the first place.

That’s why major suppliers are now openly warning that memory shortages could last well into the second half of the decade.

But here’s the thing.

I don’t believe consumers will be told there’s a memory shortage at all. Not in the mainstream press.

They’ll simply notice that their next laptop costs more. Or that the basic storage and memory haven’t improved. Or that companies are either delaying tech upgrades or charging more for them. Likely both.

You see, memory is embedded in almost every piece of modern electronics. This means when memory gets more expensive, you won’t see it come up as a single line item. It’ll be diffused into the cost of the entire system.

And that’s why most people won’t even realize what’s driving higher prices.

Here’s My Take

Artificial intelligence is often described as deflationary technology.

Over time, that’s probably true because AI will automate work and increase productivity across the economy.

But the path to that future relies on physical infrastructure. And infrastructure booms have a history of creating short to medium-term inflation along the way.

The irony is that the race to deploy AI as fast as possible could temporarily push costs higher, even as the software promises long-term efficiency gains.

That’s not a reason to be bearish on AI. But it is a reason to keep an eye on how those costs get passed on to consumers and businesses.

Memory used to be a background component that was often taken for granted. But AI has made it a strategic asset that could reshape pricing across the economy.

That means memory chips are no longer boring.

And I don’t expect them to become boring again anytime soon.

Regards,

Ian KingChief Strategist, Banyan Hill Publishing

Editor’s Note: We’d love to hear from you!

If you want to share your thoughts or suggestions about the Daily Disruptor, or if there are any specific topics you’d like us to cover, just send an email to [email protected].

Don’t worry, we won’t reveal your full name in the event we publish a response. So feel free to comment away!