It is an exciting time to be a token consumer. Competition is heating up, usage resets are being doled out, and the battle for your tokens now extends beyond the OpenAI-Anthropic duopoly. Fireworks alone is processing over 40T tokens per day—2x the OpenAI API’s volume at the end of March.

However, major FUD has also emerged as a result of open model success: if open models stay capable enough relative to the closed frontier at a fraction of the cost, won't the model layer become commoditized? This outcome would obviously be disastrous for frontier lab margins. For full details on Anthropic and OpenAI’s financials, see our Tokenomics Model.

To project how the open vs closed capability gap will progress in the future, we first need to measure the past. Naively, you might pick a single set of benchmarks to measure all historical models, but this is a mistake. Every benchmark is a product of a particular era. When someone creates a new benchmark, their goal is to discern differences in model capabilities at the time. If they’re successful, the model makers will climb said benchmark until it becomes saturated. Once that happens, everyone stops caring about the benchmark, and the cycle repeats.

There have been three eras thus far in the history of LLMs: early scaling, reasoning, and agentic. Each era represented a step-function increase in model utility, and rather than trying to plot a single continuous trend, we believe it’s better to evaluate the models and benchmarks from each era individually.

When viewed this way, it becomes clear that the open vs. closed gap moves in cycles. At the start of each era, a frontier lab completes some promising research, trains an impressive model, deploys it at scale to their users, and jumps ahead. Then, other labs identify the key advances, reverse-engineer what the frontier lab is doing, replicate them in their own models, and close the gap. Nothing stays secret forever—especially when you factor in distillation. It’s just a question of how long it takes.

To answer this question, we took all the relevant models from each era and ran a curated set of benchmarks to get a composite capability score. The result is a clear trend: with each generation, open-source models take half as long to catch up to the first closed-source model of the era.

Of course, benchmarks don’t tell the full story, and we’ll highlight all the relevant caveats below. Finally, we’ll extend this analysis into the future, and explain why it’s less bearish frontier models than you might initially think.

How we measured

Here’s an overview of the models and benchmarks we selected for each era:

Picking a single SOTA closed and open model at a particular time is subjective, but our selections reflect the general consensus among AI experts. In cases where there’s debate—e.g. Fable 5 vs GPT 5.6 today—we were conservative and tested both.

For benchmarks, we relied on a combination of personal taste and popularity. Humanities Last Exam (HLE), for example, is known to have lots of issues, but was also truly one of the defining benchmarks of the reasoning era with no close substitutes. SWE-bench Pro, on the other hand, is similarly popular and problematic, but also closely approximated by DeepSWE.

Most of the benchmark scores here we ran ourselves using Prime Intellect's evaluation stack, specifically their environments hub and the evals harness included in Prime-RL. The rest come from runs by our friends at Artificial Analysis and Datacurve's DeepSWE leaderboard. Open models were served the way they would have been at release: vLLM versions, hardware that was in use at the time, and sampling settings from the model card. For closed models, we ran against their pinned API versions. Where our numbers share a chart with third-party values, we matched their rulesets.

We’d like to give a huge thank you to Florian Brand (@xeophon) from Prime Intellect for helping us pick benchmarks/models, implement evals, and check for correctness.

Era 1 | Early scaling (2022-2024)

It’s June 2023. The world is reckoning with ChatGPT, and Mark Zuckerberg just agreed to fight Elon Musk at the Colosseum. But while Zuck is training jiu-jitsu and doing Murphs, his company is doing some training of their own. FAIR is about to push past the Mistral exodus and other drama, and successfully ship Llama-2-70B. The first open model that approached the frontier.

How far behind the frontier was it? Four benchmarks helped define SOTA at the time: GSM8K, HumanEval, TriviaQA, and MMLU-Pro:

These benchmarks are representative of what the frontier models were capable of at the time. Simple multiple choice questions, word problems, and programming problems scoped to single functions. How times have changed! Here is how Llama-2 stacked up against GPT-3.5 Turbo in a cage match of their own:

To account for differences in benchmark difficulty, we normalized the scores. Each era’s best result is set to 100, and every other model is scored relative to that. The composite score represents the equal-weight average of the four: 75.7 for GPT-3.5 Turbo on the frontier, and 39.9 for Llama-2-70B. A measured, but considerable gap. This initial lag creates the storyline for the rest of the era: Mixtral-8x7B released in December 2023 created momentum towards GPT-4 capability, only to have GPT-4 Turbo and GPT-4o race ahead:

It took until the Llama-3.1-405B release in July 2024 for open models to close the GPT-3.5 Turbo gap, with a composite score of 86. The last frontier model, GPT-4o, was matched in capability by DeepSeek V3 in December 2024, scoring 95.5 and 94.1 respectively. Qwen2.5-72B landed within striking distance of GPT-4o at a sixth of the 405B parameter count, pre-trained on 18T tokens.

This is the first instance of the gap closing. Through this era, we didn’t see the frontier rise much beyond the capabilities of GPT-4, but this was mostly due to priorities: Turbo and 4o were built to make GPT-4 cheaper and faster, not smarter.

Meanwhile, OpenAI had been working towards a different kind of model. The process-reward paper and Noam Brown hire both pointed at reasoning, and by mid-2024 every major lab was publishing test-time-compute research. Seven weeks after 405B, on September 12 2024, OpenAI shipped o1-preview: a model that sparked a new era of innovation.

Era 2 | Reasoning (2024-2025)

o1 reset the choice of benchmarks, along with the gap. The elementary evals from Era 1 were no longer difficult enough to test o1’s full abilities. Grade school math problems were replaced by the AIME. Scale AI collected some of the most esoteric, PhD-level multiple choice questions in the world and provocatively called it Humanity’s Last Exam.

It’s hard to overstate how important the release of o1 was in tech circles. Many consider it the day “we knew for sure we’d get AGI.” However, unlike Llama-2-70B vs GPT-4, the gap between open vs closed source started off much smaller during Era 2. The culprit? A little known model called DeepSeek R1.

A 12.1 point gap vs 35.8 at the start of the previous era. The market puked in response. Fortunately, the AI capex trade quickly recovered, and the "we're so back" open model momentum established by R1 was soon squashed by Meta's Llama-4 Maverick.

Gemini 2.5 Pro and o3 continued to push the reasoning frontier, and the R1-0528 checkpoint closed the initial gap in May 2025 with a score of 78. An 8.5 month window to close a 12.1 point gap:

Notably absent from the charts so far is Anthropic. Their model cards reported these benchmarks like everyone else's, but they never fought for the top of the leaderboard in this era. While OpenAI and Google traded crowns, Anthropic was turning Claude into the default coding agent. This set the terms for the next era: the benchmarks that now matter run in a terminal.

Era 3 | Agentic (2025 - Today)

Prior to Claude Code, agents had their moments (like Cognition’s viral demo of Devin in March 2024), but Anthropic was the first to nail a model + harness product—and it paid off. Since the general release of Claude Code in May 2025, Anthropic has added north of $65B in ARR. For in-depth Anthropic and OpenAI ARR projections, see our Tokenomics Model.

With agents came yet another new set of benchmarks. Fancy math problems were no longer the best test of model capabilities. Instead, people wanted to know how well models could write code, do web search, and generally use a computer like a human.

Terminal-Bench 2.1, BrowseComp-Plus, 𝜏³-banking, and DeepSWE cover the long-horizon work agents are used for today: software engineering, deep research, and knowledge work. We also picked benchmarks that skew newer by design to limit memorization.

Most AI experts consider Opus 4.5 the official start of the agentic era due to the reliability of the model. Interestingly, GPT-5.2 (OpenAI’s flagship at the time) performed better on our benchmark suite, but this didn’t correspond to a better user experience. The full agentic product (model + harness) was now what mattered, and Anthropic had been laser-focused on iterating towards a harness that excelled at general agentic work. Codex, in contrast, was comparatively crude, and OpenAI was simultaneously pursuing side quests like web browsers.

Model releases also compressed between the two frontier labs. OpenAI and Anthropic created their duopoly by releasing a model every 51 days on average throughout this era. Compared to the 213 and 120 day release averages throughout Era 1 and Era 2, respectively, this is a massive speed up.

Yet despite the massive explosion in economic value created by frontier models, the gap closed faster in Era 3 than either era before it. Kimi K2.6 surpassed Opus 4.5 with a score of 56.3 in 4.8 months, and GLM-5.2 cleared GPT-5.2 with a score of 72.4 in 6 months. The trend of the closing time halving with each subsequent era is remarkably consistent.

Looking forward

So, what does this all mean for the future of closed vs open source models?

First, we want to acknowledge that benchmarks are not the end all be all. Kimi K3 may score higher than Fable 5 on our curated composite, but we still prefer using Fable at SemiAnalysis for our day to day work. This is partly because Anthropic has done a better job productizing their model via things like Claude Code and Claude Tag, but also largely because benchmarks aren’t a perfect proxy for real work. This is especially true for public benchmarks, which model makers can easily hill climb by simply creating a bunch of RL environments that closely mimic the benchmark tasks.

Second, you may argue that the closing time for Era 3 is artificially deflated due to Anthropic and OpenAI spending more time on safety testing than Moonshot and Zhipu, but this is not a new phenomenon. GPT-4, for example, finished training 218 days before release. Even if we assume Mythos finished training in mid February, that’s still only a 114 day delay before the Fable release.

The Upcoming Era

Read more

This all translates into CS-4 being able to double the tokens/s/user per wafer from CS-3, and at around the same cost as the previous generation. This is a no-brainer for customers who can enjoy double the token revenue with the same hardware spend. It’s not only the tokens that are getting faster, but so is time to market: the rack architecture itself is redesigned to be more modular, allowing improved manufacturability and deployment times. Last but not least, a new I/O module will enable open, heterogeneous and disaggregated inference architectures going forward. These disaggregated inference setups will go a long way to help overcome the memory capacity constraints of the CS-4 by pairing it with HBM-based systems.

Same Wafer, Double the Clock

The CS-4 uses the same 5nm WSE-3 as the CS-3, but Cerebras is extracting double the performance by doubling clock speeds. This comes from feeding dramatically more power to the wafer, and is enabled by the CS-4’s improvement in power delivery and cooling technology. While staying on the same 5nm silicon sounds underwhelming, Cerebras can still double the metric that matters most: memory bandwidth. The doubling in memory bandwidth should translate into a near doubling of tokens/sec/user all else equal, and this is what customers want from Cerebras. Clock speed doubling also drives double the peak theoretical FLOPs and the WSE’s parallel off-wafer I/O, allowing the CS-4 to upgrade to 2.4Tb/s of off-wafer I/O from 1.2Tb/s with CS-3. However, what remains the same is 44GB of SRAM capacity per wafer, as this is determined by the number of SRAM bit cells available on each wafer. so we’ll have to wait for the next generation silicon before we can see any improvement here. This is the main drawback of re-using the same WSE-3 as the low memory capacity per wafer is one of the key tradeoffs inherent with Cerebras’s architecture.

As we described in our previous article on Cerebras, the wafer has an incredibly unique architecture due to the use of SRAM that makes it well suited for running kernels with low Arithmetic Intensity, such as low-batch size decode.

Network Improvements

Beyond the doubling of off-wafer I/O bandwidth, there are further improvements made on off-wafer communications coming from a new Wafer I/O interface, which is an upgraded FPGA card that is used as a NIC to convert Cerebras proprietary I/O to standard ethernet. From the image below, we can see that there are 2 I/O modules coming off the north and south of the wafer.

This I/O module is field-upgradeable, so Cerebras can move to new networking standards without redesigning the chassis. This seems like a small change but there are big implications. This makes it easier for the CS-4 to interface with other systems for disaggregated inference setups. Pairing the CS-4 with HBM-based XPUs in a disaggregated attention feed-forward network setup is one way to overcome the CS-4’s low memory capacity, in the same way that Nvidia is positioning the Groq LPUs. We write more about this below. This seems to be designed especially for AWS in mind, who would like to have its EFA NICs on CS-4 to interface with Trainium servers for disaggregated inference.

In addition, latency through the 2 layer fat-tree network (using Arista ethernet switches) is reduced to 3 microseconds from 5 microseconds for CS-3 through a new low latency package processing pipeline. Now, direct wafer to wafer links are also possible rather than going through the switched network which further reduces latency to 2 microseconds. The direct wafer paths are also configurable so this means that the FPGA has switching capability to route data through various wafers.

This is a real improvement, but with many Cerebras competitors now quoting all in switch latencies in nanoseconds, “ultrafast” networking is relative and we view it as a modest improvement. We believe this 3µs and bandwidth limitations continues to be a bottleneck that prevents parallelism setups such as EP and ETP where expert layers span multiple wafers. Token dispatch and combine from router to expert is latency sensitive, and the expert imbalance problem coupled with an extra network hop makes pipeline parallelism the only viable solution.

The Backpack Rack

At the system level, the headline change in CS-4 is physical. Cerebras has split the rack into a front half dedicated to power delivery and a rear half dedicated to compute, packaged as modular, pluggable “backpacks.” Each backpack houses a single wafer-scale engine, and a CS-4 rack holds three of them, up from two wafers per rack in CS-3. The cooling infrastructure such as the pump and the heat exchangers are also removed from the rack, as data centers nowadays are built to support fully liquid cooled racks.

The backpack is a vertical enclosure that is split into power, cooling, I/O, and WSE-3 modules independently, which makes the whole system meaningfully simpler to manufacture than CS-3. The WSE-3 wafer sits vertically with the power side facing the front of the rack. The power modules will deliver power via the front side of the wafer, which is the same as CS-3. The cooling of the wafer will be approached from the rear side of the wafer. The I/O modules are attached to the top and bottom edges of the wafer forming a rectangular surface with the wafer.

The backpack design allows a smoother deployment process. Customers can set up the rack with the power modules before simply socketing in the the wafer backpack onto the rack on site. Given the upgrade to three wafer engines per rack running at higher clock speed. One CS-4 rack lands at 125-135kW TDP, which is up around or just short of double the 23kW power draw of a single CS-3. Overall, this means performance/W has at best a slight improvement over CS-3.

The big improvement gen-on-gen should be cost. While more cooling and power alone would bring up the BOM, the CS-4’s reduction in components and simpler assembly should be a significant offset to these cost items, and we think the effective BOM per wafer could end up similar to the CS-3. To customers, this means getting nearly double the interactivity and token revenue but at similar TCO which is a very attractive proposition.

Cerebras’s favorite number for CS-4 is 43 PB/s of total on-chip memory bandwidth, which the company markets as roughly 2,000x more memory bandwidth than Nvidia’s Rubin. That’s the number that will lead most coverage of this launch, since on-wafer SRAM bandwidth scales up with the increase in power.

The result is that in spite of 2,000x more memory bandwidth, Cerebras claims a more reasonable interactivity improvement of up to 30x when compared to GPUs, which it’s branding as a new “ultrafast” performance tier. We believe CS-4 will hit near 4,000 tok/sec/user on frontier models, while CS-3 hits 2,000 tok/sec/user. Meanwhile we expect Blackwell GPUs will continue to top out at a theoretical 200 tok/sec/user (which no one actually runs at), and a more realistic 100 tok/sec/user for reasonable amounts of concurrency. That looks like 20-40x more interactivity to us, so why not “up to” 40x faster? Seems fair enough.

Parallelism Strategies

Because a WSE does not have enough SRAM on wafer to hold an entire model’s weights, Cerebras continues to focus their efforts on pipeline parallel inference with this system. On CS-4, every MoE expert for a given model will sit on a single wafer, interleaved. Pipeline parallelism by default is different than GPUs, where tensor and expert parallelism are most common to get big models to fit in the available HBM. Cerebras has always maintained that using the GPU’s HBM to store weights is slower, more power hungry, and more expensive than doing everything on one wafer.

Of course, when comparing a cluster of WSEs to a cluster of GPUs on performance, power consumption, and cost, it really depends what parallelism strategy you choose. GPUs have a very wide range of configuration options (from high throughput/low interactivity configs to high interactivity/low throughput) while the wafer’s range is more modest. Only high interactivity/low throughput is considered.

In order to compare WSEs directly to GPUs, we are particularly interested in NVIDIA’s release of TileRT, which brings high interactivity/low throughput configs to GPU clusters. We discussed some of this in our TileRT article last week

Because the wafer itself is 44GB, Cerebras hasn’t yet needed to shard individual experts in a MoE model across multiple wafers, even for frontier models that they run today such as GPT 5.6 Sol. However, due to the demands of long context inference, we expect Cerebras customers such as OpenAI to try and save on the cost of keeping KV Cache on-wafer for long context workloads, and run 5.6 Sol at 256k context window, rather than the full 1M context.

As we described in our Cerebras IPO article, the cost of supporting long context inference is massive. Most people still understand that the amount of memory required to get one forward pass from a model is proportional to the total amount of parameters in the model. However, it still seems to be a well kept secret that the amount of memory required to hold KV Cache grows proportionally to the number of concurrent users and the average/max size of those users requests (which is dictated by the context window of the model). Running a model with a large context window, and supporting many concurrent users, requires lots of memory capacity.

When we run a simple analysis of what it would take to run a large model (say, the 1.6T parameter DeepSeek V4 Pro), we find that the minimum number of Cerebras WSE’s needed to run this model at 1M ctx is around 20 systems, and at a reasonable concurrency of 256 requests, its around 40 systems. That’s over $20M of CAPEX and 1MW of power consumption before you can get a forward pass on a frontier model.

This analysis is available on our public tokenomics website:

Of course, after this up front investment comes some pretty remarkable aggregate performance metrics. Lets dig into that.

Betting on Disaggregation

Cerebras is positioning CS-4 as being built for open, heterogenous, disaggregated inference from the ground up. They are currently working with AMD and AWS Trainium as partners, but have more coming. In all these configurations, Cerebras will serve as the decode chip, since its rooflines aren’t optimal for compute-bound prefill. The company believes in all disagg setups, and claims support for attention feedforward disaggregation (AFD) in addition to the traditional prefill decode disaggregation (PDD).

Software approaches trying to replicate dataflow execution on GPUs can approach the HBM roofline, but cannot compare to SRAM-only architectures on bandwidth. However, all heterogeneous disagg setups are double-edged, since the ratio of Prefill to Decode resources in your cluster is fixed the day that the hardware PO is signed. Meanwhile a fleet of GPUs or TPUs can be dynamically allocated into different ratios as workload profiles from users shift over time. And in the real world, workloads do shift over time. We saw reasoning models drive decode costs up as models thought longer, and then agentic cache hits drive prefill costs down while decode costs remained the same. One P:D ratio to rule them all is unlikely to be perfectly optimal for the 5+ year lifespan of these systems.

To get a better understanding of these dynamics, and the thought process that infrastructure teams are using to design high performance inference clusters, read our TileRT article.

Roadmap: Nexus and CS-5

Cerebras has already co-designed the next-generation wafer-scale engine alongside a new rack-scale platform “Nexus” meaning the CS-4 rack chassis will likely carry forward into future chip generations without major mechanical redesign. The company’s stated roadmap commitment is roughly 2x faster performance every year, with a specific target of 20x throughput improvement by 2027.

On the reliability side, Cerebras is addressing the operational difficulty of swapping wafer trays in the field, and continues investment in error recovery, tray-level redundancy, and yield harvesting across cores and channels on the chip itself.

Read more

Armed with a reconstructed model we argue that the problem is worse than we thought. PJM’s model includes errors that we estimate have cost all of its 66 million residents a total of $12B between 2025 and 2027 alone. We share our method in the annex of this newsletter for our subscribers; as well as the results of the model in our PJM Model dashboard. Our live rebuild of the Reserve Requirement Model is a tab available exclusively to our Energy Model clients; which also includes a quarter-by-quarter forecast of the whole US grid tracking >40,000 grid-connected power plants, and every single behind-the-meter datacenter power order.

PJM’s model is structurally anti-growth with a poorly designed capacity market that is globally unique and a governance system that is too big to function.

These failings magnify the negative impact of bad system modeling, which is the focus of this report:

1. PJM underestimates by ~4 gigawatts the existing power plants it already has; owing to a methodology which doesn’t account for the higher efficiency of power plants in winter and improved power plant resilience since Storm Elliott.

2. PJM has wasted ~$12 billion of ratepayers’ money from 2025 to 2027 due to this weak methodology, which dramatically overstated the supply/demand shortfall it faced. Household electricity bills would have risen much less if PJM’s model was accurate.

3. PJM’s emergency auction is putting ratepayers at risk by signing contracts for too much power without committed counter-parties.

Using PJM’s demand and supply curves we calculate that better modeling would have resulted in $6.7B of savings with only 0.014GW (yes, 14MW) less power procured for 2025/26; then $4.9B and 0.8GW for 2026/27. More power meaning less in savings might be counter-intuitive but we have the supply and demand curves to show how these disproportionate impacts occur. PJM has forced itself into operating at the limit, so inaccurately modeling power plants’ capacities has a massive impact on auction costs.

PJM has constrained its own supply of new power by having one-year contracts that start too soon after they are signed, as well as being slow to connect these new plants to the grid. This means fulfilling any demand growth requires paying a big premium for new power plants to be built at unrealistic speeds. PJM also runs the only capacity market in the world that does not distinguish between new and existing power plants. So that premium paid for new power plants is also paid to existing power plants for doing nothing. The same growth in a market that splits new and existing power plant auctions avoids the volume effect of repricing the entire fleet; which means ratepayers are better protected from price spikes.

Despite four record-breaking auctions costing $63B, PJM will be short of the amount of generation it needs to run reliably, and plans to run an “emergency auction” from September 30th to October 21st with results by December 2nd. SemiAnalysis concludes there is an additional 3.8GW of reliable power on PJM’s system by taking into account increased turbine efficiency from cold air, and reduced risk of winter failure after federally mandated asset winterization investments. This is the equivalent of eight large gas power plants which would cost around $10 billion to build today. This 3.8GW would negate 56% of the 6.8GW that PJM plans to procure in its emergency auction.

PJM’s emergency auction presents serious risks, it will sign contracts running to 2043 that are supposed to be paid for by new large loads. These are not sales of electricity; the plants get paid just for existing, whether or not the forecast demand ever arrives. But PJM is doing this with no committed counter-parties. PJM does not have its own money, everything it spends is from ratepayers.

Every PJM state will have to pass (but none have done so yet) their own cost-allocation policy to pass on costs to participating new large loads. But those large loads can opt out if they contract their own additional power plants directly. If they participate there is little clarity on how much of the auction’s capacity they are expected to pay for. Participation brings no other benefits, like accelerated interconnection timelines. Many datacenter developers have written off PJM regardless, going elsewhere or planning behind-the-meter power configurations. If no other counter-party emerges, then once again those left holding the bag will be the residential ratepayers.

In this report, we dive into PJM’s capacity market and introduce our proprietary reverse-engineered model. We believe that PJM fails to accurately model two critical supply characteristics, despite key stakeholders recognizing them to be both real and of importance:

1. PJM does not credit how cold, dense air makes gas plants stronger electricity generators by up to 25% in winter; and

2. PJM’s risk calculations have not been updated to recognize that as long ago as 2024, 400 of PJM’s ~700 gas plants had invested in winter reliability measures in response to the outages from Storm Elliott.

There are deeper problems at PJM than just modeling. Its current auctions were designed two decades ago, and it has not experienced demand growth since. Today PJM is structurally anti-growth. PJM is failing to adapt to this new era of load growth because of a governance deadlock, a vetocracy of vested interests. If it cannot achieve market reform perhaps it can stem the ratepayer bleeding by improving how capacity is modeled, especially when winter is coming.

This report will dig deep into PJM’s capacity market. After providing some introductory context, we adopt the following structure:

• We start with the fundamentals and explain how PJM models supply, demand, and reliability risks. We cover their major modeling methodology change in 2024 from EFORd to Reserve Requirement Study and the implications of that change.

• We introduce our reverse-engineered model and cover in greater depth how we calculate the role of various resources, outages, reliability, and fleet size.

• We get to the core issue: PJM’s lack of consideration of the increased throughput of gas power plants in the winter, and their improved reliability through Asset Winterization.

• We then dive into the pricing impact. PJM’s capacity auctions are highly complex and we explain why and how PJM’s 66M residents’ electricity bills were $12B higher than they needed to be.

• Lastly, we provide some thoughts on the future of PJM and its upcoming Reliability Backstop Auction. This is an emergency mechanism hoping for datacenters to pay for their own capacity. We believe that ratepayers could, again, be at risk.

Let’s go.

This report is a collaboration between SemiAnalysis and Nathan Iyer in his personal capacity.

Context

If you have not been following along, let us catch you up:

• Power bills in PJM states jumped 17-24% in June 2025, and the state regulators labeled the capacity auction as the main driver. Read our PJM vs ERCOT article to understand PJM’s capacity auction, and how singular it is relative to other energy markets.

• PJM constrains its own supply of new power plants by requiring them to be built unrealistically quickly and not even accelerating the grid connections of critical projects. This constrained supply greatly increases prices in the auction.

• PJM’s capacity auctions restarted in July 2024 after extended delays. This cut the lead time for new generation to get built from the intended 36 months (per the Independent Market Monitor) to 10 months, rising to only 23 months for the latest auction. Power plants take time to build and need long term visibility so giving only a 23-month lead time makes it near impossible to build new generation.

• The four auctions since have each procured 134-138GW a year of accredited capacity at prices that jumped from $28.92 before the delay to $270-333 per megawatt-day, raising the cost of a capacity auction from $2.2B to $16.4B, and the four-auction total to $63.6B.

• This money is paid by every single household and business in the PJM area, and is little more than a transfer from ratepayers to existing power plants as over the last four auctions only 4.8GW of new capacity has been procured.

• Already built capacity does not need $325/MW-day: PJM’s own market monitor says existing generators bid at $8-14/MW-day, and Great Britain’s latest auction paid existing generation $18/MW-day (£5/kW-year).

• These gas plants do not need the money to stay online. The median existing combined cycle made 407% of its going-forward costs in the energy and ancillary services markets alone in 2025, before a dollar of capacity revenue.

• PJM prevented new projects from getting a grid connection. From October 2021 there was no study path while PJM cleared its backlog; a general application window did not reopen until April 2026, when 220 GW applied.

• Now PJM has two under-performing interconnection fast-tracks: 1. The Reliability Resource Initiative, 51 shovel-ready projects picked May 2025, now 41 with 31.5% of the megawatts withdrawn, first output scheduled 2030; 2. The 10-units-a-year Expedited Track opened July 31, 2026. Neither has energized a single megawatt.

• PJM cannot reform itself: rule changes need a two-thirds majority across five equally weighted member sectors, so any two sectors can veto anything, and they often do. The Board can override depending on the reform but has chosen not to on these problems.

• FERC has now given PJM until the end of September to reform its governance otherwise FERC will intervene. But FERC mainly has reactive rather than proactive powers and this threat is not strictly about the capacity auctions or modeling.

• PJM is also the only market in the world where the capacity auction price is for the whole generation stack. Japan was the last other market to price the whole fleet on one year terms but has now concluded the design builds nothing and bolted on a separate 20-year auction for new entries.

Action

FERC is accepting comments on PJM’s backstop filing, until 21st August. Anyone may participate at ferconline.ferc.gov. SemiAnalysis will be submitting this analysis. We encourage others to do the same and we welcome any input on the model, please write to us at energy@semianalysis.com.

We also encourage PJM’s Board to file improved modeling criteria at FERC to change the tariff and Reliability Assurance Agreement, through the same fast-path process it used to install the current rules in 2023 and to launch the backstop this July. FERC must accept it if it is just and reasonable, and FERC accepted seasonal accreditation at MISO in 2022. The winter-ratings text was written in Package C in August 2025, passed by PJM’s own task force, but was not passed by the senior committee. The Board can pick it up tomorrow. Crediting mandated winterization has not been drafted. The closest attempt was the market monitor’s blunter version to delete the storm data outright, which was refused in the same session. FERC should require that the emergency auction’s 6.8GW target be recalculated before PJM signs contracts that will last until 2043.

Our main recommendations are:

1. These modeling recommendations should be implemented for the emergency auction as well as future auctions;

2. There should be an inquiry into the overspending of ratepayers’ money in the 2025/26 and 2026/27 auctions; and

3. Auctions for existing vs new capacity should be split.

Our more loosely held thoughts are:

1. The Emergency Auction should probably be canceled for posing too great a risk to ratepayers if new large loads do not arrive, instead only use direct capacity contracting as part of accelerated interconnection agreements; and

2. If the Emergency Auction is kept to make up the shortfall it should be paid for by existing demand with a short term fix of further lowering the price cap for the Base Residual Auction, this would effectively create separate existing and new capacity auctions.

3. Datacenters are how electricity can become cheaper than ever before by paying down high system costs using economies of scale and unprecedented value creation . To attract more datacenters and achieve this affordable energy future State Governors and customer advocates need to focus on the bureaucratic failures to create functional markets.

1. PJM underestimates the power plants it already has by ~4 gigawatts of reliable power

For a primer on PJM’s capacity market and auctions, read our previous report. As a TLDR: PJM acts as a central planner and runs auctions to ensure there is enough supply to meet peak demand, with a safety buffer. PJM made its planning more sophisticated in 2024, but due to failed stakeholder votes, did not include important inputs. The result of leaving out these inputs is that PJM overestimates system risk during winter months which leads to trying to procure too many new power plants as backup. But PJM also has a constrained supply of new power plants which leads to price spikes, trying but failing to fix this scarcity.

Things weren’t always this way.

In 2024, the same year the capacity auction prices first spiked, PJM overhauled their resource adequacy modeling. Previously, the PJM system operated under a system called “Equivalent Demand Forced Outage Rate” or EFORd. The capacity target was set by taking the peak demand and adding a margin based on the chance a generator might be offline. However, this system is not able to effectively capture correlated failures, which are more common in cold winter months for natural gas power plants. It also fails to account for newer technologies on the grid. Specifically variable resources which require modeling on at least an hourly basis to capture their contributions to capacity. These are intermittent generation such as wind, solar, or run-of-river hydro; as well as ‘limited’ resources, like storage of all kinds.

These limitations came to light in 2022. During Winter Storm Elliott, PJM lost 24% of its total generation due to forced outages and derates, and two thirds of the loss came from natural gas generators. Storm Elliott led to a substantial overhaul of the Reserve Requirement Study making it into an hourly risk framework. In this study PJM models demand against supply for every hour of the year for a number of theoretical years, using both historic and forecast-based randomized data. This more detailed approach identified that at certain times of the year there is not enough supply relative to demand.

As a result, gas in particular suffered a 10-20% knock-down. These two changes had a large system impact, cleared capacity fell by 8% (12GW from 147GW to 135GW) between the auctions for 2024/25 and 2025/26. It is as if the system lost a score of large power plants. The capacity auctions have been expensive since PJM basically had to buy replacement plants.

The shift from a summer risk system to a winter risk system is a much bigger deal in PJM. It has one central auction, versus other capacity markets, which instead distinguish specifically between winter and summer. For PJM, a winter risk system degrades the reliability value of stronger summer resources (e.g. solar and storage which can bridge afternoon peaks), and enhances the stronger winter resources (e.g. nuclear, coal). Even a minor improvement in winter gas risk can have ripple effects across every resource, with major commercial implications.

The study’s hourly matching against weather and demand scenarios defines how each new generator improves reliability of the system. To many this new system is a black box. PJM, to its credit, publishes a portion of its work and data. It is complex but its nuances are important for understanding PJM’s current problems. With which, combined with other public data, we have given our best shot at reconstructing the model and scrutinizing its inner workings. In doing so, we’ve identified and corroborated a number of weaknesses that are of great importance to recent price spikes; reliability concerns; and the upcoming backstop auction.

The Reserve Requirement Study

PJM’s new(ish) system overestimates the risk in winter months which leads to overbuilding power plants. Our model re-runs the Reserve Requirement Study on the same whole-system basis PJM uses. Our conclusion is that the same fleet is worth more reliable capacity than PJM credits it, so the auction requirement against total electricity demand falls.

While PJM models demand with its year-round variations, it caps supply at generators’ summer accreditations. On one hand, that makes sense as PJM is a summer peaking system. There is more air conditioning electricity demand in the summer than there is heating demand in the winter, and so the highest demand points in the year are in summer. Even when netted off against the increased output of solar, the demand for reliable power plants is still higher in the summer than it is in the winter.

On the other hand, historically failure rates of generation are much higher in the winter, tend to be correlated across fleets, and aligned with periods of highest demand. Higher failure rates mean that winter can be the most probable time for supply to fall short of demand. Gas is less reliable in winter as storms can cause them all to freeze at the same time, as seen with the Polar Vortex and Storm Elliott.

This type of winter risk is recognized in PJM’s modeling but the details are out of date and lack precision. They do not consider how cold improves the operations of the plants that do not fail, or how plants have hardened themselves against the weather in recent years.

PJM uses historical data for performance. This means that if a resource falters one time, the only way to increase reliability is to wait over a decade to flush the event out of the system. No amount of investments, regulations, or assurances can lift this judgment. As a result, the largest driver of reliability risk to this day was driven by a polar vortex in 2013 and the second is Storm Elliott in 2022.

Self-Supply

Why aren’t PJM ratepayers better protected from price swings? In most other power markets, retail customers have long-term contracts with utilities and are not exposed to yearly capacity market changes. The market next door, MISO, also experienced substantial capacity swings, but the vast majority of households were unaffected because utilities had long-term contracts.

There are a number of reasons PJM is so all-or-nothing, but it comes down to a combination of retail choice and deregulation (shorter and less certain contracts), a Supreme Court case that struck down state efforts to support long-term contracts, and a long-standing FERC fight that boiled down to a partisan battle between resource preferences.

The other main option remaining is for utilities to procure enough capacity for themselves and then effectively eject themselves fully from the capacity market for 5 years. This shows up on the capacity markets as big swings, but ends up being relatively neutral impacts because the supply and demand impact is balanced.

Fleet Uncertainty

While data for PJM’s overall fleet is relatively transparent and drives their risk model, the auction itself diverges in often unpredictable ways and cannot be fully replicated.

The Reserve Requirement Study is bigger than just the Base Residual Auction, as some utilities ‘self-supply’. The Fixed Resource Requirement (FRR) alternative allows utilities to opt out of the auction and file their own capacity plans, with an obligation set by their share of load, which is ~8% of the PJM system (e.g. 12GW for 2026/27).

So PJM’s Reserve Requirement Study models an ‘Assumed Fleet’, and produces a whole-system number (e.g. 146GW of reliable capacity for 2026/27). The auction’s target is the requirement minus the self-supply obligation (e.g. 134GW for 2026/27).

The ‘Committed Fleet’ is the result of the auction and self-supply. In 2026/27 the auction cleared 134.3GW, 0.2GW short of its target, while the self-supply plans committed 12GW, about 0.3GW more than their obligation. The system ended the auction 0.1GW above the whole-system requirement, but only because self-supply over-delivered by more than the auction missed. Behind all of this sits the total fleet, the power plants that physically exist.

Reverse-Engineering the Model

To get the real numbers we needed to rebuild the model in full. So we took PJM’s Manual, which explains how they do this study and turned it into code. Then we ran on that code the weather, demand, and generation data that they publish. For data that they do not give we used publicly available proxies, mainly from the U.S. Energy Information Administration. We got a high level of accuracy to the outputs that PJM publishes across the last four auctions. The detailed step by step is included in the Methodology Annex of this newsletter to help subscribers who wishes to recreate or audit our work.

Improving the model

In the auction for 2026/27, cold air uplift alone should have reduced the required capacity by 1.5GW and weatherization alone 2.5GW at only 60%. Together they would have been worth 3.1GW as there is a significant overlap between their effects.

We calculated this by using our reconstructed Reserve Requirement Study to test how much the outputs would change if we altered two input assumptions that have been widely debated and previously proposed: crediting cold air uplift and asset winterization. Again, the detailed steps are in the Methodology Annex.

Crediting Cold Air Uplift

A turbine is a mass-flow machine, cold air is denser, so turbines can squeeze more electricity out when temperatures drop. If you ignore ways that a system is more efficient in winter, but still recognize winter failure risk then you double count the risk and buy too many power plants.

PJM’s modeling does not recognize that cold weather makes thermal plants more efficient, and increases the available capacity on the grid during cold periods. Cold temperatures make the air denser and improve temperature gradients, which improves turbine flow and increases the peak electricity output of a unit. For combustion turbines, this impact is as high as 25%, and PJM’s class table averages 8.4% for turbines over the winter months. While gas plants are more likely to come offline in the winter, their fellow gas plants are better able to produce more power to make up for it. Steam plants also benefit from colder cooling water which lowers condenser pressure, improving cycle efficiency. Instead these plants are modeled as if it were the summer months.

Crediting cold air counterbalances the greater breakdown risk. Our model runs these two dynamics side-by-side hour-by-hour. It keeps PJM’s winter breakdown rates exactly as PJM sets them, and credits extra output per thermal plant running in each simulated hour. Netted over 41,600 simulated years, crediting cold air alone, with PJM’s failure rates untouched, still cuts the capacity requirement by 1.3 to 2.2GW depending on the year.

PJM was already told they were modeling this wrong. E3, the consultant PJM hired to review its methodology, highlighted the bias in its December 2025 evaluation: “PJM asymmetrically applies ambient derates to the ICAP of unlimited resource classes when temperatures rise but does not apply ambient uprates when temperatures fall.” PJM’s own May 2025 sensitivity tested crediting it in the model which resulted in 33 percent less winter loss of load hours and a 1.1 percentage point lower reserve margin against the 2026/27 auction model. E3’s consideration 4 of 14 was to adopt seasonal or daily capability ratings.

PJM measured winter capability uplift as 8.5GW nameplate. Despite this massive impact (worth at least $13B in new gas plants at a conservative $1500/kW), PJM did not implement this adjustment despite having all the core information. One of the primary reasons is that they would have to run deliverability studies to determine if the additional capacity could serve the whole system. To our knowledge, PJM never ran these studies. It is worth noting that the likelihood of deliverability is high. In stressed circumstances, a large portion of the otherwise deliverable fleet would be offline, and the cold also improves the thermal capacity of all grid components (including conductors, substations, and breakers). Using the summertime capacity ratings for wintertime thermal production is physically inaccurate, but due to stakeholder disagreement (detailed in Package C of the ELCC taskforce), proposed reforms to implement this basic change did not pass.

Our modeling methodology is in the annex.

Updating Risk with Asset Winterization

PJM does not properly credit power plants that invest in hardening themselves against extreme events. This means it underestimated winter risk before Storm Elliott and could now overestimate the risk of their newly winterized fleet. This lack of accuracy and forecasting drives up the cost of electricity. Crediting weatherization alone for the 2026/27 auction would have been worth up to 3GW of reliable power.

Crediting winterization and other forms of resilience investment would be a strong financial incentive for plants to invest as they would then earn more in capacity auctions. Instead PJM establishes plant failure rates based on past performance, and the large collapses of the gas fleet during the Polar Vortex and more recently during Winter Storm Elliott play an outsized role in reducing the capacity value which PJM credits to gas plants. For increased statistical heft PJM uses as many historical weather years as possible. But once there is a big miss only time and good performance can slowly return this to baseline. A painstaking process of asymptotic averages. Even if gas plants invested billions in weatherization or demonstrated extraordinary resilience during a similar event.

Plants fail because components freeze: instrumentation and sensing lines, valves, fuel handling. During Winter Storm Elliott (December 2022), PJM saw 46GW of coincident outages. The FERC/NERC joint inquiry (October 2023) found that at least 75% of freezing-caused failures occurred at temperatures above the units’ documented operating limits. Which means a properly winterized unit would not have failed.

By the winter beginning November 2025, the existing fleet was winterized substantially more than PJM’s model assumes. Mandatory cold-weather preparedness plans, annual training for every unit, and reporting of each unit’s cold-weather operating limits were mandated by April 2023. By January 2024, more than 400 generators had reported winterization improvements to PJM, a figure PJM published in its own review of Storm Gerri. PJM has roughly 450 gas units and about 750 thermal units in total, so that is the majority of the relevant units. The reports are self-made but they were filed under a regime that already required every unit to hold a cold-weather plan, and they came nearly two years before the auction’s winter we are modeling. A legally required calculation of every unit’s extreme cold weather temperature was due by October 2024. The event-CAP rule was enforceable from October 2024; any unit that froze during winter 2024/25 owed a completed corrective plan before December 2025. In addition, binding NERC winterization standards ramp through 2027, although this had no impact on recent auctions that target years after it goes into effect.

There is evidence of winterization in the fleet’s recent performance. Gerri (January 2024) had a peak of 16GW or 9% in forced outages against Elliott’s 46GW. The MLK storm (January 2025) also had a 9% forced-outage rate against Elliott’s 24%, on similar weather over a holiday weekend, while serving an all-time winter peak. Storm Fern (January 2026) had outages of 18-19GW or 10%. During each of the cold events since Storm Elliott, PJM’s fleet has behaved like one substantially winterized. MLK in January 2025 with Elliott-like weather showed no excess clustering at all.

Specific methodology is in the annex but the key point is that the winterization lever does not assume plants fail less often. It preserves each class’s annual average failure rate exactly and removes only the weather clustering. A genuinely winterized fleet would also fail less on average. We do not include credit for that. To be highly conservative we assume a 50% winterization for 2025/26 then step it up by 10% each year from there.

Overlap

The outcomes of these two modeling changes have significant overlap, so together they are less than the sum of their parts. This is because they fix the same hours. PJM’s requirement is set by a handful of deep-cold hours in which the system might run short. In 2026/27, cold air alone is worth 1,535 MW of requirement relief on the auction’s scale and weatherization alone set to 100% is worth 3,143 MW, but together they are worth 3,320 MW, not 4,678. In the overlap winterization (when set to 100%) particularly dominates cold air’s effect, covering up to 76-96% depending on the year. At lower settings the overlap is reduced. For the annual settings we have chosen the overlap range becomes 53-69% depending on the year.

2. PJM has overspent by ~$12 billion

The result of these improvements to the model inputs is that PJM has a more reliable fleet than its Reserve Requirement Study currently recognizes. So relative to modeled demand load it needed to procure less capacity than it has done for the last four years. In the two auctions where the price was free to move (2025/26 and 2026/27) that difference was worth $12B.

Auction Design

Auction design is much harder to reform than modeling methodology. But you need to understand the design in order to see that relatively small shortfalls in modeled capacity can have outsized impacts on auction costs. PJM has two markets: one for energy which is in real time and one for capacity which is on an annual basis. A capacity contract pays a plant for being available, not for making energy. PJM interconnection queue is famously slow and long, but power plants applying joining the queue do so at their own risk, plants joining the capacity market are the ratepayers’ risk.

Two costs matter in any power market: what it costs to run a plant for another hour, and what it cost to build and keep the plant at all. The first short-run marginal cost and the second long-run marginal cost. The daily energy market clears at the short-run marginal cost (fuel, labor, variable maintenance) of the last needed producer. This incentivizes bidding true cost rather than gaming the system as any more than that would risk not being dispatched and so earning nothing. The difference between the short-run and the long-run marginal cost (capital, assets, fixed maintenance) creates a “missing money” problem as power plants struggle to make this back in the daily energy market. Some electricity grids, most famously ERCOT in Texas, only have an energy market. Power plants make that missing money by putting prices really high when supply is scarce. But many markets use a capacity auction to provide this missing money, considering it the more stable approach. PJM’s annual capacity auction also pays all bidders the price of the last needed capacity.

Every year PJM’s planning model (the Reserve Requirement Study, RRS) decides how much reliable power keeps shortfalls to one event in ten years (the 0.1 Loss of Load Expectation, LOLE). Because risks are spread out everyone in the market tries to free-ride on everyone else which means power capacity has no natural buyer. PJM acts as the insurance intermediary. It draws a market demand curve, a sliding schedule of what price it will pay at each level of supply (the Variable Resource Requirement, VRR). The yearly auction in which generation bids and PJM buys against that curve is the Base Residual Auction (BRA).

Unfortunately, PJM runs one capacity auction that includes ~92-93% of fleet capacity, both existing generators and new generators. Most of the rest is covered by self-supplied utilities. This is a problem as these are substantially different products, new generators have much larger long-run marginal costs. Existing generators only need to cover their annual fixed costs, the Independent Market Monitor’s offer data shows most bids at $8-14/MW-day. New generators have substantial risk-related costs and much greater initial fixed costs. New and existing plants also receive the same one-year contract, even though a new plant normally needs a long-term contract with which to raise financing cost-effectively. This auction design means that if there is electricity demand growth which requires new plants to be built then prices will need to increase by a lot compared to the baseline. In PJM’s case the whole fleet’s price went from $28.92 to $270-333 per megawatt-day, roughly 20 to 40 times the $8-14 the existing plants themselves were offering. This creates huge windfalls to existing generators without necessarily increasing generation capacity on the system in an effective manner.

Happy to be corrected, but we believe PJM is also unique in the world for how much of its fleet is put through the same marginal-priced auction. 93% of PJM capacity settles at the clearing price, against 15% in MISO’s residual auction, monthly rolling in New York, bilateral contracts in California, none in Texas, and New England just abandoned the design entirely. MISO also had a sharp auction price swing, but its auction is residual and utilities self-supply or contract for most capacity beforehand. Therefore about 86% of its 2025/26 summer requirement was met outside the auction, leaving most load protected against the swing. Great Britain, Ireland, Italy, and Japan run centralized auctions, but give qualifying new plants contracts of 10 or 20 years while existing plants generally receive one year.

All of these markets make customers pay for growth. But the same growth in those markets would not drive the repricing of nearly the entire fleet as it did in PJM. That cost shift is a product of market design rather than inherent to load growth. Reform is coming, and datacenter developers have largely supported reforms to pay for capacity as it is in their interests for growth to happen painlessly. However, currently no proposals provide any respite for ratepayers who are still paying for design choices made two decades ago.

This design is not without its reasons. For over 20 years, PJM has not experienced any load growth, and benefited from an early gas boom by Independent Power Producers that proved disastrous for early investors. Over that time less efficient plants have been pushed off the system by more efficient ones, in many ways a good system, just not one built for growth.

The Wasted $12B

For the four most recent auctions we have calculated what price and volume the auctions would have cleared at if the Reserve Requirement Study had been more accurate. That modeling improvement shifts the demand curve to the left. We find that the auctions for 2025/26 and for 2026/27 would have seen savings of $11.57 billion, with a sensitivity range between $8.0 billion and $14.5 billion combined while procuring more than the reliability requirement. The auctions for 2027/28 and 2028/29 would not have cleared at a lower price or volume because supply was tight enough that the price cap for those years still binds, even under improved modeling.

The capacity auction price is set where the supply curve crosses PJM’s demand curve (VRR). The demand curve is a published price schedule stating what PJM will pay at each level of supply. The supply curve is made up of all the bids into the market; PJM and its market monitor publish the shape with key data points. We have reconstructed these curves for each of the four auctions.

We then used our reverse-engineered Reserve Requirement Study to calculate how the curves move with improved inputs to incorporate cold air and asset winterization. Given the greater reliability of the existing fleet the need to procure a greater margin of supply over the electricity demand is reduced. This shifts the demand curve for capacity (VRR) to the left. We computed this by year given different inputs and concluded that for a 0.1 LOLE, whole-system reliable capacity requirements fall by: 2,879 MW (2025/26), 3,396 (2026/27), 4,513 (2027/28), 3,824 (2028/29). We have to assume that bids do not change as modeling bid behavior is complex, in reality there would be some change but we do not think it would be material in this context.

PJM’s capacity market has a right-angled supply curve. The vast majority of it is low and flat, then some steady increase, before right at the end it spikes straight up. For example the 2025/26 auction had the first 79% of the bids between $2-5, the next 20% rose until they peaked at $105, and it was only the last 1% that went as high as $352, with the market clearing at $270. This is reflecting that most of the power plants in PJM have existed for a while and bid low as they have little difference between their short-run and long-run marginal costs. But then new power plants bid high to be able to afford to be built quickly and to cover their own risk. The shape of PJM’s supply curves is why, if you are at their limit, a small change in quantity demanded has a huge impact on price and on total cost, even when having a small impact on volume procured.

The Base Residual Auctions do not stop at their required capacity, they clear on their demand curve. So if there is more capacity that can be bought at a price that PJM has decided is acceptable then it clears the auction. For the 2025/26 auction PJM’s ceiling price was $452 and enough bids came in that 135.7GW was procured for $270 against a required capacity of 133.6GW.

If the Reserve Requirement Study had been computed differently then the demand curve would have been shifted to the left by 2.7GW. The adjusted target required capacity would have been 130.9GW and the auction would have cleared 135.7GW at a price of $135 (a 50% discount). Only 0.014GW difference, but that $135/MW-day saving is multiplied by the full 135.7GW (plus $270 times 0.014GW), then multiplied by 365 days to give $6.7B for the full auction.

There is even a magnitude of impact sense check we can compare against. The independent market monitor ran a similar calculation in September 2024, for the cold-air credit alone. It found $2.7 billion with the reserve margin held fixed, and up to $8.0 billion without. Our figure also includes the reliability effects of winterization.

Price Caps

The auction for 2026/27 included the first politically imposed price cap. PJM has price ceilings in its demand curve modeling but price caps were set as a hasty measure by governors to prevent PJM’s awkward auction design from harming ratepayers even more. While this was achieved, the price cap of $329 also prevented the auction from clearing properly. 134.2GW was procured against a required capacity of 134.5GW, only 0.3GW short.

If the Reserve Requirement Study had been computed differently then the demand curve would have been shifted to the left by 3.1GW. The adjusted target required capacity would have been 131.4GW and the auction would have cleared 133.6GW at a price of $230. Only 0.6GW less, with an improved reliability margin, and $99/MW-day discount. Total auction savings of $4.9 billion.

Price caps can have many unintended consequences. In this instance one result has been an emergency auction for 11-15 year contracts at elevated prices. Any generators that chose to withhold incremental capacity may well have benefited from doing so.

Constrained Supply

2025/26 and 2026/27 are the two auctions in which we found PJM had wasted ratepayers’ money. The next two auctions would not have had price or volume impacts from improved modeling because supply was so constrained that even 4.5GW and 3.8GW of movement respectively would not get the demand curve to cross with the supply curve below the auctions’ political price caps.

The 2027/28 auction held in December 2025, cleared at $333.44, at the price cap and 6.5GW short. So a 4.5GW leftwards shift in the demand curve is not sufficient to make a difference.

The 2028/29 auction held in July 2026, cleared at $325.00, at the cap and 6.8GW short. The 6.8GW target falls to about 3GW with improved modeling.

That 6.8GW vs 3GW is now the billions-of-dollars question for PJM’s fast-approaching emergency auction.

3. PJM is putting ratepayers at risk by trying to buy too much in its upcoming emergency auction.

Who Pays?

Buying too much is bad in any auction, but this backstop auction has no defined counter-party. PJM plans to pay for power plants to get built on the expectation that energy demand arrives later. But if that demand does not arrive then PJM will still have to pay.

In theory new large loads are supposed to pay for this emergency auction. But the forcasted large loads are hypothetical and won’t be signing on the dotted line. The plan is for PJM to sign the contracts and for the States to each pass their own cost-allocation policy to allocate costs onto participating new large loads. PJM’s filing clarifies that “Because PJM does not have jurisdiction to allocate retail costs directly to individual data centers, state action will be essential.” This creates a lot of uncertainty for large loads as cost pass-on will be bespoke state-by-state.

Only new resources in service by June 2032 may bid. Contracts all end in 2043, so a power plant online in 2032 gets about eleven years. The costs are then assigned to transmission zones in proportion to forecast load growth between 2026/27 and 2028/29. Those shares are frozen for the full fifteen years. Inside each zone, a state will use bespoke policy design to direct the costs onward to specific large loads or distribution companies. If a state does nothing, the filing’s default applies and the costs fall on every load-serving entity in the zone by peak-load share.

There is no true-up, the zone shares are set by forecast growth and never revisited. The supply contracts are structured as guaranteed top-ups. Each winner is made whole to its bid against whatever the capacity market later pays out to 2043. Load-serving entities assigned backstop costs must post credit support of $1.5 million per megawatt of obligation, within ninety days of each delivery year. At the ceiling average of $555/MW-day the maximum liability for 6.8GW is roughly $21 billion.

There is no sweetener to encourage large loads to participate such as accelerated interconnection. There is not even assurance on ‘unaccelerated’ interconnection timelines. Then on the supply side there is little penalty for under-delivering once a plant is running. PJM itself projects that even with the backstop the next auction “may clear short”.

Overruled

Through PJM’s fast-track stakeholder process this spring, the members developed and endorsed a backstop design built around a voluntary subscription model. This was championed by a coalition of the electric distribution companies and the Data Center Coalition. The idea was for large new loads to subscribe first, committing to pay for the new capacity before it’s procured. Unbelievably for PJM that design won more than two-thirds sector-weighted support.

Then the Board overrode its own members. In its decisional letter of 27th July 2026, the Board rejected the subscription framework on the stated ground that it “would not provide sufficient assurance that capacity equal to the identified near-term shortfall would actually be procured”.

That is why improving this system modeling is crucial. The Board decided this high-risk emergency auction was necessary because of modeling which overstates the near-term shortfall. The near-term shortfall is currently 6.8GW, recomputed it would be 3.0GW, an amount a subscription model could plausibly have covered. The total requirement of 156GW is built from the 2028/29 load forecast, which already embeds the anticipated datacenter growth.

PJM has filed the Board’s proposal at FERC on 31st July 2026. The Board’s design uses a $555 average-cap and they added back a badly designed curtailment scheme (IRAS) that the members had rightly voted down. Both of these measures seem designed to dissuade new large loads from joining this scheme.

An average-price cap with pay-as-bid supply contracts means that participating demand loads do not know what price they will sign. The $555 cap constrains the portfolio average and not any individual contract. PJM’s own worked example shows a $600 offer clearing. The quantity being bought is also unknown, because the window for reporting bilateral offsets runs concurrently with the offer window. The load’s own share is unknown, because it depends on state allocation mechanisms that do not yet exist and on how many other loads participate. The average is applied at selection. If contracts are voided later, the remaining portfolio is never re-tested against $555. This risk is asymmetric on the upside as lower priced contracts are more likely to be abandoned or fail. Pay-as-bid makes this worse. In a marginal-price auction a bidder’s best strategy is to bid its true cost. Under pay-as-bid the best strategy is to bid your forecast of the highest accepted price. PJM’s own witness says this in the filing: pay-as-bid sellers “are more likely to offer the expected clearing price,” and market-power mitigation in this setting is “difficult and ineffective.”

The Interim Resource Adequacy Service (IRAS) scheme applies to loads over 50MW after 2027/28 that are not covered by one of: the central procurement, self-supply, or a bilateral contract. These loads connect as non-firm and are curtailed first during capacity shortages, ahead of even the pre-emergency demand response programs. As an aside, non-firm connections are fair in extreme cases, but this is mad. Demand response participants are paid to curtail; it is a service they sell. The new large loads would be paying for this service in their electricity bills and still be turned off first. These demand loads will be in the hundreds of megawatts and are not designed for this kind of fast response or high-fidelity manipulation. It could even be dangerous to the system. PJM’s own staff had dropped mandatory curtailment days before the final stakeholder meeting, citing doubts about PJM’s jurisdiction to curtail retail load at all, and the generators’ own trade group warned that reviving it invited protracted litigation.

For any load that can manage it, self-supply strictly dominates the auction. This will drive defection and greatly increase the risk of yet more over-procurement by PJM to the detriment of its rate-paying households and businesses. For the rest, greener pastures await in other power markets.

Governance

Comments to FERC close 21st August at 17:00 Eastern. The auction runs from 30th September to 21st October, with results by 2nd December. This is only one week before the next regular auction opens on 9th December. PJM posted the class ratings for the 2029/30 auction on 7th August. Once again they were computed for endless summer. PJM will post the remaining parameters on 31st August. Everything in this piece is about decisions being made in the next ninety days.

Other than our reverse-engineered model almost none of this is new. In 2023 PJM proposed a full two-season capacity market itself. But after the Independent Market Monitor and others objected that the design needed more development time, PJM withdrew it and filed an annual-only package; FERC approved that package in January 2024 and declined to require seasonality. In September 2024, the Independent Market Monitor concluded that crediting the winter capability of thermal plants would have cut that year’s capacity bill by $2.7 billion with the reserve margin held fixed, and up to $8.0 billion otherwise. Its standing recommendation read: “There is no reason that excess winter CIRs cannot be assigned to these resources immediately.” In May 2025 PJM’s sensitivity analysis found the fleet’s winter capability runs 8,561 megawatts above its summer ratings. That same month, members voted 3.699 (of 5 sector-weighted votes) to defer adjacent transmission work “until such time that stakeholders undertake work on a seasonal capacity construct.” In July 2025 PJM’s own task force endorsed the fix in Package C. This included winter ratings and passed 178 votes to 54, 77%. Twenty-six days later, on 20th August 2025, the senior committee killed it with 30.7% of the weighted vote. Generation owners argued a higher winter rating raises the benchmark a plant is penalized against when it underperforms in summer. The consumer sectors voted PJM’s package down while backing two blunter alternatives, and the generators killed those in turn. In December 2025 E3 recommended seasonal or daily ratings, Consideration 4 of 14. No vote has been held since.

Conclusion

PJM’s centralized capacity market has fully failed at its core role of solving capacity shortfalls. Faced with the first instance of modest growth in its modern history, the capacity market was unable to procure capacity. Instead PJM’s design misallocated those dollars as windfall profits to existing generators while constraining supply with short lead times and long interconnection queues. PJM is making progress at creating a third energy market in the form of this emergency auction specifically for the new generation. But in doing so it is converting long-standing errors into long-standing liabilities with basic resource modeling accuracy still in the air.

These specific modeling points have been debated before but without success in implementation. Our hope is that this effort to reverse-engineer and release PJM’s black box will move the debate through exposure to the size of impact this change in methodology could have. That is what we have attempted to do with the information that is publicly available. We must emphasize that there is key information which is not available which we have highlighted. This has the result that our modeling is at best a good approximation. If we are even half right, we need to change this modeling urgently to avoid further waste and create a clearer signal of what new capacity is required to enable PJM to grow once again.

Methodology Annex

Read more

GPUs perform exceptionally well at high throughput and low-to-medium interactivity, but their architecture is less suited for ultra-low-latency inference. An 8-GPU HGX B200 server provides a theoretical HBM memory bandwidth of 64 TB/s of in aggregate. At batch size 1, GLM-5 at NVFP4 requires only approximately 21 GB of active-parameter traffic per generated token. The B200 HBM bandwidth roofline would therefore suggest up to 3,047 tokens/s/user without speculative decoding. In practice, GPUs come nowhere close to this limit.

The gap comes from latency rather than bandwidth. The traditional GPU programming model launches and synchronizes many individual kernels, whose setup and teardown overhead becomes significant at ultra-high levels of interactivity. While these latency costs are less visible at conventional serving speeds, even with CUDA graphs, they dominate as token latency approaches the sub-millisecond Time Per Output Token (TPOT) range. Furthermore, although GPU memory bandwidth increases by roughly 2–3× each generation, memory latency has not improved at all.

While using alternative hardware is popular, there are ways to use GPUs to do this too. This is where TileRT’s persistent engine comes in. TileRT statically compiles the entire decode graph into a single persistent kernel on NVIDIA GPUs, maximizing overlap across computation, memory loads and stores, and communication. On the InferenceX GLM5 FP8 744B benchmark on a single B200 decode server, tileRT has been verified to reach up to 500 tokens/s/user, approximately 3× faster than GB300 NVL72 running traditional inference engines. Iso-cost per output token, TileRT can achieve up to 2x faster interactivity than traditional engines.

We thank the TileRT maintainers for collaborating on TileRT InferenceX benchmarks and also in general thankful to the vLLM community for their amazing design on the V1 connector. TileRT comes from the same community maintainer organization that built the widely popular TileLang DSL.

Subscribe now

With PD disaggregation inference technique, the hyperspecialized TileRT engine handles latency-sensitive decode while throughput-optimized engines such as vLLM and SGLang continuing to serving prefill. The TileRT decode engine is already being deployed in production at Xiaomi for MiMo V2.5 Pro UltraSpeed and ZAI with GLM 5.1 HighSpeed.

In the article, we shall deep dive into the TileRT InferenceX results, what TileRT is, how it composes with the existing inference ecosystem along with the tradeoffs and challenges with TileRT.

We will also elaborate on the tradeoffs of using TileRT on standard GPUs vs. ultra low latency specialized chips like Nvidia Groq LPU, Cerebras and Sambanova, weighing in on if there is a potential for TileRT software running on GPUs to disrupt these specialist chips’ TAM. The SemiAnalysis Accelerator Model provides quarter by quarter estimates of Nvidia LPU30, LPU40, Cerebras WSE-3 & WSE-4 shipments and much more.

InferenceX

InferenceX is our open-source, vendor-neutral, continuously updated AI inference benchmarking and research platform. We measure leading models, inference frameworks, and hardware across the latency-throughput Pareto frontier, tracking how real-world inference performance and economics improve over time.

Our benchmark has been widely reproduced, validated and/or supported by almost every major buyer of compute from Google Cloud to Microsoft Azure to Oracle, to Meta and many more. Furthermore, it has the support of the ML community including from vLLM, LMCache, SGLang, PyTorch, Huggingface and the support of major labs like OpenAI, MiniMax, ZAI, Qwen, Moonshot Kimi, etc.

Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. As previously mentioned, Nvidia has committed to submitting verifiable Vera Rubin numbers to InferenceX. We will have Google TPUv7 results soon, and AMD has committed to MI455X UALoE72 this year too.

Throughput vs Interactivity Curve

Every inference system must balance two competing goals.

• Interactivity (tok/s/user) measures how quickly a single user receives tokens, the inverse of time per output token (TPOT). It determines whether a response feels snappy or sluggish.

• Throughput (tok/s/GPU) measures how many tokens the system produces in total across all users. It largely determines the cost per token.

Batching increases aggregate throughput by processing more requests together, but each user typically waits longer for each token. Small batches do the opposite: they improve per-user speed while reducing the amount of useful work each GPU completes in aggregate.

A bus amortizes its cost across many passengers but makes each passenger wait for shared stops. A race car carries only one or two people and reaches the destination faster, but at much higher cost per passenger. Inference has the same trade-off: batching improves aggregate throughput and cost per token, while small batches improve per-user responsiveness. There is no one-size-fits-all operating point.

In the configuration shown below, increasing interactivity from roughly 25 to 260 tokens/s/user reduces per-GPU throughput from about 5,900 to 200 tokens/s/GPU. That is roughly a 30× reduction in aggregate throughput for a 10× increase in per-user speed.

TileRT results

As we describe in the next section, GPUs already perform well in high-throughput scenarios but struggle in high-interactivity ones. This weakness has created an entire market segment for dataflow chips. TileRT targets the same weakness and therefore focuses exclusively on high-interactivity operating points.

TileRT on B200 is in a class of its own. For the 8k/1k input/output token scenario, TileRT reached 340 tokens/s/user on an eight-GPU B200 node. The fastest result in the current dataset was previously 181.4 tokens/s/user on GB300 NVL72 with NVFP4 and MTP, making TileRT 1.9× faster on this metric. Of course - this is on Batch Size 1, where all that extra trouble to set up the complicated copper backplane in the case of the GB300 NVL72 does not come into play at all in boosting interactivity.

Meanwhile, the fastest FP8 result was 113.6 tokens/s/user on B300 with MTP, making TileRT 3.0× faster at the same precision.

At 1k/1k input/output, TileRT FP8 reached 494.2 tokens/s/user. That was 1.9× the best conventional result, at 256.3 tokens/s/user using FP4, and 3.6× the best conventional FP8 result, at 136.3 tokens/s/user. TileRT doesn’t yet have FP4 support, but it is already beating non TileRT FP4 implementations! The result is also notable because it comes from an eight-GPU B200 node rather than the 72-GPU NVLink scale-up domain of GB200 or GB300 NVL72. This comparison concerns per-user interactivity, not aggregate throughput or cost.

However - there are always tradeoffs when it comes to inference! TileRT’s interactivity advantage comes with lower aggregate throughput. Conventional engines can amortize weight loads and fixed kernel costs across more users as concurrency rises. At 8K/1K input/output, the GB300 FP4+MTP point at concurrency 12 delivers approximately 240 total tokens/s/GPU while maintaining 154 tokens/s/user. TileRT delivers 160.4 total tokens/s/GPU while reaching 340 tokens/s/user.

The trade-off is therefore: TileRT provides much higher per-user speed, but the conventional GB300 point completes more aggregate work per GPU. TileRT as of publication also serves only one in-flight request per decode node, making this a deliberately specialized operating point rather than a general throughput configuration. Thus, with support only for a batch size of 1 user, TileRT is not just a race car, but it is more like a private rocket ship with room for just one passenger. Engineering TileRT to support more passengers might be possible, but it is an ambitious goal.

Star InferenceX GitHub

For end-to-end latency, TileRT at FP8 outperforms the best previously recorded GLM-5.1 result by 4.5× at 1k/1k and 3.0× at 8k/1k. As expected, TileRT’s time to first token (TTFT) is good but not exceptional. The decisive advantage comes from the decode tail: 3.01 seconds, compared with 6.54 seconds for the best NVFP4 + MTP competitor and 18.18 seconds for MI355X.

But What Exactly is TileRT?

We briefly introduced what TileRT does and showed some benchmark results, but let’s pause and explain more deeply what TileRT is and how it works. Traditional serving engines run as thousands of separate GPU program kernels launched one after another. All that setup and teardown means the GPU spends a surprising amount of time waiting, and while this setup/teardown time might not matter for low to medium interactivity inference but it definitely does for ultra high interactivity inference (aka low latency inference). Worse, each kernel writes its half-finished work out to HBM. At small batch sizes, this is a bigger problem as kernels aren’t large enough to amortize launch latency, synchronization, and scheduling overhead.

As mentioned earlier, when running TileRT at batch size 1, for just an single HGX H200 server (38.4TB/s of aggregate HBM memory bandwidth), the active parameter memory bandwidth stands at 42GB per token at MXFP8. In theory, if we were only bound by memory bandwidth, then even without spec decoding, inference should be able to reach up to 1,000 tok/s/user interactivity. This is obviously not the case in the real world! The roadblock is that GPUs’ programming and architecture model is traditionally not built for low latency. Even though memory bandwidth per GPU increases 2-3x each generation, memory latency has not improved at all, even as HBM prices continually increase!

Instead of continuously launching kernels, TileRT has the GPU continuously execute a persistent pipeline, statically compiling the whole model ahead of time into a persistent Engine Kernel: the host launches once, execution stays resident on the GPU for the whole decode lifecycle, and most runtime orchestration moves into compile time.

This is different from CUDA graphs, which captures the DAGs(directed acyclic graphs) of kernel launches and memcpys once, then replays it with a single cudaGraphLaunch. But the kernels themselves are still separate kernels, this boundary between kernels carries device-side costs and the on-chip state is wiped at every boundary. A CUDA graph optimizes the launching of kernels, while TileRT abolishes the kernel as the unit of execution.

Also, through decomposing work into tile-level tasks with warp and block specialization, the runtime dynamically reschedules computation, I/O, and communication in a highly overlapped way. Inside the Engine Kernel, different warp groups take on different jobs: asynchronous data movement, tensor computation, and communication overlap. Where stages used to run serially as load → barrier → compute → barrier, they now overlap at tile granularity, and intermediate results flow forward through registers, shared memory, and L2 instead of repeatedly spilling to global memory. Effectively, each CTA(Cooperative Thread Array) becomes a small heterogeneous factory rather than a uniform SIMT(Single Instruction Multiple Threads) worker.

The next optimization TileRT introduces is specialization extended to whole GPUs. Most TP frameworks assume all ranks execute identical logic synchronously, but sparse routing, Top-K selection, dynamic indexing, long-context attention, and MTP don’t fit homogeneous scale-out well; they’re not compute-heavy but depend on global information, so forcing every rank through them adds redundant work and synchronization amplification. So, if warps can specialize, so can GPUs. In GLM-5.1’s attention layer, GPU 0 becomes a Sparse Indexer worker handling Top-K selection, sparse index construction, and routing, while GPUs 1 through 7 run the MLA workers doing RMSNorm, GEMM, flash sparse attention, and AllReduce.

Finally, instead of treating communication as an external stage, broadcasts, reductions, and synchronization execute directly inside the tile-level flow; with TileRT, an entire attention layer corresponds to a single kernel launch at the host, and execution shifts from compute → sync → compute toward a continuously overlapping compute ↔ communication ↔ compute pipeline.

PD Disaggregated Engine with vLLM<>TileRT

LLM inference consists of two distinct phases: prefill and decode. Prefill processes the input prompt in parallel and is primarily compute-intensive, making aggregate throughput the key performance metric. Decode generates tokens sequentially and repeatedly accesses the growing KV cache, making it memory-intensive and highly sensitive to per-token latency.

TileRT does not replace vLLM, vLLM remains the high-throughput prefill engine and the surrounding serving layer, including its scheduler, chunked prefill, prefix caching, OpenAI-compatible API, and operational tooling. Only latency-critical decode traffic moves to TileRT. TileRT is engineered to be a single-passenger rocket ship, and vLLM remains the plane, car, bus, and train.

Star InferenceX Github

The prefill and decode phases can be disaggregated into seperate nodes. With disagg, one shared vLLM prefill pool can feed two entirely different decode pools.

• Pool A: Ultra high interactivity decode with TileRT

  • Latency-critical requests pass through the TileRT PD Router, which instructs vLLM to generate the first token and marks the request with the destination TileRT node in kv_transfer_params.

• Pool B: General low to medium interactivity decode with vLLM decode

  • General traffic continues through vLLM’s native disaggregation proxy to a conventional vLLM decode pool.

This is done via vLLM’s MultiConnector API that composes the TileRTConnector with its native connector. The TileRT connector claims only marked high interactivity traffic class requests and becomes a no-op for everything else, meaning both traffic classes can share the same prefill server. Between the Prefill and Decode, TileRT uses Mooncake Transfer Engine and NIXL Transfer Engine to move KVCache. In TileRT v0.1.5, each decode node serves one in-flight request at a time. The router gates dispatch and applies back-pressure when the node is occupied.

How does TileRT compare to Cerebras/Groq/SambaNova?

Purpose-built inference vendors identified the same execution bottleneck years ago, but encoded more of the solution in hardware. The SemiAnalysis Accelerator Model has our quarter by quarter estimates of NVIDIA LPU30, LPU40, Cerebras WSE-3 & WSE-4 shipments.

Groq uses deterministic, compiler-orchestrated execution and a large on-chip SRAM hierarchy. Cerebras maps computation spatially across a wafer-scale processor; the CS‑3 provides approximately 900,000 cores, 44 GB of on-chip SRAM, and 21 PB/s of memory bandwidth. SambaNova maps model graphs onto reconfigurable dataflow units backed by a tiered SRAM, HBM, and DDR memory system.

The silicon differs, but the systems share the same idea: latency-sensitive inference benefits from reducing runtime scheduling, operator boundaries, synchronization, and unnecessary movement through external memory. At large batch sizes, those costs are easier to amortize. At batch size 1, they occupy a much larger share of each token’s latency.

TileRT imports software analogues of several dataflow ideas: AoT scheduling, persistent execution, specialized workers, and tighter overlap between communication and computation. The resemblance is architectural rather than literal. TileRT still runs on a SIMT GPU with dynamic hardware scheduling, HBM, and a model-specific compiled schedule.

Star InferenceX GitHub

However, TileRT is still software only: dataflow is imposed on a machine that was never specialized designed for it. A GPU carries dynamic warp schedulers, a SIMT model, and an HBM hierarchy, and TileRT gets its numbers by spending enormous compiler effort convincing that machinery to impersonate a spatial pipeline through statically expanded persistent kernels, hand-carved warp specialization, and per-model compilation against pinned driver stacks. Native dataflow silicon never fights its own substrate. Purpose-built accelerators encode more of the execution model in hardware and can avoid some of the overhead TileRT must hide in software. Their advantage still depends on the model, precision, memory hierarchy, compiler quality, system scale, and serving configuration. That is why Cerebras serves a dense 70B at speeds no eight-GPU node can reach regardless of scheduling: software can approach the HBM roofline, but it cannot raise it.

The market’s early answer is that purity is negotiable. TileRT’s decode engine is already in production behind Xiaomi’s MiMo V2.5 Pro UltraSpeed and Z.ai’s GLM-5.1 HighSpeed, and the deployment pattern is the tell. Neither company procured a new dataflow chip. They carved a speed tier out of the accelerator cluster they already ran, with vLLM keeping prefill, scheduling, and the API while TileRT takes over decode behind the same endpoint. Good enough on hardware you already own tends to beat architecturally pure on hardware you have to buy.

That points at the deeper structural problem: fungibility and flexibility in prefill-decode (PD) ratio.

A GPU pool is one liquid resource, excellent at prefill, excellent at high to medium-batch decode, and now somewhat credibly strong at ultra-interactive decode, with capacity moving between those roles as software scheduler decision that can follow demand hour by hour. An ASIC fleet is the opposite: the ratio of speed-tier capacity to everything else is fixed in hardware the day the purchase order is signed. Changing the ratio of the physical fleet will take months to physically re-rack and re-cable. That would be fine if the workload mix were stable and known. Unfortunately, the split between users who need ordinary conversational latency and users, increasingly agents, who will pay for extreme-interactivity SLOs has a lot of different variables at play when estimating. Guess wrong with GPUs and you rebalance in software. Guess wrong with dedicated silicon and you either strand capital in idle speed machines or turn away the exact premium traffic you bought them for. On top of that - requirements may shift over time, so a correct guess will only be right for a limited period of time.

Going back to the shared prefill pool mentioned earlier, providers do not need to pay the TileRT premium for all traffic. General requests can stay on throughput-optimized vLLM or SGLang decode pools, while only latency-critical requests are routed to the TileRT decode pool.

None of this kills the top of the speed market. The SRAM roofline is still better, certain sizes of models still favor it, and some workloads will always want maximum tokens per second at any price. But TileRT reframes what most buyers need: not a speed machine, but a speed tier, provisioned dynamically out of the fleet they were going to own anyway. Cerebras, Groq, and SambaNova are no longer competing against a clumsy kernel-launcher. They are competing against their own execution model, running on fungible hardware, reallocated by a config file. TileRT may be a single-passenger rocketship, but it allows providers to strap solid rocket boosters to your Metro Bus instead of having to design an entirely new launch vehicle.

Why Is TileRT Development Slow?

GLM5.1 is a generation behind, and has already been deprecated on mainline InferenceX. TileRT’s model catalog is very limited, currently supporting GLM-5/5.1, DeepSeek-V3.2. MiMo-V2.5-Pro-UltraSpeed is the result of a co-design partnership and has yet to be open-sourced.

TileRT inherits ASIC vendors’ biggest weakness. Static ahead-of-time compilation means a tiny model catalog (currently GLM-5/5.1 and DeepSeek-V3.2), hard-pinned dependencies, and real engineering effort per new architecture. There is no fully generic path, a persistent engine kernel means the model is statically expanded ahead of time into one resident program, so decisions have to be made on tile shapes, pipeline depth, buffer residency across registers/shared memory/L2, how warp groups split between loading, compute, and communication, where collectives get fused into the tile flow, and which GPUs take specialized roles like GLM-5.1’s dedicated sparse indexer rank. Change the attention mechanism or the routing scheme and much of that schedule is invalidated. Dataflow chips also face this same issue, good compilers can be notoriously difficult to create.

Share SemiAnalysis

Work is being done to simplify this, especially as software development can be accelerated with AI. TileOPs is intended to reduce this burden. Each operator is declared in a machine-readable manifest specifying its signature, workloads, and roofline model. The manifest drives code generation, testing, and benchmarking against hardware bounds rather than only against earlier implementations.

AI coding agents accelerate tuning within known templates, but novel transformations still require expert judgment. A monolithic persistent kernel also reduces the usefulness of conventional per-kernel profiler timelines, making automated feedback loops more difficult.

Next steps with TileRT<>InferenceX

We are actively working on moving TileRT benchmarking from InferenceX’s single-turn 8k/1k as well as our new agentic coding benchmark, which we call AgentX. This scenario replays real Claude Code and Codex traces with long-context, multi-turn requests, realistic subagent activity, and dynamic tool-use delays. Its median input length is 140k tokens, while the theoretical median cache-hit rate roofline reaches 99.2%.

This workload will test the entire TileRT<> vLLM system, not just decode speed, including incremental KV transfer, prefix-cache reuse, cache retention and offloading, routing, and scheduling. The critical question is whether TileRT can transfer only the newly introduced context between turns while preserving its ultra-high interactivity advantage.

The second step is to move beyond just batch size one. We will also benchmark TileRT at batch sizes 2, 4, and 8. The goal is to map its throughput–interactivity Pareto frontier and identify the point at which the persistent Engine Kernel’s latency advantage begins to flatten.

Perf per TCO of TileRT Ultra Fast Speed

Next, we do an deep dive analysis on TileRT’s cost per million output tokens at ultra-high interactivity compared to decode at normal lower-interactivity operating points. The results are quite interesting with TileRT boosting up to 1.9x faster interactivity when iso-cost with traditional engines. We use our AI TCO Model as baseline for the capex & opex for each chip SKU.

Read more

Yet, we believe that the number is real. We see SpaceX on track to build about 10GW by year-end 2027. We’ve evaluated all sites suitable for SpaceX and provided the list to our Datacenter Model subscribers. Our Energy Model subscribers also have the precise list of gas generation equipment available, quarter by quarter, by 30+ turbine, engine, fuel cell suppliers. We provided much of this data, before the market woke up to it. Below, we discuss how Elon bypasses typical datacenter construction constraints.

As explained in our Meta Compute deep dive, large-scale + near-term compute is a remarkably scarce combination, and it’s priced at a huge premium – up to $50B/GW/year. However, AI labs can handle it and make a good living off it.

Our Tokenomics Model and our Inference Simulator demonstrate that at realistic performance levels (e.g. tokens/sec per GPU), both OpenAI and Anthropic can generate over $100B/GW/year of revenue when selling API inference on a GB300 cluster. This is significantly more than the costs of renting a GB300 cluster for a year at current neocloud prices.

Serving inference tokens is unbelievably profitable for the frontier model companies.

We assume around $12B/GW/year of cost per year, using a conservative rental pricing rate of $3/GPU-hr, and make a token production estimate using our Inference Simulator with a frontier-class model architecture and our agentic coding benchmark, AgentX (part of InferenceX), which is built by collecting real production coding traces. We blend that token production rate between input, cache-read, cache-write, and output token costs at our real workload ratios, and produce the final estimate, exceeding $100B/GW/year.

For background, our Inference Simulator is built from the ground up with a fundamental understanding of how modern AI accelerators work. We build a roofline and realistic performance model for how frontier models work during inference, with timings for every operation and a real trace output. It is an end-to-end simulation of the actual workload executing on the actual silicon. We have validated the simulators fidelity on a wide range of accelerators and workloads and continue to improve its ability to accurately forecast performance of future accelerators based on design specifications.

Please reach out to sales@semianalysis.com for more information on how we apply the Inference Simulator for custom research and analysis.

Beyond OpenAI and Anthropic, there is actually a third company in the world capable of printing such economics per GW: Microsoft. Having full access to OpenAI models, they can generate the exact same revenue and margin per MW, while paying none of the training costs. Satya nailed the negotiations with OpenAI: the deal reworked in April 2026 dropped the old 20% revenue share from the equation. Put simply, Microsoft has a giant incentive to procure as many MWs as possible, as fast as possible. While much of their datacenter capacity currently goes to OpenAI at ~14M/MW/year, they have the opportunity to improve that mix. The potential impact is Microsoft Azure accelerating revenue growth from ~42% to over 100% by next year. A once-in-a-generation opportunity, that SpaceX is incredibly well positioned to serve.

While Microsoft signing 3GW with SpaceX for 50B/GW/year sounds insane, we view it as possible for two reasons:

• 1/ Microsoft is already preparing for an epic datacenter ramp. As discussed below, they’ve signed 10GW of contracts year-to-date, for over $300B of total contract value (not including the GPU cost). We expect much more to be signed. Caveat: these contracts contribute to late 2027 and 2028 capacity. There is a near-term gap to fill.

• 2/ With a 90-day cancellation policy, akin to the SpaceX deals with Anthropic and Google, there is zero balance sheet risk. This is remarkably easy for Amy Hood to sign off, given the revenue opportunity.

For SpaceX, the next natural question is financing. How can Elon afford to pay so much CapEx without the balance sheet of the leading hyperscalers? We expect a combination of the two following items:

• 1/ Support from Nvidia, in the form of vendor financing to lower the upfront cash cost. This is likely why Elon declared to be Nvidia exclusive on the earnings call! As our Accelerator Model has repeatedly explained, xAI/SpaceX have actively evaluated alternatives like TPU and AMD – so the financial argument likely made them abandon these and focus on Nvidia.

• 2/ Operating cash-flow financing led by industry-high pricing, enabled by fastest timelines: SpaceX will continue to sell large-scale compute with 3-5 months lead time, an unbeatable offering, and price it accordingly at 30-50M/MW/year. That pays back the capex in less than a year. We dived into this in our Meta Compute article.

The implications of this are a path to $300B of ARR by the end of 2027 for SpaceX. This assumes only 50% of their 2027 incremental compute is monetized, the reminder being for the Grok & Cursor teams for training (no inference revenue modelled).

Let’s now dig in. We begin with Microsoft, who has spectacularly, finally, woken up: last year’s pause has reverted, with 10GW of signed binding contracts year-to-date. We’ll briefly discuss economics to get to $100M/MW/year of inference revenue. We then shift to SpaceX and analyze their datacenter ramp, and the feasibility to achieve 10GW+ by year-end 2027.

Microsoft’s 10GW awakening to capture the 100M/MW/year opportunity

In December 2024, we called out before anyone else in our Datacenter Model a dramatic pause in Microsoft’s leasing activity. Today, the giant awakened. Our models tracks quarter-by-quarter leasing activity, neocloud contracting, self-build construction starts, and large-scale binding PPAs and ESAs. We show below the outputs. Microsoft has contracted over 10GW across all these surfaces, which is the equivalent of ~$300B in new binding commitments.

A key reason for this awakening is their desperate need for compute to capture a $100M/MW/Year revenue opportunity. Microsoft signed in October 2025 a $250B agreement with OpenAI, which we estimate at ~7GW in total in our Tokenomics Model – the world’s best tool to understand the nuances of the dollar-to-watt math. This massive Infrastructure-as-a-Service deal has left Microsoft highly compute-constrained on their other use-cases. They’ve been unable to leverage their access to OpenAI models for their API business Foundry, or for their applications like Copilot.

Yet, these are the services that come at the highest margin and revenue per MW, by far. We’ve explained that in depth in our AI Value Capture piece.

Over the past month, it’s finally become consensus among sophisticated investors that serving frontier tokens at API prices is actually an extremely high margin business. We were the first to call this out to our Tokenomics Model subscribers back in Janurary, when we explained why inference gross margins are north of 60%. Then in June, we followed up with a deep dive that showed how Opus 4.8 in particular had 85%+ margins. This has since become the default number everyone cites when analyzing Anthropic.

To arrive at these margin estimates, we had to carefully synthesize leaked financials, InferenceX data, microbenchmarks on all the latest accelerators in the industry, papers, blogs and tweets from open source labs, and more. New datapoints such as the leaked DeepSeek investor call (which said they have a 10-month GPU payback period) confirm we’re in the right ballpark, but we’ll be the first to admit that the lack of granularity is extremely unsatisfying. Rather than a single company wide inference gross margin number, what you really want to know is the gross margin for every (model, accelerator) combo along the entire throughput vs latency pareto frontier. For example, what’s the gross margin for serving Opus 5 Fast on Trainium3 vs Fable 5 on TPUv7?

We answer that question with our Inference Simulator, available exclusively to SemiAnalysis consulting clients.

Our AI Cloud TCO model already answers the cost side of the equation, but the revenue side has historically been unknowable. To solve this, we wrote a simulation framework that simulates real model execution on virtual hardware, backed by tuned fine-grained performance models covering a variety of accelerators and operation types. We run each model on simulated XPUs across every possible serving configuration, with a mix of real-world and idealized serving conditions. This allows us to, given some well-informed assumptions about model architecture, accurately estimate the performance of any combination of software, hardware, and workload.

Thanks to this simulator, our Tokenomics Model now includes high-level revenue per MW numbers for running the flagship OpenAI/Anthropic models on all the relevant chips. Workload shape is obviously a huge factor, and we simulate running over $1M worth of agentic traces collected from our own usage while meeting the real interactivity and TTFT levels observed from hitting first party endpoints. As a teaser, here are our numbers for serving Fable 5 on GB200 vs GB300:

This is Microsoft’s $100M per MW opportunity. Given the recent surge in Codex demand and corresponding OpenAI ARR acceleration, we believe Microsoft would be able to monetize compute at similar rates by serving OAI models.

Now, to capture this once-in-a-lifetime opportunity, Microsoft and AI Labs datacenters, and they need them quick and big. SpaceX has already proven twice that they can build faster than others, but their compute capacity will “only” be 2GW by year-end 2026. Can they really build 10GW+ in just a single year, in 2027?

SpaceX: building datacenters at a stellar pace

In our Meta Compute article, we explained in depth why Elon has proven, yet again, to be a commercial genius. He understands that AI lab margins have dramatically surged, and accordingly introduced a “value-based pricing” for his GPU clusters, as opposed to the more common “cost plus”.

To keep the machine going, Elon needs to build datacenters faster than anyone else. We believe that he can. What gives us this confidence? We’ve written a few times about Elon’s speed, with 122 days to build Colossus 1’s 300MW, six months to build 200MW at Colossus 2, the decision to build an onsite generation plant 1km across the border to avoid permitting, and much more.

There’s been even more displays of speed since then. The power plant in Southaven has expanded from 27 turbines (~495MW) in February 2026, to 69 turbines (1.7GW) in July 2026.

As well as the arrival of “MiniHard,” which upon vertical construction in March 2026, will likely reach 450-500MW in just ~5 months! That doesn't mean Elon builds better than everyone else. He's just applying a different playbook.

Switchgear and large power transformers are sold out for 2 years? Just buy power modules from China, and skip LPTs by delivering medium voltage power from power gen to low voltage transfos, which are much more widely available. Elon’s companies have the world's most talented electrical engineers - no one knows better than them how to manage speed vs efficiency tradeoffs. Most datacenter operators in the world optimize for efficiency and quality - it's the only way to land a 15-20 year take-or-pay hyperscaler datacenter contract. SpaceX will focus on entirely different tradeoffs: it's speed above all. In times of compute constraints and extremely high AI token margins, a 500mw cluster available in three months, with 90-day cancellation policy, is one of the most scarce and valuable asset in the world. All typical “quality” metrics don't matter. Proof? Google, the most vertically integrated infra company in the world, somehow ended up signing with SpaceX.

Gas turbines are 5yr+ backlogged? GEV’s are, but there are plenty of other options - our Energy Model 30+ manufacturers of gas gen equipment that have secured large scale orders to serve datacenters. There is plenty of available capacity if you look hard enough and you're open to working with new suppliers. Secondary market volumes for turbines is surging - for example, all the turbines initially scheduled to go to Oracle’s New Mexico site are on the market. Secondary market prices are very high, but Elon can pay.

Labor is the ultimate constraint? Just parallelize as much as possible, reduce the commissioning process, and preassemble as much as possible. Reports out Colossus 2’s peak daily labor at ~3k construction workers, which as about ex lower than other gigawatt-scale datacenters under construction. Elon has a long history of accomplishments with less staff than industry standard - just look at Tesla and SpaceX’s history.

That leaves more than enough time to build many such shells by 2027. It also was his first true greenfield, so he can probably do better for the next. Another option is, of course, to retrofit. Colossus 1 and 2 have been built remarkably fast through retrofits, as explained in our xAI deep dive last year.

Building 10+GW in a year will be a different story. SpaceX will need to scout all over the country to find suitable land, with easy permitting and access to gas. We however believe that there are more than enough options to support a material ramp-up. This will, naturally, extensively rely on onsite gas generation – check our energy deep dives here to understand how it works and why it’s necessary.

Beyond the paywall, we will discuss some of the sites that we suspect Elon might take.

How is this possible?

Read more

• Demis Hassabis, DeepMind co-founder and former CEO, is no longer involved in day-to-day operations.

• Jeff Dean, former Google Chief Scientist and Gemini co-lead is leaving to start a neolab called Discovery Loop. Jeff is the undisputed GOAT of Google engineering, co-founded Google Brain, and started the TPU program.

• Joining Jeff are Sanjay Ghemawat, Quoc Le, and Oriol Vinyals. Sanjay and Quoc were both Google Fellows, which is a title reserved for the company’s top dozen or so technical contributors.1 Oriol was a Gemini co-lead.

• Koray Kavukcuoglu, former DeepMind CTO and the one remaining Gemini co-lead, is replacing Demis as the leader of DeepMind/Gemini.

Obviously these are not the actions of people excited about Gemini 4 Pro.

For all intents and purposes, we believe DeepMind is no longer a frontier lab. We said as much a few months ago to our Tokenomics clients due to large numbers of departures from their reinforcement learning teams and poor compute allocation. Google will continue meandering on and releasing models, but their odds of reaching SOTA again have dropped to zero.

Furthermore, the biggest beneficiary of today’s news is neither Anthropic nor OpenAI—it’s Google Cloud. Whereas Gemini and GCP used to desperately fight for compute allocation, it’s now clear that Thomas Kurian won. We expect GCP revenue growth to meaningfully accelerate as a result.

Gemini 3 Pro was the peak

At the end of 2025, Anthropic, OpenAI, and Google were the clear AI big 3. Some still held this opinion as recent as last week, but the downfall of Gemini in 2026 was clear to those paying attention. Here’s an excerpt from an institutional note we published to our Tokenomics Model subscribers on July 9th:

• In November 2025, Gemini 3 Pro was arguably the best model in the world, forcing Sam Altman to issue a “code red” at OpenAI.

• Since then, the gap between Deepmind and OpenAI/Anthropic has widened dramatically. Gemini 3.5 Flash was a total flop, and industry chatter suggests the upcoming Gemini 3.5 Pro is roughly Opus 4.5 level. This is quite literally worse than GLM 5.2, much less Fable 5 and GPT 5.6.

• We believe things will only get worse for Gemini, as Google lacks the religious conviction necessary for building RSI. We think it’s likely both MSL and SpaceXAI surpass them in model quality by the end of this year.

• Along with Noam Shazeer and John Jumper, most of the best RL people at Gemini recently left the company.

• OpenAI has overcome their pre-training issues, and a much larger model code named “Doug” is actively in the works.

The second to last point is worth emphasizing. Jeff, Sanjay, Quoc, and Oriol are just the latest in a long string of high profile departures from DeepMind. Despite discovering most of the foundational breakthroughs behind the current LLM paradigm, Google is now simply unable to retain top AI talent.

Notably, Jeff and co. have followed a playbook to create their neolab, Discovery Loop, that was established by David Silver of Ineffable Intelligence in November last year. To pursue their mission, top AI researchers are now choosing to leave Google, raise billions from outside investors and Google Ventures, and then spend that money on Nvidia GPUs in GCP.

Since our institutional note, Google has silently canceled Gemini 3.5 Pro and is now coping by hyping up Gemini 4 instead. As a bridge model, they released Gemini 3.6 Flash, but it’s generally worse than Muse Spark 1.2, Grok 4.5, and the tier 1 Chinese open-source models. Depending on how you count, Gemini is currently in 8th or 9th place, and we don’t see Gemini 4 reversing their downfall.

At the same time, Gemini first-party API token growth has famously decelerated. Whereas they grew 60% in 1Q26, going from 10B to 16B tokens per minute, 2Q26 saw just 38% growth to 22B. This has caused a corresponding decline in Gemini 1P API revenue growth.

However, Gemini Enterprise Agent Platform (formerly Vertex) has been meaningfully stronger overall thanks to third-party models like Claude. To see a full breakdown of Amazon Bedrock, Microsoft Foundry, and Google Vertex revenue separated by model provider, as well as all the implications for OpenAI and Anthropic, see our Tokenomics Model.

There’s no substitute for conviction

We believe Gemini’s core issue has always been a fundamental lack of conviction. Compute is the lifeblood of AI progress, and all the AGI-pilled labs are desperately trying to acquire as much as possible.

Of course, pre-paying multiple GWs is also very scary and expensive, especially if you don’t have a rapidly growing, high margin API business. However, as we’ve explained in previous newsletters, there are creative ways to overcome this limitation and still monetize excess capacity at very attractive rates while maintaining the optionality to claw everything back for your research team. Explicitly, this is what we see happening at Meta and SpaceX.

Google, on the other hand, decided it was totally worth it to sell enormous amounts of compute to Gemini’s fiercest competitors on long term contracts without any hope of ever returning it to DeepMind.

More than 20% of total TPU shipments from 3Q26 to 4Q27 are being sold directly to Anthropic.2 This is excluding the hundreds of thousands of TPUs GCP already rents to Anthropic today, and the many hundreds of thousands more they’ve committed to rent to Anthropic and Meta over the next 6 quarters.

For full details on the number of TPUs, GPUs, Trainiums, etc being produced each quarter till 2030 and their end customers, see our Accelerator Model.

If you’ve ever listened to an interview of Google Cloud CEO Thomas Kurian, you know he is not AGI pilled. In one podcast, for example, he argued that it’s great for TPUs to become “general purpose infrastructure” that supports customers like Citadel, the Department of Energy, and generic high performance computing. And when asked why he was selling compute to Anthropic despite them competing with Gemini, he said this was the natural consequence of Google being a “platform company.”

This same man likely just won a major internal political battle to have even more control over Google’s compute allocation.

Attempting to steelman DeepMind’s frontier ambitions

We’ve obviously been quite bearish on DeepMind thus far, and if we had to steelman the case for why they’ll still be able to train a true SOTA model in the future, it would go something like the following:

• The current setup clearly wasn’t working. With the existing leadership team, their odds of catching up to Anthropic/OpenAI looked extremely slim.

• Now that they’ve cleaned house, the new guys can start from a blank slate. Maybe they’ll even acqui-hire a neolab like SSI or Thinking Machines.

• With this new team, their odds of catching up to the frontier actually increase.

Perhaps there’s some world in which this happens, but we think the odds are basically zero. The issue with Google was not Jeff Dean nor Noam Shazeer, but rather their extremely bureaucratic, painfully slow, and strategically timid culture. Remember that DeepMind had an AI chatbot 1 year before ChatGPT but was not allowed to release it due to fears of disrupting their core business.

Tibo, funny enough, is another great example of brain drain from Google. A 9 year Google/DeepMind veteran, he left in July 2024 to join OpenAI, co-led the original coding agent team, and is now the head of Codex. Meanwhile, Google “acquired” Windsurf to build its own agentic coding platform in July 2025, and they don’t exactly have much wind in their sails thus far.

Let’s also not forget that DeepMind was the original AI acqui-hire! And with the benefit of 12 years’ hindsight, we now know how that played out. Adding another neolab to the roster would likely just result in more of the same.

The Financialization of GCP

Our Tokenomics Model estimates that Gemini ARR was $12B in 2Q26. In contrast, by the end of 2027, GCP will be doing over $73B in third party AI ARR IaaS/TaaS and another $120B of TPU sales. $200B of external sales at high 30s EBIT margins vs a first party business generating just $12B today shows where the focus is. Google management is clear: the short term benefits from focusing GCP on financialization is worth abandoning any long term competitiveness at the frontier.

Despite the weaknesses with Gemini, GCP continues to accelerate. This last quarter, GCP growth was 82%. However, that wasn’t all from what we would traditionally call the “cloud” business. GCP has a new revenue source in selling full systems of its TPUs to external SPVs that run datacenters for customers such as Anthropic. These TPU sales are accounted for on a gross basis at ~$35B/GW. In 2Q26, we estimate that TPU sales were around $1.2B in the quarter, and thus core GCP grew in the low 70s. Given over $150B of TPU systems backlog, these TPU system sales will drive GCP’s 2027 growth rate into the mid 100s vs sellside consensus at 64%.

Although EBIT margins for these system sales are slightly lower than core cloud margins in the low 30% range, we still expect total GCP to deliver mid to high 30s EBIT margins going forward. How investors decide to capitalize the current TPU backlog and any large future sales is an open question. However, given the strong compute demand from labs, we expect more multi-GW deals to be announced soon adding to this backlog. In all, we estimate that over $250B of additional TPU Bookings could be added to GCP RPO in coming quarters. Until then, this eye-popping acceleration in GCP growth coupled with good margins will add ~$3 to Google EPS in 2027.

For an even more detailed breakdown of Amazon, Microsoft, and Oracle’s cloud businesses, as well as updated Google projections come earnings season, see our Tokenomics Model.

Be Careful What you Wish For

Read more

Kimi Delta Attention

Kimi Delta Attention (KDA) is the linear attention layer in Kimi K3’s hybrid attention mechanism. We trace the origins of KDA, starting from linear attention, DeltaNet, Gated DeltaNet (GDN), then to KDA.

Linear Attention

The derivation of linear attention stems from removing the softmax operation in the standard softmax attention. Below we compare the iterative inference formulas, which show the computation of the output vector at token position t:

By removing the softmax operation, we can reorder the operations and reduce the computation complexity of attention from quadratic to linear:

The new equations are as follows:

Vectors q, k, v, have dimensions L by d. The computational complexity of both equations are O(Ld²), thereby making the computation linear. Comparing the new equations with softmax attention’s equation, we see that softmax attention requires accessing all past key and value vectors, whereas linear attention compresses all past key and value vectors into one hidden state S.

We reinterpret the new equations as an online learning objective. We view matrix S as an associative memory that stores the associations between key vector k and value vector v, and we retrieve v by multiplying S with k. We can then interpret the first equation as continuously updating the matrix S at every position to perfect the retrieval. Finally, we can interpret the vt @ kt.T term as the gradient of loss function -(S @ kt.T) @ vt with respect to S.

DeltaNet

Under the online learning objective view, we see the values of matrix S will grow unboundedly: old and new information gets blurred together in S as the sequence grows, which destabilizes learning. Without softmax giving well-scaled and bounded outputs, linear attention typically lags behind softmax attention on long-range recall tasks.

DeltaNet improves upon linear attention by changing the loss function to minimizing the L2 norm of the value retrieval. Unlike linear attention’s loss function, DeltaNet’s loss function regularizes the growth of S. This creates a new matrix S update rule, the Delta Rule, as below:

Source: Linear Attention and Beyond (Interactive Tutorial with Songlin Yang)

The Delta Rule becomes the basis of DeltaNet’s attention equation:

Conceptually, Sₜ-1 @ kₜ - vₜ represents the associations irrelevant to the current key and value, and DeltaNet performs targeted removal of those associations.

Gated DeltaNet

GDN and KDA are adaptations of DeltaNet. Gated DeltaNet applies the LSTM forget gate alpha on the matrix S, allowing the model to control memory lifespan with weight decay. KDA further expands alpha into a diagonal matrix that enables fine-grained per-channel memory decay and positional awareness.

FlashKDA Algorithm

Moonshot developed FlashKDA, their custom kernels for KDA, and open-sourced it. Here we explain the algorithm and derive the arithmetic intensity.

Algorithm

First, let’s start from an alternative formulation of the recurrence formula:

u_t = beta_t * (v_t - (D_t @ S_t-1).T @ k_t)
S_t = D_t @ S_t-1 + k_t @ u_t.T
o_t.T = q_t.T @ S_t

Here, D_t is the diagonal matrix of the alpha forget gate, and u_t is the delta in the delta rule. For decode, the kernel roughly follows the formula. For prefill, we parallelize the operation by unrolling the recurrence formula in chunks of tokens, in order to efficiently execute the operations on GPUs. Assume we unroll token i to j, and the starting state is S_i-1, we get:

S_j = D_j:i @ S_i-1 + sum(D_j:t+1 @ k_t @ u_t.T, t=i:j)
o_j.T = q_j.T @ S_j
      = q_j.T @ D_j:i @ S_i-1 + sum(q_j.T @ D_j:t+1 @ k_t @ u_t.T, t=i:j)

D_j:i refers to the cumulative decay from token i to j: D_j @ D_j-1 @ D_j-2 @ … @ D_i. In FlashKDA’s matrix form, the formula becomes:

S_out = D_j:i @ S_in + K_restore.T @ U
M_qk = tril(Q_decay @ K_inv.T)
O = Q_decay @ S_in + M_qk @ U

The vector to matrix mapping is as follows:

• S_in refers to the state at the starting position of a chunk

• S_out refers to the state at the end position of a chunk

• K_restore is the matrix form of D_j:t+1 @ k_t

• Q_decay is the matrix form of q_j.T @ D_j:i

• Q_decay @ K_inv.T the matrix form of q_j.T @ D_j:t+1 @ k_t, derived from (q_j.T @ D_j:i) @ (D_t:i^-1 @ k_t)

• M_qk is the causal mask, so it’s a lower triangular matrix

U is the matrix form of unrolled u_t. To compute this, we apply UT transform and compute the following:

B = Diag(beta) @ (V - K_decay @ S_in)
L = StrictTril(Diag(beta) @ K_decay @ K_inv.T)
U = (I + L)^-1 @ B

Please consult Songlin Yang’s blog post and Kimi Linear paper section 3.1 for the full derivation. Note that here U corresponds to the pseudo-value term in the Kimi Linear paper.

Implementation-wise, FlashKDA launches 2 kernels: K1 and K2. K1 prepares chunk-level tensors in parallel, including:

a = exp2(cumsum(g))
K_decay = Diag(a) @ K
Q_decay = Diag(a) @ Q
K_inv = Diag(a)^-1 @ K
K_restore = a[-1] * K_inv
L = StrictTril(Diag(beta) @ K_decay @ K_inv.T); INV = (I + L)^-1
M_qk = tril(Q_decay @ K_inv.T)

Here a is the cumulative decay, where each element is the cumulative decay at a token position.

K2 performs chunk-level recurrent computation:

U = INV @ Diag(beta) @ (V - K_decay @ S)
O = Q_decay @ S + M_qk @ U
S = Diag(a[-1]) @ S + K_restore.T @ U

Complexity Analysis

Here we analyze the complexity of an attention head. For decode, the critical path computations are:

• D_t @ S_t-1: Element-wise multiplication, D × D

• S_t-1.T @ k_t: D × D × 1

• k_t @ u_t.T: D × 1 × D

• q_t.T @ S_t: 1 D × D

The decode kernel roughly performs 7*D² FLOPs.

Reading and writing the FP32 recurrent state dominates the memory traffic, so the memory traffic is roughly 8*D² bytes.

For prefill, the critical path of K1 is at computing L, INV, and M_qk.

• L: C × D × C

• INV: Neumann factorization, performs 6 C × C × C matrix multiplications

• M_qk: C × D × C

For K2,

• K_decay @ S: C × D × D

• Q_decay @ S: C × D × D

• M_qk @ U: C × C × D

• INV @ B: C × C × D

• K_restore.T @ U: D × C × D

Combining K1 and K2, FlashKDA performs 12*C^3 + 8*C²*D + 6*C*D² FLOPs. Since we analyzed at the chunk level (chunk size C), assuming sequence length T >> C, the overall FLOPs is T/C * O(C*D²) = O(T*D²).

For memory traffic:

• K1 read Q, K, g: C × D

• K1 write and K2 read Q_decay, K_decay, K_restore: C × D

• K1 write and K2 read INV, M_qk: C × C

• K2 read V and write O: C × D

• K2 read and write S once per kernel: D × D

In total, FlashKDA accesses 3 * 2*C*D + 2 * (3 * 2*C*D + 2 * 2*C*C) + 2 * 2*C*D = 8*C² + 22*C*D bytes. At the kernel level, it accesses T/C * (8*C² + 22*C*D) + 8*D² ~ O(TC + TD + D²).

This concretely shows that the computational complexity of KDA:

• Prefill: Linear to sequence length for both computation and memory

• Decode: Constant to sequence length for both computation and memory

Kimi Linear

Moonshot trained Kimi Linear models as proof of concept for their KDA design, so we can infer Kimi K3’s architecture design from Kimi Linear. Comparing the K3 release tech blog with Kimi Linear, we see Kimi K3 shares the shared expert count, the hybrid linear attention ratio, and the general attention module design.

The diagram above shows the operations performed on the inputs of KDA. For the query, key, and value, we apply linear transformation and short convolution. Applying short convolution effectively capturing local token dependencies, and doing a left padding convolution avoids breaking causality. We additionally apply L2 norm to the query and key to stabilize the eigenvector of the transition and the output matrices. For the decay memory gates, alpha is a low rank projection, and beta is a down projection. The KDA output is normalized per head and controlled by an output forget gate, implemented as a linear transformation in K3, instead of a low rank projection in Kimi Linear. Finally, we apply a linear layer to mix per-head information.

Kimi Linear interleaves KDA with full attention Multi-head Latent Attention (MLA). Kimi Linear showed that 3:1 is the ideal KDA to MLA ratio that balances performance and efficiency. KDA also serves as a strong position-aware operator, replacing the RoPE in MLA.

Moonshot kept MLA layers as full attention, unlike labs that used MLA previously. Namely, Zhipu now uses DeepSeek Sparse Attention (DSA), DeepSeek further evolves DSA to Compressed Sparse Attention and Heavily Compressed Attention. Jianlin, one of Moonshot’s most prominent researchers, cited two reasons for this: They haven’t found a better attention design yet, and they don’t want to apply too many model architectural changes at once. MLA doesn’t suit agentic workloads, and below we explain why.

Multi-Head Latent Attention

MLA performs different operations in prefill and decode. The DeepSeek V3.2 paper defines two modes of MLA: Multi-head attention (MHA) mode and Multi-query attention (MQA) mode. Here we omit the RoPE dimensions for simplicity, and we use the DeepSeek V3 configuration as an example.

Multi-Head Attention Mode

MHA mode is MLA’s default mode for prefill. In MHA mode,

• We do a low rank projection of the hidden state h_t to produce query q

• We down project h_t to produce a kv entry kv

• We up project kv separately to produce multiple heads of keys k and values v

• After scaled dot product attention (SDPA), heads of output o is linear projected to attention output out

The key connection between MHA mode and MQA mode is what serves as KV cache. We save KV entry kv, instead of the keys and values, as KV cache for decode. By saving KV entries, we reduce the memory usage by 42.67x per token.

Multi-Query Attention Mode

MQA mode is MLA’s default mode for decode. To avoid rematerializing the full KV cache, we reorder the operations around attention, derivation below:

After reordering, the differences are as follows:

• We multiply query with the transpose of key’s up projection matrix W_UK

• We use the KV entry (denoted C) for both key and value in SDPA

• We apply W_UV after SDPA

As we show in the diagram, the query is muti-headed, while the KV entry is single-headed. This is effectively multi-query attention, hence the name.

FLOP Comparison

Comparing the FLOPs of MHA and MQA mode, we can first cancel out the query low rank projection projection and the output linear projection. We then observe that the FLOPs that KV up projections incur are roughly equal, regardless of where the projection is applied.

This leaves us with the SDPA. SDPA performs three matrix multiplications: (Q @ K.T) @ V, so the FLOP count is L * d * L + L * L * d = 2*d*L^2. MHA mode performs in standard model head dimension 128, whereas MQA mode performs in latent dimension 512. This means MQA mode incurs roughly 4x FLOPs for SDPA, and overall can cost up to 3.4x FLOPs per token, compared to MHA mode.

Append-Prefill

MLA suits reasoning workloads perfectly. Reasoning often starts with a relatively short input sequence, followed by a large number of output tokens. Each output token is one decode step, and MLA’s reduced KV cache size per token lowers the memory bandwidth requirements, speeding up the decode steps.

However, agentic workloads behave differently. Agentic workloads involve tool calls, which return long tool call outputs. This means the model often operates under the case where the sequence is a long cached input, followed by a long sequence to prefill. This type of workload is referred to as append-prefill (aka extend, append, incremental prefill).

Neither mode of MLA suits append-prefill. MHA mode is not ideal because materializing the full KV cache is too memory intensive. On the other hand, MQA mode isn’t suitable either due to the 3.4x FLOP cost. As discussed in this Twitter thread, modern inference engines (vLLM, SGLang) choose MHA mode but apply the chunked prefill technique to avoid materializing the whole context’s KV cache. To tackle this agentic workload issue, DeepSeek and Zhipu opted for adapting MQA mode to sparse attention in order to reduce the FLOPs. We suspect Moonshot’s future models such as Kimi K4 will feature attention mechanisms that replace MLA.

KV Cache Efficiency

We argue that one should not infer KV cache efficiency solely based on KV cache space complexity. KV cache size is not a standalone factor but a property of the model design: no open weight models are released with static KV cache compression techniques, and model architecture inference efficiency affects KV cache efficiency. The effects of KV cache size also vary, depending on the total memory capacity of a deployed model instance. For example, deploying a model with wide expert parallelism has very different memory profiles than doing so with tensor parallelism, which affects the memory capacity left for KV cache. Thus, we propose considering both the model architecture system efficiency and the KV cache size to understand the KV cache efficiency, and we quantify that with KV throughput.

KV Throughput

KV throughput is defined as KV cache size divided by the prefill time (Time to first token), given a specific sequence length. KV throughput represents the minimum bandwidth required to reliably serve a model with PD disaggregation, but it is also a good proxy for understanding KV cache efficiency. Prefill time encapsulates the efficiency of the model architecture, and as the sequence length increases, we will see the memory-bounded and the compute-bounded situations. As shown in the table below, we can see the benefits of hybrid linear attention become more pronounced as sequence length increases.

This is also a good way to understand the bandwidth requirements of KV cache offloading to different memory tiers in a cluster.

KV Cache Residency

The location where KV cache is stored follows the memory hierarchy. First, KV cache resides in HBM, the fastest memory in a GPU cluster, consuming whatever capacity is left by model weights and activations. As KV cache size exceeds the HBM capacity, it spills into server DRAM, a higher capacity but lower bandwidth memory pool. Finally, when KV cache exceeds DRAM capacity, it spills to disk storage such as SSD. This is analogous to the computer architecture cache hierarchy: register, cache memory, main memory, disk storage.

The analogy continues for memory coherency. Popular distributed KV cache framework Mooncake Store supports write-through and write-back policies for KV cache loading. Mooncake Store features a distributed KV cache pool that makes all KV cache visible to all workers. Implementing write-through policy between DRAM and the lower-level distributed KV cache pool offers multiple benefits in multi-node scenarios, including sharing prefix cache across nodes, avoid KV cache duplication for tensor parallelized MLA, and KV cache redundancy when a node goes down.

KDA Prefix Cache Management

At each token position in a request, Kimi K3 KDA’s recurrent state is fixed in size, whereas standard attention KV cache grows with sequence length. This KV cache space reduction comes at the cost of complicating prefix caching, especially when Kimi K3 is a hybrid attention of KDA and MLA.

Roughly speaking, modern inference engines identify prefix cache hits by matching the longest token prefix in the existing cache.

Identifying the longest prefix becomes a problem for linear attentions like KDA. Without prior knowledge of where the boundary of a prefix is, we will have to cache KDA’s recurrent state at every token position. This means every token has a cache, and the KV cache memory usage regresses to growing with sequence length, defeating the purpose of using linear attention. To tackle this problem, Moonshot saves recurrent states at a coarse granularity, e.g. vLLM caches every 32K tokens. vLLM additionally caches at prompt boundaries, since for agentic workloads, a new turn typically starts at the end of a prompt.

This shows that even though linear attentions like KDA greatly reduce KV cache memory consumption, realistically during serving, they do not consume a constant amount of KV cache memory.

Attention Residuals

Residual Connections

Residual connections are one of the key innovations that allowed us to build bigger deep neural networks through scaling model depth. The deeper the neural network, the more expressive they become but training them naively is hard. Signals from the earlier layer need to be preserved till the last layer and gradient need to survive from output to first without vanishing.

Instead of modeling whole networks as a single function, passing information only through nonlinear transformations, residual networks connect smaller blocks with identity paths. Each block fᵢ ​ learns a change to its input xᵢ, given by the recurrence:

The identity mapping allows features to carry from shallower units to any deeper unit and gives the gradient a path highway so they do not vanish.

While residual connections allow us to build deeper networks, they come with challenges.

Early layers heavily influence residual stream to have effect on final output. Because of which residual stream has irreversible information loss with increasing depth. Later layers increase output gain to have effect on this modified residual stream which can destabilize training. Another variant like highway networks allow gating mechanisms for information flow but they suffer from the same crucial problem. Layers don’t have selective access to information from earlier layers.

Recurrence In Time and Depth

Sequence modeling dominated by recurrent neural networks had the same recurrence formulation.

Where each step has identity mapping with previous state for direct information flow and the sequence model faced the same challenge: depth in the time axis dilutes signal.

Attention machines transformer removed this constrained by retrieving any token in past with powerful and expensive attention mechanism

Attention on residual stream

Motivated by attention mechanism in sequence modeling, kimi developed attention residual, where they take attention over depth blocks,

Standard causal self-attention computes the output of token $t$ as a weighted sum of previous token representations:

Attention Residuals use the same attention mechanism, but replace the sequence dimension with the depth dimension. Instead of attending over previous tokens, each layer attends over representations produced by previous layers.

Unlike standard attention, the query is a learned parameter for each layer rather than being generated from the current token.

For each layer ℓ, we define:

Each colored block represents the output token representation from a previous transformer layer. Just as standard causal self-attention performs softmax attention over tokens in the sequence, Attention Residuals perform softmax attention over the representations produced by previous layers.

Attention residual allows the model to get fine grained control over what inputs to pick from past layers making the model more expressive.

Block Attention Residuals

Attention residual need to all past layer outputs for attention. For large models distributed over many GPUs this creates O(Ld) communication overhead. To overcome this block attention residual dividends layers L into N blocks of S layers. Block AttnRes applies attention over completed block outputs and for current block its evolving partial sum.

Block Attention has minimal tread over full attention residual but they cut down communication from O(Ld) to O(Nd).

Let bₙⁱ denote the partial sum over the first i layers in block n, such that

For the i-th layer in block $n$, the available block representations are

Unlike standard attention, the query is not input-dependent. Each layer learns a query vector:

Attention weights over the available block representations are computed as

The output is the weighted sum of previous layer representations

Rather than depending only on the residual stream to preserve information, Attention Residuals give every layer direct, selective access to earlier representations. This block based variant of attention residuals greatly reduces communication overhead while having competitive performance.

Block residuals show better scaling compared standard residual connection achieving 1.25× compute efficiency. Consistently lower validation loss compared to baseline and gap widening with decay phase. Unlike standard residual networks where output magnitude increases as depth increases. selective aggregation of block attention has bounded output. And consistent gradient magnitude.

Subscribe now

Training

Unlike standard residual networks, attention residuals need all N-1 block input for computation of the Nth layer. This becomes a problem for pipeline parallelism as all N layer blocks output need to be transferred across stages.

With clever cross stage caching and activation checkpointing, Kimi reduced overhead to only 4% compared to standard architecture for pipeline parallelism.

Cross-stage caching

For P physical stages and V virtual stages. Each block N needs C=PV communication for each chunk. Naively this needs transferring all accumulated blocks for each stage. This is quadratic cost growth for each physical and virtual stage

This high communication can be reduced by caching input across virtual stages. Blocks computed in earlier layer can be stored in local memory,

For the first virtual stage all block embedding needs to be transferred in the physical stage, each completed block is stored on respective rank. For all subsequent virtual stages all cached blocks can be reused for computation. Only the block not present on rank need to be transferred for attention to residual computation.

These split communication costs for first and subsequent virtual stages. For the first virtual stage its need incur the same quadratic cost for all physical layers. In subsequent virtual stages we need cached inputs from local devices and Transfer of only PNp chunks needed cutting down total communication from O(C) to O(P)

The cutdown of communication is directly proportional to virtual stages V. Because of this for full stage of one forward and backward pass all computation and communication can be overlapped

Memory overhead

Due to cross stage caching all blocks are stored once across all V virtual stages. With Activation checkpointing all inter-block chunks for attention are eliminated. Each stage activation checkpoint Pl matches memory size of Hl of standard architecture and has no extra memory cost.

Inference

Because Attention Residuals need the output of all previous blocks to compute attention, a naive implementation has excessive memory accesses. To reduce overhead, inference is split into two phases which mirror prefill and decode stages of autoregressive attention. This computation is divided into inter-block attention for completed blocks and intra-block attention for evolving attention in the running block.

Phase 1: Parallel Inter-Block Attention

During decoding, we have to output the completed block and the query vector learned per layer. All inter block layers simultaneously with a single batched query against the completed block representations, returning both outputs and softmax statistics which can be reused for further computation. This phase is similar to prefill phase decoding

Phase 2: Sequential Intra-Block Attention

This phase is analogous to the decode phase, Similar to flash attention, evolving sum can be computed with online softmax for intra blocks combined with precomputed inter block results. Which reduces redundant memory access.

With this two phase design, the IO footprint is similar to standard residual architecture, with only the addition of phase ones inter block computation, amortized by batching all queries in the block.

LatentMoE

LatentMoE compresses the routed tokens before the dispatch operation, and then decompresses them after the aggregation operation. In Kimi K3’s Stable LatentMoE, they apply an RMSNorm before the up-projection (decompressing) operation to reduce sensitivity to scale variations and improve model performance.

Here we explain the design principles behind LatentMoE regarding MoE communication. As shown in the LatentMoE paper, the communication volume is proportional to total routed tokens t, number of active experts K, and expert input dimension d, while being inversely proportional to the expert parallel size E. This is potentially the reason behind Kimi K3’s latent MoE dimension size and active expert count configuration. Kimi K2 series feature 8 active experts with input dimension size 7168, so Kimi K3’s latent input dimension size being 3584 (half of 7168) would allow the active expert count to double to 16 without increasing the communication volume.

However, the ratio of communication to computation time is arguably more important for estimating system efficiency (Discussions here and here). The ratio indicates the roofline of how well MoE kernels can overlap communication with computation at a throughput-bound regime, and expert intermediate dimension size is the only model configuration that affects the ratio. Specifically, increasing the expert intermediate dimension size would decrease the ratio, meaning that the theoretical maximum fraction of communication that can be hidden is higher. Here we derive the formula:

• t: total input tokens across the expert parallel (EP) domain

• K: number of active experts per token

• N: number of total experts

• E: Ranks in the EP domain

• d: Expert input dimension

• m: Expert intermediate dimension

• P: Aggregate bytes communicated per activation element (dispatch + combine)

• F: Effective FFN expert (modeled as SwiGLU) computation throughput per GPU, FLOP/s

• B: Effective uni-directional network bandwidth per GPU, B/s

1. Assuming uniform expert routing, each GPU is assigned t * K / E tokens

2. Assuming uniform expert routing, an average 1 / E tokens are local to the source GPUs, so each GPU dispatches (t*K/E) * (1-1/E) tokens

3. Each token is a d dimensional vector, so the communication volume per token is d * P

4. The communication volume per GPU is (t*K/E) * (1-1/E) * d * P

5. The communication time T_comm = (t * K * d * P) / (E * B) * (1-1/E)

6. The SwiGLU computation involves 3 matrix multiplications:

   1. Up (First) projection: d to m

   2. Gate projection: d to m

   3. Down (Second) projection: m to d

So the computation is 2*d*m + 2*d*m + 2*m*d = 6*d*m FLOPs per token

7. The computation time per GPU is T_comp = (6*d*m) * (t*K/E) / F

8. The communication to computation time ratio is

   T_comm / T_comp

   = ((t * K * d * P) / (E * B) * (1-1/E)) / ((6*d*m) * (t*K/E) / F)

   = (P*F) / (6*m*B) * (1-1/E)

We believe this formula also motivates an increase in expert intermediate dimension to 3072 in not just Kimi K2 to K3, but all recent open weight models, including DeepSeek V4 Pro, MiniMax M3, MiMo V2.5 Pro, and Inkling. As hardware improves and expert weight precision reduces to save memory capacity, the compute throughput increases, so one way of reducing the ratio is by increasing the expert intermediate dimension.

Quantile load balancing (QB)

Many previous load balancing methods require careful hyperparameter tuning. Quantile balancing is hyperparameter free aux-loss free load balancing technique developed by Jianlin Su in Feb 2026 blog post

Base principle QB is the same as auxfree load balancing where router biases are updated dynamically based on the system’s load. But instead of updated bias by some small coefficient like aux-free lb, QB directly computes the next bias from the distribution of router scores relative to routing cutoff threshold. Bias updates become small naturally when the router balances load evenly.

QB tries to find the bias that would have approximately balanced under the current cutoffs and routing on the current batch, solving constraint optimization problems and applying these updates for the next batch. The first constraint is that each token is routed to exactly k experts. The second constraint is a batch of m tokens each picks k experts, gives (mk) assignments in total, to spread load evenly across n experts each expert should process $q=mk/n$ tokens.

Each token finds the cutoff threshold as the (k+1)-th highest biased router score and uses it to calculate the bias update needed to balance load for each expert. For each expert, QB sorts the margins between its router score and every token’s cutoff. It sets negative bias to q+1 the largest margin, leaving exactly q margin above the threshold. Since q/m=k/n, this is (1-k/n) quantile of the margin, which is why it’s called Quantile Balancing.

Inference performance

We are actively tracking Kimi K3’s inference performance on InferenceX.

As of 30th July, all providers on OpenRouter have a floor of $3 per million tokens input and $15 per million tokens output. Both Nvidia and AMD had Day 0 recipes on vLLM, boasting DRAM offload and DSpark speculative decoding.

On InferenceX, we benchmark Kimi K3 serving performance directly on recorded internal claude code traces. We replay an hour of these traces as they reach a steady state. There is a median of 142k input tokens and a median of 444 output tokens per turn with a median of 65 turns per session. The short output tokens per turn is typical for workloads on agentic harnesses, where the agent calls tools frequently, even edits are tool uses.

This benchmark is a big step up from our previous 8k1k/1k1k benchmark, as it truly reflects real-world agentic use cases. From a systems perspective, it is also realistic and closest to production systems. It can reflect KV cache behavior, including prefix cache and KV offloading to DRAM.

For Kimi K3, Day 0 bringup was easier than DSv4 due to better documentation and preparation ahead of weights release. Appropriate images and a speculative decoder model were released at the same time as the weights.

For Nvidia, bringup was simple. But due to the models’ sheer size, it doesn’t fit on a single B200 node. We had to use PP to get it working. DSpark also didn’t work with PP.

For B300, the model fits on 1 node and serves well. After accounting for the weights, GPU HBM can only hold 3.25M tok. In the graph below, throughput goes up as batch sizes increase until concurrency increases above 8. This roughly correlates to the 3.25M tok KV cache budget, and cache starts to thrash, resulting in hit rates falling to < 10% when theoretical hit rate is 95%.

Read more

Today we dig into the world of datacenter construction, because how datacenters are built now bears little resemblance to how the industry has historically done it. Concrete walls arrive as finished panels, mechanical and electrical rooms arrive wired, and sometimes even entire data halls arrive on the back of a truck. Some of the largest datacenters in the world are increasingly assembled the same way you assemble your new Spider-Man LEGO set, only that the bricks weigh 50,000 pounds and are a tiny bit more complex. This is the world of modular construction.

From Hyperscaler to Colos to now even the AI labs, modular construction has become the default playbook for building fast. Our Modular Tracker, included in our SemiAnalysis Industrials Model, tracks over 61GW of modular capacity and 1,000+ sites using some form of modularization or prefabrication strategy. Full breakdown by modular category and equipment type is included in the Industrials Model. We estimate that modular penetration will reach 30%+ of total live capacity by the end of 2028.

Ultra-fast modular designs are increasingly the norm. Over a year ago, we were the first to call out Meta’s drastic change to using “tent” buildings. As shown below, AWS is now rolling out at very large scale their own modular design codenamed “SAMDC”.

To understand the reason, we need to start looking at one of the structural bottlenecks that capitalist incentives alone cannot build past: labor.

Our recent articles have been a journey toward that bottleneck. In “The Case for Space Datacenters”, we showed the ceiling on terrestrial capacity. Last month, in “Stop Saying Half of 2026 US Datacenter Capacity Is Canceled”, we argued that most bottlenecks are misunderstood and solvable. Trade labor is an exception here, as you cannot quickly solve for a shortage of electricians and pipefitters. The race for that talent became a true constraint long ago, visible when operators like Crusoe pumped wages by 30% to bring talent to Abilene’s site, which required over 9,000 workers at its peak.

Aiming to size the labor shortage trade by trade, we now also include the Labor Model as part of the Industrials Model. It translates the state-by-state buildout from our Datacenter Model into hours of demand for every trade and sets them against reachable labor supply. To frame the problem before modularization enters the picture, the chart below is ex-modular construction, with labor demand curve assuming labor per GW stays roughly flat over the forecast period and doesn't yet reflect the benefits we'll cover later in this article. Reachable labor supply in each state, on the other hand, is affected by how much capacity is being built, and how much labor is being pulled, in other US states.

Electricians are a clear case, as they represent 30-40% over the total construction man hours in a datacenter project. The chart below shows an estimated electrician shortage emerging in 2027 driven by the huge mission-critical demand. On a state-by-state basis, the shortage is most acute where the buildout concentrates, like Texas and Ohio.

The response is that every operator and vendor are now racing toward modularization, which essentially means pulling repeatable work off-site and into factories, where everything from wall panels to power rooms and cooling skids are built in parallel with the site and delivered as finished units. Besides, it does more than ease the labor crunch, promising large speed and time-to-build gains. And today, speed is revenue.

The shift is already underway, with Compass and Switch being the first operators to move parts of their datacenter builds offsite. Using some form of skidded solution for electrical equipment is now pretty standard for every operator. AWS’s Project Houdini prefabricates the white-space buildout and collapses the time before servers go in from months to weeks. Meta is standing up fabric-clad “Tent”-like halls. And a wave of new entrants, both in the OEM and the System Integrator space, are building specifically around modular.

In this deep dive we rebuilt the modular case bottom-up against some of the speed and cost claims made by vendors like Vertiv or Schneider, finding that modular construction can compress the construction window by ~36%, or 7-9 months, and is ~8% cheaper on a Capex/MW basis. We also analyze how vendors like Vertiv are able to expand their value capture per project by offering the full stack solution, going from their historical ~$3.5M/MW content to ~$7M/MW with the modular solutions.

The problem, though, is that today everything seems to be modular and vendors, EPCs and Colocation providers many cases are describing entirely different things. To bring some order to this wild west, this article unpacks what modular actually means. We map the vendor landscape building a modular universe of more than 80 players and test whether the vendor claims hold up.

For subscribers, we focus on the main beneficiaries and break down how each player is positioned, from the public names (FIX, STRL, PWR, VRT, SU, FLEX…) to private challengers such as Infra Partners, Bladeroom, and Faith Technologies, distilling the key insights from our recent Core Research subscriber notes on Comfort Systems: “Modular Capex Is The Moat” and Sterling Infrastructure: “Winning Where It Counts: Quadrupled TAM via Texas, Pacific Northwest, and the Midwest; 2X Content per MW from CEC Attach; ~$6B Run-Rate In View”.

To start off, we’d like to thank QTS, EdgeConneX, Aligned Data Centers, Schneider Electric, Applied Digital, DG Matrix, Aran Industries, Karman Industries, Radiant, and Rajat Bhagat for their contributions and insights during the preparation of this deep dive.

The Modular Taxonomy

Before we go into the detail taxonomy, let’s start with the basic definitions, because two words concepts often get mixed up: prefabrication and modularization.

• Prefabrication is the broader concept: any part of the build manufactured offsite and delivered ready to install. It is a statement about where the work happened, not about the shape of the thing.

• Modular is narrower. It refers to the actual self-contained units (rooms, boxes, blocks) that ship complete and get bolted together on site. Every modular unit is prefabricated, but prefabrication may not be directly modular.

Hold onto that distinction, because it is the spine of everything below. From here on, we will walk through the landscape the way a datacenter is actually built up, then we will go into details on the taxonomy that makes up the modular market starting from the ground up.

Understanding The Datacenter Anatomy

At a high level, a datacenter can be seen as simple three stacks of layers: Site, Shell, and Systems.

At the bottom we have the site, or the physical land of the datacenter buildout. This is where grading, wiring, and foundation building take place. This layer cannot be modularized because you have to physically break ground on a parcel of land and pour foundations into it on the set up.

Above that sits the shell, which means the structure, skin, and roof that serve as the backbone and weatherproof the entire datacenter buildout. Inside the shell is where all the equipment and subsystem, including all the mechanical and electrical systems, sit.

Considering that the site itself can not physically be moved, prefabrication strategies are focused on the other two layers, and we take them in that order, working from the outside in.

Modularizing the Datacenter Shell

The shell is the structure, walls, and roof that hold the datacenter up and keep the weather out. It generally follows either a frame-and-skin design, where the structure and cladding are separate, or a load-bearing panel design that combines both.

In a traditional build, both the skin and the frame need to go up on site. Crews break ground, pour the foundation, then form and cure concrete right where the building stands, one piece at a time. A modular shell starts the same way, on a poured foundation, but from there the structure and panels arrive as finished pieces from a factory, craned and bolted into position once arrived.

The time saving is evident from the graphic above. In the traditional cast-in-place buildout, each pour has to reach roughly 75% of its design strength before the next can go on top. Prefabricated structure sidesteps that wait.

However, the evolution did not stop with prefabricating the same conventional building. The larger gains now come from simplifying the building itself: moving from complex multistory facilities toward repeatable single-story halls, and then toward narrower, purpose-built structures.

Phase One: Precast Industrialized the Conventional Shell

The first phase is precast concrete described above. Instead of forming and curing the full structure in the field, panels are manufactured under controlled conditions, transported to the site, and craned onto a prepared foundation.

This is not new. Northern Virginia has used precast extensively for years because construction labor was already constrained. CloudHQ’s two-story LC-2 facility in Ashburn is a representative example: its load-bearing shell supports long, column-free spans and enough structural load to place mechanical equipment on the roof.

Nevertheless, the building still took roughly 18 to 20 months to deliver. Precast reduced field forming and curing, yet the underlying facility remained a large, multistory structure facility.

Tilt-up concrete follows a similar logic but casts the panels on the building slab rather than in a remote factory.

This method avoids long-haul transportation and can be the lowest-cost route for a large single-story box, although quality and schedule remain more exposed to site conditions and weather.

Phase Two: Simplifying the Building

The second phase is where design changes happen. In order to further speed up the time, the industry turn towards alternating the design for simplicity. These buildings use regular structural bays, fewer architectural features, and standardized exterior panels. Steel is often favored because the frame can be fabricated off site, shipped efficiently, and bolted together quickly across a large flat campus.

At the light end is the pre-engineered metal building, or PEMB built in three parts:

1. The primary frame serving as the structural skeleton

2. The secondary frame tying the main frames together, these are lighter steel member like roof purlins that span between the primary frames.

3. The skin, keep in mind this is different from the frame. They are the thin light weighted metal panel whose job is to protect the interior from extreme weather conditions

This is the fastest option since all parts arrive cut, punched, and labeled, a crew simply need to bolt them together on the site. Furthermore, a light steel structure needs far less material than concrete.

More demanding halls use structural steel - heavier, hot-rolled beams and columns fabricated off site and bolted into a rigid frame. It costs more than a light PEMB but supports wider spans, heavier loads, and more complex layouts.

QTS’s Cedar Rapids campus shows the speed and scale this approach can unlock. The current 420 MW phase spans approximately 2.8 million square feet and uses roughly 28,000 tons of structural steel. QTS moved from groundbreaking to topping out in about five months, with the broader building delivered in approximately 11 months.

The exterior is then closed with prefabricated cladding, most commonly insulated metal panels. Similarly, we see this approach with Crusoe’s Stargate campus in Abilene, each building used roughly 672 factory-made panels. The panels were fabricated in under 40 days and installed at approximately 15 to 20 per day, helping bring each building to a dried-in shell in under eight weeks.

The speed advantage therefore comes less from steel itself than from what it enables: simple single-story halls, repeatable structural bays, fewer field interfaces, and a supply chain that can be reproduced across markets. It can also reduce labor and structural material per MW compared with a dense multistory design.

The main trade-off is land. Single-story campuses require more acreage, but that is often acceptable in newer AI markets where land is cheaper and deployment speed matters more than maximizing MW per acre.

Phase Three: Purpose-Built Rapid-Deployment Shells

The third phase pushes simplification further by designing the enclosure around a more specific deployment model. Narrower, lighter structures can reduce the amount of conventional shell work and support faster repetition, although tighter optimization may leave less flexibility for future equipment or layout changes.

Meta’s rapid-deployment structures at Prometheus campus in New Albany are the most visible extreme. The aluminum-framed, fabric-clad halls provide enclosure and weather protection without constructing a conventional permanent shell. Each structure is roughly 125,000 square feet, and satellite tracking showed eight standing by April 2026 after the buildout was announced in July 2025.

That does not mean Meta completed a full datacenter in nine months. The tents accelerate the enclosure, not utility interconnection, power, cooling, or commissioning. They also trade away some of the durability and long-term flexibility of a permanent concrete or steel building.

AWS is moving in a similar direction with its newest modular builds. Rather than treating the shell as a large generic warehouse, AWS is using narrower and more repeatable structures organized around the systems installed inside. The result is less building per MW, shorter structural spans, and fewer interfaces for field crews to assemble.

The common thread is that shell modularity is increasingly about design simplification, not just prefabrication. Precast moved concrete production off site but largely preserved the conventional building. Standardized steel made single-story halls easier to repeat across markets. Purpose-built structures go further by reducing the size and complexity of the shell itself.

The cost savings thus come from removing floors, reducing structural complexity, decreasing labor on site, and repeating the same enclosure / supply chain across the campus. Once the shell is dried in, the larger modularization opportunity moves inside, to the power, cooling, and white-space systems that turn the enclosure into an operating datacenter.

Modularizing the Equipment and Subsystems

Equipment and subsystems are where most of the real modularization is happening, and the offerings span an enormous range, from a single piece of equipment all the way up to an entire building delivered ready to switch on. Previous deep dives already covered in big depth the anatomy of Mechanical and Electrical systems. Besides, before we start naming categories, it helps to fix some vocabularies:

1. Component: The lowest level of form factor. It is a single piece of equipment manufactured in a factory.

2. Skid: First common modular form. It is a group of components mounted on an open frame. Instead of shipping each piece separately, the equipment is pre-arranged, configured, and shipped together as one package.

3. Module: A skid but in an enclosed space. It can go anywhere from a simple power room to a prefab mechanical/electrical room similar to a skid, but only it will become a module if you put walls and roof on top of it.

4. Container: A specific form of module using ISO container for packaging. An ISO container is built specifically for standard shipping dimension, which means it can travel anywhere on normal transport with a truck without limitation.

5. Prefab Datacenter block: Facility scale buildout that stitches multiple factory-built module into a much larger facility block. The intent of these block is to serve as an end-to-end datacenter buildout

This is like a ladder, from 1-5 increasingly factory integration and scope. If you look carefully, you may also be able to realize the first 4 levels are better known as the subsystem modularization. This is where the supplier delivers one part of the datacenter as a factory-built unit. That unit is arrived assembled and tested, but still has to be connected into the broader facility before it becomes useful.

The last level and sometimes the fourth level, moves closer to a whole facility modularization. Here, the supplier is delivering a much larger portion of datacenter as an integrated product.

Subsystem Modularization

With the vocabulary in hand, we can now start climbing the ladder, and the natural place to begin is at the bottom, with subsystem modularization. This is the larger of the two families and where most of the market lives today. We start off with the grey space, and modular power block is where most people think of when discussing modular design.

Modular Power Blocks

A power module is a factory-built electrical room or power block that packages the major electrical equipment into a containerized box. Among all subsystem modules, power is one of the most natural areas to modularized because the equipment lineup is well defined and lots of different pieces are needed to assemble the units. The electrical fit-out and commissioning take on average 5.5 - 16.7 months on a 50MW power hall.

To illustrate with an example, let’s see Flex’s modular power solution, built through the Anord Mardix unit below:

Flex sells this in two versions, the power skid and the power module pod, which is the same lineup wrapped in a secure enclosure. Inside the room, you will find major electrical components laid out like a power train.

As you may have noticed, the module also contains the busway system like the IBAR feed above the building which bridges the power pod to the data hall. The CRAH units, or computer room air handler and fire protection system, also exist to help manage the air temperature inside the enclosed shell and protection against unexpected conditions.

By our own estimate, the power block is where the schedule payoff concentrates. Moving just the power scope into the factory, roughly ~26% of the build by content, gets a hall to IT-ready ~22% faster, ~13 months against ~16.7 for stick-build, and around ~5% cheaper per MW, largely by compressing the mechanical-and-electrical fit-out from ~5.5 months to ~2.5.

Lastly, within modular power blocks, a new subcategory is emerging, which we can define as “software defined” power routing blocks. Instead of packaging the conventional transformer+switchgear+UPS+battery chain into a box, companies like DG Matrix replace portions of that chain with power-electronics-based multi-port routing. These systems can connect grid, generation, storage, and DC loads through a common controlled power platform.

Modular Cooling Blocks and Prefabricated Cooling Infrastructure

Similar to a power module, the cooling block packages the datacenter’s cooling loop into one integrated subsystem. At first glance, the case for modularizing cooling is weaker than for power, since there are simply fewer pieces to pre-assemble. But as primary and secondary loops grow more complex, the ability to add cooling capacity in repeatable, modular increments becomes far more attractive to operators.

Focusing on the TCS loop, the majority of modular efforts are centered on skids for CDUs. Airedale by Modine’s skid-based CDU (for more on cooling systems, read our Cooling deep dive here) is an example of this category.

The Airedale by Modine skid-based CDU is a 2 MW-class unit arrives on a pre-manufactured skid with everything included, from the red-and-silver cooling loop to the buffer tanks on the top right, and even the leak detection system built into the end of the skid. On site, all the crew needs to do is hook up the two loop connections and a power feed.

On the other hand, the value proposition for prefabricated cooling systems is stronger when looking at the secondary loop and the outdoor mechanical yard, where piping and other outdoor cooling infrastructure is prefabricated, significantly reducing the civil, piping, and controls work completed onsite.

When thinking about cooling equipment, keep in mind that much of it was originally designed for hospitals, campuses, and industrial process cooling rather than GW-scale datacenters. At that scale, large footprints and hundreds of co-located units can create issues such as hot-air recirculation and heat islanding.

The Vertiv installation above shows a conventional mechanical yard. The cooling plant sits outside the data hall as a separate piece of infrastructure, with chillers, pumps, piping, and supporting steel assembled around the building.

Some datacenter developers, like QTS, are now prefabricating most of the piping infrastructure, which allows for faster installation while maintaining high quality. That becomes especially valuable today, given the increasing piping requirements of dense liquid-cooled deployments.

A handful of new entrants are now even attacking the whole yard. Karman Industries’ CO2-based Heat Processing Unit (HPU) is a purpose-built unit borrowing SiC power electronics and permanent-magnet motors from EVs, and compact turbomachinery and advanced heat exchangers from aerospace. The HPU, which is configurable to each site, ships as an outdoor-rated NEMA skid at 4 to 5 times conventional power density and cuts yard footprint by 60 to 80%.

Other Modularized Options

Power and cooling are not the only parts of the datacenter that can be move into the factory. The same concept can be apply across the gray space such has the energy storage (BESS) system, water treatment skids, Fire safety system, and many more.

Many are not on the critical path for construction timeline, or they are small enough to be able to build on site. Moreover, where these systems do get modularized, they often ride along inside a large unit rather than shipping separately. For example, the Schneider EcoStruxure for example was built in with the fire protection system.

Factory Built White Space

The intention of the factory built white space module is to replace hand-built data hall and manual on-site wiring with a unit made factory product. This means operator can directly put compute in place without the need to figure out wiring and connection.

The whole package comes ready for the rack frames, and all the last mile connection points the rack need to operate. Think of it like a prepared envelope for compute, its organized where racks go, provide the cable and connection to how they receive power and extract heat, and include prefabricated power busway and technical water loop situated above the rack.

Take Schneider’s EcoStruxure Pod as an example. The image above shows the black cabinets forming the IT rack rows where products like Nvidia GPU servers would be installed, a total of up to 40 racks can be placed in this one system. Above the racks, the gray overhead infrastructure is the distribution layer. It carries the busway that delivers power to each rack, containment to capture hot air as some of the solution will still be air cooled, technical water loop to distribute liquid cooling that extracts heat, and cabling that connects each servers.

The interesting thing about this is the product design behind it. A factory white space must be able to serve different custom needs and therefore, suppliers like Schneider work with Nvidia to support more than 30+ reference designs. The buyer can essentially pick the specs that matches the chips it wants and gets a hall that is pre-coordinated with the matching required power and cooling module. In the coming section we will study in more detail how the design process takes place.

Whole Facility Modularization

Whole Facility modularization is the literal datacenter-in-a-box model. Instead of delivering individual parts, the supplier delivers a complete or near-complete datacenter block.

Containerized Datacenters

Starting with a containerized datacenters, this is the 4^th part of the form factor design where the datacenter itself is packaged into an ISO-style container or purpose-built weatherproof enclosure. Like we had discussed, the reason why this option exist is for the ease of transportation.

As the image shown above, you can see almost everything inside the box: the IT racks, the power equipment, the batteries, and even the cooling system.

These types of design are commonly used in edge computing, industrial environments, remote or unused spaces, and the product is most useful when the buyer needs a smaller datacenter quickly. For AI workloads, the buyer is usually not a compute startup chasing scale but an asset owner that needs low-latency inference at a fixed physical location.

The main limitation of this buildout is density. The same form factor that makes the unit portable and fast to deploy also fixes the layout. That’s why suppliers are pushing beyond the containerized model toward all-in-one prefab datacenter blocks.

All-in-One Prefab Datacenter Block

The all-in-one prefab datacenter block is the more ambitious version of whole-facility modularization. Here the supplier delivers a larger facility block with more of the datacenter already integrated before delivery.

Vertiv MegaMod shows what this can looks like in practice. The structure effectively is a modular datacenter with the major systems packaged into one enclosure. The center of the block contains the IT racks, where servers are installed. Above and around the racks runs the fiber optics and cable-management pathways, along the perimeter are all the supporting infrastructure systems like cooling and power units.

The system’s 1 MW reference design can stretch to approximately 26.5 meters long, 24 meters wide, and 4 meters high in dimensions. While the MegaMod plus version can extend to as much as 31 meters wide. You may be wondering, if it’s this big how can it be ship to the site? In reality, the structure needs to be broken into transportable prefabricated sections, shipped through a standard heavy-haul logistics truck, and then connected and commissioned as one.

Platform Modularization and Reference Designs

The last step in whole-facility modularization moves beyond any single vendor’s block into a standardized reference design for the facility itself. In the same fashion the industry has reference designs for rack systems and CDUs, Nvidia now publishes one for the entire AI factory: Nvidia DSX. It was first unveiled as an Omniverse digital-twin blueprint at GTC Washington in October 2025, formalized as the Vera Rubin DSX reference design in March 2026, and more recently expanded into the full DSX platform.

The DSX reference designs are validated AI factory architectures covering compute, networking, storage, hardware cluster design, and also the facilities side, including power, cooling and controls. Even civil, structural, and architectural design. The value proposition behind is that Nvidia’s DSX Max-Q maximizes token per watt within a fixed power budget, and DSX Flex facilitates the connection the facility to grid services, dynamically adjusting power draw and orchestrating demand with hybrid onsite generation.

When deploying a DSX facility, through the Omniverse DSX Blueprint, an operator first builds a digital twin of the facility, simulates layouts, power topologies, thermal behavior, and operational policies in real time, and optimizes the design before construction begins, then reuses the same validated architecture across sites. For example, CoreWeave is already using DSX Air to build and test digital twins of its AI factories.

Besides, the whole DSX ecosystem includes pretty much all the supply chain: Cadence, Dassault Systemes, Eaton, Jacobs, Nscale, Phaidra, Procore, PTC, Schneider Electric, Siemens, Switch, Trane or Vertiv. Vertiv’s OneCore, for example, packages power and cooling into standardized 12.5 MW pods that can be combined into larger AI-factory deployments

EdgeConneX estimates that a common design can advance a project to roughly a 30% to 60% permit set before site-specific localization is complete, allowing substantial off-site work to begin earlier.

Vendor landscape

If you have made it this far, you should have a working feel for the categories. You should also, maybe be a little buried in names. We have put a power module from Flex, a CDU from Airedale, a data-hall pod from Schneider, and an all-in-one datacenter from Vertiv all in front of you. So before going further, it helps to step back and lay the whole market out on a single map.

Our universe runs to over 80 players, and laying them out this way is useful because two patterns jump out right away.

1. The depth is in the subsystems. The power-room and cooling modular are by far the most crowded

2. The same names keep showing up across columns, because a vendor like Vertiv, Schneider, or Eaton sells a power module, a CDU, a white-space pod, and a whole block all at once.

Owning the Integration: Who actually does the modularization?

The vendor landscape above maps who builds each piece, but not how those pieces become a module or who is on the hook when they do. From the solution provider’s point of view it has three answers:

1. At one end the operator holds both rights: it specifies the equipment, buys it directly, and hands it to an integrator purely for assembly.

2. In the middle sits the EPC- or integrator-led buildout, where the operator still sets the performance requirements but hires an EPC or integrator to source, coordinate, and build.

3. At the far end is the OEM-led model, where a vendor like Vertiv designs and sells its own stack as one finished product, as it does with the OneCore portfolio

Operator-Led Modularization

This is where the operator engineers the specification themself, procures the equipment directly as owner-furnished gear, and hands it to an integrator purely for assembly.

This model requires the operator to have a deep in-house engineering and procurement team to specify and source every component, and the willingness to carry all of the cost, inventory, and lead-time risk.

In a supply-constrained market that risk is sharp, since the operator is competing for scarce transformers and switchgear without a vendor’s allocation leverage, unless it buys at enough scale to have that leverage of its own.

That is why operator-led modular is effectively confined to the largest hyperscalers. AWS, for example, engineers its own prefabricated data-hall skids under Project Houdini and procures the equipment directly, using Cupertino Electric as design partner.

Aligned is another example of operator-led modularization, although it relies on external integration partners for manufacturing capacity. Aligned defines the architecture, owner-furnishes the major components, and controls the commissioning and quality program, while its integration partners receive, store, assemble, and test the equipment across multiple factory locations. The partner provides the production footprint, but the modular system remains Aligned’s design.

System-Integrator or EPC-Led Modularization

The EPC-led modularization includes system integrators or construction companies that take mostly third party equipment and convert it into a skid/module. The integrator does the assembly, installation, factory testing, enclosure, and is the party in charge of delivering the finished skid to the end customer.

The companies acting an integrators are both construction companies that have the footprint and capabilities to do the integration, like Comfort Systems, Sterling Infrastructure or Quanta’s Cupertino Electric, and specialized modular integrators, like PCX, Nautilus, DXN, Infra Partners, Bladeroom, etc.

This type of modularization is vendor agnostic, which means that the customer keeps more control over the datacenter design, while the contractor moves part of the construction sequence into a prefab shop. The EPC buys the equipment, assembles it, wires it, pipes it, tests it, and ships it as completed construction scope.

Take Comfort Systems as an example. It is a MEP contractor, not an equipment maker, so it acts as the layer that procures and assembles the gear on the operator’s behalf. The operator decides what equipment it wants and Comfort Systems does everything else. It runs that work through Environmental Air Systems and TAS Energy across 3.5+ million square feet of shop floor in Texas and North Carolina.

That concept is particularly attractive to hyperscalers. A big operator usually already knows exactly what equipment and design it wants, so it has no interest in buying someone else’s fixed system. Working with EPC integrators like the kind for Comfort System, the operator keeps its own design and its own gear, and simply hands the building of it to a factory instead of a jobsite.

OEM-Led Modularization

Here the equipment vendor turns its own datacenter infrastructure stack into a repeatable module or platform. Although the product can still be configured for a specific site, the starting point is usually an off-the-shelf module using OEM’s own architecture.

Vertiv OneCore is a clear example. Instead of selling single equipment devices, Vertiv combines all the layers into a single modular platform. This platform integrates Vertiv’s power, thermal, cooling, and IT infrastructure technologies inside a Vertiv’s supplied steel shell.

That is what makes it OEM-led, the customer is buying into Vertiv’s entire integrated stack rather than asking an EPC integrator to assemble equipment line ups. This also allows Vertiv to capture higher content by selling the entire stack end to end, with the TAM expanding up to ~$7M/MW for some of their full-stack solutions.

The tradeoff is capacity and execution risk. Besides the fact that these companies are going up the value-chain toward market segments they were not previously involved in, OEM-led modularization can only scale as fast as the OEM’s factory capacity, supplier base, and integration teams can support. Vertiv’s modular solutions run lead times of over 12 months today. This is also pushing big OEMs, not only Vertiv but also companies like Schneider and Siemens, to be selective with capacity slot allocation, requiring certain capacity minimums and favoring bigger projects. As a result, operators or developers looking for smaller scale capacity are increasingly working with the System Integrators.

The Modularization Cycle

By now our readers should be familiar with the different forms of modular solutions and the players leading this transformation. They will also have noticed how different this looks compared to traditional datacenter construction, and will have many questions about the operational implications. Those are the topics we address in this section.

Stage 1: Solution Design and Simulation

Engineering the solution is the first step. In a field build, some of these decisions can change while construction is underway. In a modular build however, they need to be predetermined, frozen early, and repeatable since the box itself must come in a finish block.

The most important engineering at this stage happens at the facility level. Before a module can be finalized, the design team must define the load, select the equipment, develop the single-line diagram (SLD) and layout, and complete the short-circuit, protection-coordination, and arc-flash studies in tools like ETAP and PSSE. These analyses determine how the system is sized and which components can be used. Because they depend on the full electrical path from the utility connection through the downstream equipment, they cannot be completed on an isolated skid. The facility design therefore has to come first, with the module designed as part of that broader system.

Source: Aran Industries

For 415/480VAC this work is well templated and repeatable. When considering all the implications of the 800VDC transition, it is not as well-templated. A handful of reference designs exist, but none are well baked yet, so the facility-level architecture is still being worked out design by design. This is also the part of the cycle now being automated. Companies like Aran Industries are building build custom software that plugs into ETAP, PSCAD, PSSE, Revit and the other design tools, compressing what is otherwise a multi-month (>2 months), multi-engineer electrical design process into hours of compute plus a single engineer reviewing the output.

Ownership of the Buildout

Once the design is set, ownership comes down to two questions:

1. Who chooses the equipment;

2. Who carries the cost, the inventory, and the lead-time risk of the buildout.

Those decisions do not always sit with the same party. An operator may specify a component directly, or an integrator may select it and seek approval.

Colocation operators are a clean exception. A wholesale colo is not bound to any single end-user’s specification, and often does not even know who the tenant will be, so it is free to choose the equipment it wants and commit to it early, without waiting on anyone’s sign-off.

Even then, local availability matters, because switchgear and transformers may carry 12 to 18-month lead times, while generators can require market-specific emissions controls. Customization can also reopen engineering and add roughly eight weeks.

Stage 2: Packages and Documentation

Once the design is frozen in Stage 1, what leaves the design stage is not a single drawing but a set of documentation packages, produced by different parties in different tools:

• An issued-for-fabrication (IFF) package with the shop drawings telling the factory how to build the skid

• An issued-for-construction (IFC) package telling the site how to receive, place and connect it

• A separate permitting and commissioning documentation set, proving the design will pass code and testing.

This stage is largely a documentation and paperwork exercise. Automating the Stage-1 design model lets these packages be generated from one source rather than be recreated by hand, removing the manual redrawing between parties (IFF and IFC serve different audiences, the factory and the field, and are issued separately; they are not the same document and do not drift against one another).

Stage 3: Assembling the module and Factory Testing

Once all the prework is completed, the module still has to be physically built. Think of this step as the production line, but in a much bigger scale of a datacenter. The assembly process starts with a base, a skid, or a frame that sets the foundation of the module. From there, equipments are layered in sequence, with each component placed into its designated area like how you will assemble a LEGO building.

What runs alongside the assembly is the factory acceptance test, or better known as “FAT”. Testing happens at two levels. The first is at each station as the module is assembled, an inspection gate where the work is checked before the module advances. The second comes once the module is complete: the whole unit is powered up and run the way it will run on site to confirm its rated load and everything is in place properly. By the time it leaves the line, the module is cabled, labeled, sealed, and proven to work on its own.

Factory testing is critical. The industry often describes this through the 1-10-100 rule: a defect that costs $1 to fix during design or assembly may cost $10 once production begins and $100 after the product has shipped. Furthermore, standardization also makes testing repeatable, so same FAT procedures can be run multiple times across production.

Stage 4: Delivering and installing the modules

Now, the module leaves the factory’s controlled environment and runs into the uncontrolled area of site logistics. At the scale of today’s AI campuses, bringing the first block online is no longer enough, because operators also have to think about time to the last megawatt. The critical questions become how many finished modules can reach the site, how many can be unloaded and set in parallel, and how effectively limited rigging and installation crews can move from one block to the next.

A closer look at: Transport and Logistics

In our conversations with datacenter developers, logistics appears as one of the main challenges equipment modularization presents. All these skids and modules are big and heavy, and shipping them from the factory to the site is not a minor task. That is why footprint and location matter so much, with integrators expanding their footprint to sit closer to their customer’s sites.

Federal law fixes the no-permit envelope at 102 inches wide and 80,000 pounds gross, leaving about 24 tons for the module on a standard deck. Microsoft’s Azure Modular Datacenter and Schneider’s Easy Modular ride inside a 40-foot ISO container at 96 inches wide, so they cross any state line or fly in a C-17 permit-free.

Past that threshold, you need a permit, although the truth is that permit cost has little impact and the real implication is on schedule. A standard oversize permit runs just $15-100 a state, and even the line-haul, at $12-14 a loaded mile, makes a 500 mile move only six to seven thousand dollars a trailer.

On time, loads past roughly 16 feet become superloads, triggering a bridge-engineering review that runs 7 to 21 days per state and stacks toward months across a route, and escorts force restricted travel windows. Besides, the thresholds are not even uniform, as a module engineered to clear Virginia (superload at 18 feet or 250,000 pounds) can still trip Ohio’s far lower 14-foot, 120,000-pound line.

Transport also becomes part of the reliability program. A module may experience greater mechanical stress on the road than during normal operation, particularly through vibration, braking, and loading. In one validation exercise, Aligned DC shipped a 3 MW module from Utah to Omaha and back with force loggers to measure the conditions it experienced in transit.

On top of all this, transport can also be constrained by the insurance it carries. High-value AI racks may be shipped only one or two at a time because concentrating too much equipment on a single trailer creates an unacceptable insured loss no insurance company is willing to bet on. The risk is not theoretical and we have heard cases where truck carrying large UPS module tipped over on a road in West Virginia while traveling toward Northern Virginia, leading to large reimbursement.

Considering all these implications, operators are designing around the haul, like AWS engineering Houdini’s skids onto low double-drop trailers to stay under bridges. Also Nautilus floated its datacenter 50 miles to the Port of Stockton on a barge, and Compass put a Schneider module factory next to its Red Oak campus. That said, some operators like DXN do manage to ship containers from their factories in Perth, Western Australia all the way to the US. These are, however, mostly smaller containers.

Once the module reaches the site, it must be lifted onto the prepared foundation, anchored, and connected to upstream power and downstream load. Even crane selection becomes part of the design process. A crawler crane’s lifting capacity falls sharply as the load moves farther from the boom: a Manitowoc 18000 can lift roughly 600 tons at 7.3 meters, but only about 10 tons at 104 meters, so the module’s weight, lifting points, and final landing position determine which crane is required and how much the lift will cost, typically around $5,000 to $25,000 per day. A single Schneider 500 kW power module, for example, weighs 50,000 pounds, needs six lifting points, and its load distribution is not known until it is built.

Stage 5: Site Commissioning

With the modules in place, it’s time to commission them. Factory testing proves the individual unit, while commissioning proves the system under real conditions. Operators consistently call this the single biggest gap in the modular cycle, with a full site commissioning running 3 to 8 months end to end. While some vendors claim it can be shifted offsite, in practice the parts that matter can’t be, because the major energy sources (the utility feed, the generators, the BESS) only meet on site.

The levels of commissioning

The industry generally runs a 6-level ladder, while some frameworks add a Level 0 design review at the front.

• L1 - Factory Witness Test (the “red tag”): each skid or module proven as a standalone unit at the manufacturer. This is the level factory testing covers, and the level modular front loads.

• L2 - Delivery & installation verification: the unit is received, set, anchored and inspected on site.

• L3 - Pre-functional / startup (the “green tag”): each system energized and started up on its own.

• L4 - Functional performance testing (the “blue tag”): each system run to the switchgear, the cooling loop.

• L5 - Integrated Systems Testing / IST (the “white tag”): every system energized and running together, on site, under simulated failure, an A-side outage, a UPS/STS transfer, a pump failover, a black-building start.

From L2 onward the work has to happen physically on site, because the sources being tested against like utility feed, generators, BESS are only present there.

To understand where time can actually be recovered, we identify two levers. First, parallelize commission modules and subsystems as they land rather than waiting for the whole site, so L2–L4 on the early modules overlap the delivery of later ones and only L5 has to wait for the full set. Second, reuse the plant as its own load bank: the power equipment and its battery storage can serve as the load bank during commissioning, so instead of trucking in rented gensets and resistive banks that leave when the test ends, the same gear that proves the plant stays on as part of it.

Parallel Factory Commissioning

At scale the binding constraint is usually the Level 3 point-to-point work, because every PDU, control panel, and field device must be wired, addressed, named, and verified back to the BMS. Across hundreds of PDUs each exposing dozens to hundreds of points, this is the real commissioning bottleneck. To move faster, some big datacenter operators are running parallel factory commissioning tracks alongside the standard on-site process, completing equipment verification and control checks before shipment (repeating some tests post-transport) while the site team keeps delivery checks, live interconnection, and integrated-system testing.

This consideration also matters when studying aggressive project timelines claims. Some 6-9 month schedules are achieved partly by compressing or minimizing the commissioning process. That may accelerate initial turn-up, but it significantly shifts risk into operations.

Putting the Value Proposition of Modular Solutions to Test

All modular pitches mainly rest on three claims: (1) Speed to market; (2) Quality of Build and Safety; and (3) Total Cost of Ownership. The marketed numbers are huge figures, with Vertiv claiming MegaMod at up to 50% faster on module deployment and SmartRun at up to 85%, and Schneider claiming 60% faster and 13% lower first cost on power and cooling.

Those are big (big!) numbers, so we rebuilt the case bottom-up against our baseline reference datacenter that we include in the SemiAnalysis Industrials Model, a 50 MW liquid cooled AI hall in the US. As headline, our numbers estimate a ~36% shorter construction window for modular construction when considering the full construction timeline, and a ~8% lower all-in cost. Let’s break down these figures.

The Speed Advantage

Speed is undoubtedly the core vendor selling point. Following our previous taxonomy, the realized savings will depend a lot on which modular solutions you use in the build. To understand where the speed savings come from, we need to understand the construction timeline in commonly four blocks:

• Groundworks: 4-6 months, averaging about 5 months. Modularization changes little here beyond simplifying some pad and foundation work.

• Structure and shell: 2.5-4.5 months, with a midpoint of about 3.5 months.

• Mechanical and electrical fit-out: 7-11 months. Today’s baseline already includes some preassembled electrical equipment, such as LV switchgear and UPS lineups.

• Commissioning to IT-ready: 3-8 months, with a single-hall midpoint of about 4.5 months. Longer timelines usually reflect phased, multi-hall handovers.

Added together, a conventional 50MW building runs about 18 to 24 months from groundbreaking to IT-ready, with permitting excluded from that clock. In addition to the mentioned commissioning considerations, faster timelines claims are also sometimes related to a counting convention, often starting the clock at the electrical fit-out or the vertical works rather than at groundworks. Besides, permitting adds another 12 to 13 months that cannot overlap with construction, taking the all-in timeline to roughly ~30-35+ months for a stick build against ~24-30 for modular.

In the table below, which considers a scenario of an operator going full modular, modularization can compresses the construction window to about 12-18 months, around ~36% faster than a pure stick build and about ~30% faster than today’s baseline that considers some modularization.

The savings scale with how much scope moves to the factory. An operator taking only MEP skids, midway between today’s baseline and a full modular build, lands around 17 months. That schedule can be shortened even further when deploying all-in-one prefabricated block or containerized datacenters, which can take the building window all the way down to about 12 months. Today we can hear some claims even for under 12-month deliveries.

Digging deeper, during the MEP fit-out, an AI data hall absorbs about 12,000 field man-hours per MW, with electrical the dominant trade. A standard 50 MW hall concentrates 600,000 field hours into one building and stacks about 300 craft workers at peak. Relocating that MEP scope to the factory, on-site hours fall about ~63% to 4,500 per MW and licensed-electrician hours about 85%. The window itself compresses by less than the hours because the relocated work now runs in the factory in parallel with sitework. In addition, moving work into the factory also reduces dependence on weather and geographic constraints.

Speed equals money

Speed is compute and compute is revenue, so the true value of pulling go-live forward is the opportunity cost of the delay. In our calculations we consider as a rule of thumb that today, for CSPs, a megawatt of IT load generates on the order of $12M to 15M of revenue a year, or about $1-1.25M per IT MW per month. That said, in such a capacity constrained market, we are seeing new deals being made on much higher disclosed pays, see SpaceX and Anthropic deal, or as we recently covered in our Meta newsletter post, revenues of $50M per IT MW. Companies like Anthropic or OpenAI are making over $50M/MW on API.

On the COGS side, the relevant cost is GPU depreciation, at an average $30M/MW for an Nvidia cluster depreciated over five years, roughly $500,000/MW per month. An idle month strands at least that depreciation, on GPUs already on the clock whether or not the hall is ready, so we value an earlier month to the owner-operator at about $500k/MW, a conservative figure since the contribution margin on live compute is higher. For a wholesale colocation operator that never owns the GPUs, the value of an earlier month is just the lease it can now bill, about $190,000/MW per month, roughly $190/kW/mo.

Applying those unit economics to the roughly 8-month lead over a pure stick build, the owner-operator captures about $200M undiscounted across the 50 MW hall, or ~$4M/MW.

Of course, all of this is conditional on the building being the binding constraint. Go-live is the latest of three dates, building ready, power available, and GPUs delivered, and accelerating the building earns nothing if it was not the date that bound.

The TCO Advantage

Compared to the realized speed benefits, Capex savings are not meaningful. We estimate that same full modular 50 MW liquid cooled hall costs about ~$1.1M/MW less all-in, ~$13.5M/MW against ~$14.6M/MW, just under an ~8% delta.

When looking at the all-in content/MW for the datacenter equipment, we can separate between hardware cost and service or installation cost. The hardware piece itself does not change much: switchgear, UPS, transformers, and other core hardware cost the same whether they are installed in a field-built room or integrated into a factory skid. The gross savings come from two places mainly, the labor portion and a shorter build time.

We estimate that moving MEP into the factory saves about $0.6M/MW on construction services and $0.5M/MW on installation. The wage gap is small, as BLS puts field construction at $34/hour against $33/hour in the factory, though overtime and site premiums lift the effective field electrician to roughly $63/hour. The real lever here is higher factory throughput, dropping a full field hour of scope.

Second is the effect from shorter build time. Locking factory cost earlier and compressing the field period trims escalation, contingency, change orders and site general conditions.

Against those savings, modular carries some penalties, with double margin being the largest. A stick build marks up installed content once, through the general contractor and its subs. A modular build adds a separate module-vendor margin on top of the site integrator’s. That layer compresses when the OEM also integrates on site, as Vertiv and Schneider do on their turnkey programs. Schneider’s WP163 shows this dynamic, putting module hardware about 40% above traditional and netting to 13% lower first cost, after design and install labor savings are counted.

Two smaller penalties follow, including the module premium (chassis, bracing, extra interconnects, transport and craning), and from the vendor perspective, the additional factory burden.

The Quality and Predictability Advantage

Factory first-pass quality, the share of test and inspection points cleared without rework, is supposed to exceed 95% in modular solutions, against a field baseline of 60-70%, thanks to the on-factory fixed work instructions. The benefit is system predictability. Flex frames this as a design choice and engineers it through design failure-mode-and-effects analysis for productized configurations, design-for-manufacturing and design-for-assembly reviews for project-specific ones.

That said, recent conversations point to the opposite side, with operators and MEP contractors claiming that modular solutions have proven reliability issues and haven’t lived up to their claims. This eventually means not only losing all the time savings gained initially, with a field team having to go on site, but more importantly, putting the precious hardware at risk.

Re-evaluating Vendors’ Claims

Our ~36% time savings, or around 8 months, lands just above Flex’s published 30%+ floor on whole projects. In order to do a fair comparison, we must consider that vendors’ numbers usually come from narrower scopes or composite definitions, not the end-to-end construction timings. Vertiv’s 85% SmartRun claim applies to overhead busway and containment, while MegaMod’s 50% claim measures module deployment against on-site build. Schneider’s 60% claim is for power-and-cooling modules.

What Operators and Developers Are Modularizing Today

Across the market, operators and developers are not converging on one modular strategy.

Hyperscalers

Hyperscalers most frequently build their own fleet, vertically integrate buildouts, and engineer the datacenter design around their actual workload. That’s why hyperscalers tend to show the innovator traits and have leading experimental footprint.

AWS

AWS shows the fastest scaling buildout currently, adding almost 3.9 GW of capacity to the end of 2025. Project Houdini is their largest internal attempt to modularization model. Rather than designing a modular building from scratch, AWS took its standard data-hall map and re-cut it into a factory-built skid roughly 45-feet in length and weighing about 2000 pounds that is transportable on double-drop trailers. This puts Houdini squarely in the “factor-built white-space” category.

The approach cuts deployment from as much as 15 weeks to roughly 2–3 weeks and can eliminate more than 50,000 on-site electrician hours per module. The skids are built in Houston, Salt Lake City, and Topeka, with early deployments in Texas and South Bend, targeting ~25 weeks from construction start to the first server room.

Houdini is also notable for its approval model. Instead of relying only on site-level engineering sign-off, the factory process requires separate validation of the design and the physical build. Cupertino Electric serves as the main partner, responsible for the latter.

Meta

Meta has recently focused on how to get the building enclosed. Its “tents” datacenters, or rapid-deployment structures, are the clearest example of shell-level modularization in the taxonomy above. A lightweight structural frame supports a tensioned fabric membrane, creating a weather-protected enclosure much faster than a conventional steel-and-concrete building.

At Prometheus in New Albany, Meta is using this strategy to pull shell construction forward

Meta has built six rapid deployment structure at this campus, with each roughly 125,000 in square feet dimension. To put speed savings into the picture, it’s worth noting that the site’s first five permanent buildings took Meta two to three years to complete, while the tents took a fraction of that, with eight standing by April 2026 from our satellite tracking since their announcement of the tent buildout in July of 2025.

This does not mean Meta built the complete datacenters in nine months, because the tent accelerate the shell but not the full facility. Also, the trade off on the other hand, is resilience. A fabric structure does not provide the same long-term durability or weather protection as a permeant steel or concrete shell as we had described above, it is also not built with intent to provide decade-long protection service.

GPU Clouds and Neoclouds

Crusoe

Crusoe participates in both the shell modularization and whole facility modularization taxonomy. At the Abilene Stargate campus, Crusoe accelerated the shell using structural steel and factory-made insulated metal panels. Working with Digital Building Components, each building used roughly 672 prefabricated panels. The panels were manufactured in under 40 days and installed at a rate of roughly 15 to 20 per day, allowing the building to reach a dried-in state in under eight weeks.

In addition, Crusoe’s decided to vertically integrate the manufacturing process. Its 2022 acquisition of Easter-Owens brought modular datacenter and electrical-system manufacturing in-house, giving Crusoe greater control over design, supply chain, and production. The company is now expanding that capability through a dedicated Spark factory in Brighton, Colorado.

Each Spark unit are roughly one megawatt in scale and arrive substantially complete, placing Spark in the whole-facility modularization category.

The Redwood Materials deployment in Nevada shows how this model can scale. Crusoe initially installed four Spark units alongside a 12 MW microgrid and later announced an expansion to 24 units.

Hut 8

Hut 8 is buying the entire infrastructure stack. A former bitcoin miner with power and land positions, it is racing to convert those positions into leasable AI capacity, and the fastest way to do that is to buy a finished modular stack.

At its Beacon Point campus in Corpus Christi it runs Vertiv’s OneCore to commercialize a 704MW IT lease.

The project is being designed around Nvidia’s DSX reference architecture and delivered through a group of established counterparties. American Electric Power provides the utility relationship, Jacobs Solutions owns the EPCM scope, and Vertiv supplies the critical power and cooling infrastructure.

Nebius

Nebius takes a lighter approach to modularization by defining the facility around its own compute architecture and modularizing selected parts of the infrastructure such as power and cooling.

At the New Jersey campus, the facility is being built to Nebius’s own design through its partnership with DataOne and is planned as a phased development expandable to 300 MW. The company used precast structural components to accelerate the shell, while pairing the site with Bloom Energy behind the meter fuel cells as the power solution.

At Béthune, France, the company follows the same operating logic through a different construction method. The project reuses the former Bridgestone tire plant, avoiding part of the greenfield shell and permitting process. Azur Datacenter is responsible for the land, utility intake, construction, and physical plant, while Nebius focuses on the GPUs, racks, networking, and software.

Colocation Providers

Compass

Compass is the colo that has run the modular playbook longest. They industrialize the whole facility into a repeatable kit of parts, not just one subsystem. We estimate roughly 70 - 85% of each building is manufactured off-site and bolted together on site, standing up a building’s framework and roof in 18 to 21 days. The kit spans the full stack:

• Shell: a rebar-free, fiber-reinforced precast shell, Compass casts itself from on-site batch plants

• White space: Standardized prefabricated rack and containment systems such as using Schneider’s EcoStruxure Pod

• Power block: a repeatable ~1.25 MW Schneider power center (Galaxy VX UPS, lithium-ion batteries, QED-2 switchgear)

• MV switchgear: modular medium-voltage switchgear skids under a Siemens deal for up to 1,500 units over five years

It runs this same kit across campuses, up to 360 MW at Red Oak, Texas and eight buildings across 1.8 million square feet at Goodyear, Arizona.

QTS

QTS’s modular strategy is built around locking the design early, standardizing the interfaces between vendors, and purchasing critical equipment before a specific building needs it. The company maintains roughly 7 million square feet of warehouse capacity in Kansas for long-lead equipment, allowing UPS systems, switchgear, cooling equipment, and other owner-furnished components to be held in inventory rather than ordered after each customer signs.

The approach began with the Freedom Design, which keeps the building shell flexible but repeats the power architecture. Each factory-built pod combines a 1.5 MW UPS and switchgear package with a 2.25 MW generator, and the system scales in 1.5 MW increments. Freedom LC+ expands the same concept into cooling, with an architecture that can support either fully air-cooled or fully liquid-cooled deployments. QTS’s newer rapid-deployment design pushes modularization further into the building itself, organizing capacity around repeatable 60 MW data-hall blocks.

QTS has therefore continued to move more scope off-site: first the power train, then parts of the cooling system, and now portions of the shell and data hall. For the shell, some QTS facilities use tilt-up concrete, including Manassas, while larger campuses such as Cedar Rapids use structural steel.

The value is labor, safety, and schedule certainty. QTS estimates that labor pressure has increased datacenter construction costs by roughly 20–30% per MW. Moving to modular is the company is trend, and they begin by stocking long-lead equipment in advance removes another source of delay, giving QTS greater control over when the final megawatts can be commissioned.

QTS developed subsequently the “Rapids” design, which materially accelerates construction timelines. Two major AI companies have adopted this design at scale.

Aligned Data Centers

Aligned’s modular strategy is centered on adaptability, keeping the core power and cooling architecture standardized while allowing the hall configuration to change as customer requirements and rack densities evolve. The approach began with core MEP infrastructure, including a 2 MW UPS container with integrated distribution switchgear and power. This approach is now extending into site conveyance through examples like prefabricated chilled-water assemblies. For customers able to coordinate early, the scope can extend even further into the white space through things like secondary fluid piping.

The other core element is Aligned’s Adaptive Modular Infrastructure platform, which keeps the underlying power and cooling systems consistent while allowing the hall to shift between air, hybrid, and liquid cooling as rack densities evolve. The platform combines

• Delta³, Aligned’s air-cooling system, supporting densities of up to roughly 50 kW per rack.

• DeltaFlow, its liquid-cooling platform, supporting densities above 350 kW per rack

Because the underlying chilled-water loop and facility interfaces remain consistent, the cooling mix can change without redesigning the entire hall. The components inside the modules remain standardized and do not change. An example is found in Project Caprock in Texas, 540 MW across six buildings and 1.65 million square feet.

Read more

Six months later, in our AMD 2.0 article, we took the non-consensus position of upgrading from 0% chance to a much more meaningful chance at success. We published this opinion when sentiment towards AMD was sitting at rock bottom, and when most of the market thought we were were being far too optimistic towards AMD.

That view was based on our observations that AMD has the leadership that can create change rather than suffer from committee style leadership. Lisa quickly hopped on a calls with us and has since then implemented many of our suggestions. We saw an AMD that had finally recognized the importance of software and had a sense of urgency, moving in the right direction even if the destination was still far off. Since then, the signal has only gotten stronger.

Based on our experience on AMD software stack this year, we update our view again from non-zero percentage chance to now a great chance of success as long as AMD solves the two major risks we outline below. It is important to highlight that just because AMD gains market share, that doesn’t mean that Nvidia will do poorly. The pie is growing rapidly for everyone, and Nvidia will continue to massively grow revenue. AMD poses potential competition to Nvidia on the software front, and Jensen will need to cut bureaucracy and flatten the layers of different required internal stakeholder approvals required for even the simplest of tasks if he wants Nvidia to move faster and defend their lead.

Anthropic has publicly announced that they will deploy 2GW of AMD’s chips, Lisa Su and team have leaned into an agentic oriented engineering culture. Anthropic Head of Compute Tom Brown explained how he used Claude over the weekend with “/goal” to bring-up internal Claude inference stack on AMD hardware as a case in point. We believe that since AMD’s compiler and most of their kernels are open sourced, that they are better positioned for the agentic age besides one major risk that we will discuss later. Three months ago, our Accelerator model noted that Anthropic will be an AMD customer. We also noted this on our socials a week ago.

In 2023, Microsoft dropped AMD after the MI300X due to unreliable Samsung 2023 HBM memory and poor software quality, subsequently skipping both the MI325X and the MI355X. Microsoft has since made an about face and has announced that it will deploy MI455X Helios. We believe that OpenAI will be the main end customer for Azure’s MI455X racks. In a strategy somewhat similar to the Nvidia-Groq deal, AMD is announcing a deal with Cerebras to do PD disagg for ultra-fast interactivity inferencing.

There are two major risks that AMD needs to guard against:

1. Supply chain checks and engineering first principles analysis and understanding show that AMD’s first AI rack scale system, Helios, is going currently going through a slow rack production ramp given that it is not using a cableless tray design, which the Rubin Oberon rack is now adopting. Furthermore, since AMD has a weak SerDes design, up to 85% of its backplane needs to be retimed, requiring over over 550 Broadcom ethernet retimers per rack. Furthermore, it is running into backplane reliability challenges during rack production ramp hell.

2. The chief complaint from most AMD engineers internally is that there is a persistent lack of stable GPU clusters for internal software development teams and a lack of stable GPU clusters for automated testing CI. This is blocking AMD’s rate of progress and it is holding AMD back from harnessing the potential upside of AI coding Agents because each AI Agent requires GPUs as well and also requires a testing tool use loop. We will more explain below on our recommendation section.

If AMD can overcome these challenges, we strongly believe that AMD will be well positioned to do well and take market share. This will be helped along as well by the recent stock option based structure whereby AMD gives Meta and OpenAI close to a 105% equity rebate discount using some clever financial engineering. The full rebate triggers a AMD stock reaches the final level of $600 and once OpenAI/Meta buys enough compute. Helios performance per TCO is so great that AMD that cost per million tokens is practically negative cost when combined with this structure! AMD is practically giving away Helios racks and an 5% extra on top of that to an SF-based nonprofit called OpenAI.

In the first part of our article, we go through deep dive into the Helios architecture and the MI455X (gfx1250) instruction set, which is an clone of Hopper SM90’s ISA. We will also discuss the Helios scale-out and scale-up networking architecture. The second part of the article we will further into details of the software stack. For the third part, we will focus on the economics of owning and operating the Helios rack, and also break down the economies of OpenAI/Anthropic/Meta’s equity based rebate discounts and how this structure affects total cost of ownership (TCO).

We are a daily user of ROCm software on MI300X, MI325X, MI355X and are also the #1 bug reporter consistently every quarter! There are a few 10x engineers we want to shout out to that have been working 997 to improve AMD software and have been quickly triaging our bug reports. Many thanks to Hongxia, Chun Fang, HaiShaw, Thomas Wang, Andy Luo, Seungrok, Bill He, Teresa Shan, Parth, Duyi Wang, Gilbert, and many more. Most of AMD’s best 10x engineers are in Shanghai. AMD’s MoRI collective and UMBP KVCache offloading team, AMD’s disaggregated application forward deployed engineering team, and other AMD teams that understand how to do first principles-based inference engineering are all mostly based in Shanghai. A lot of the ROCm software stack’s most important pieces are built in China.

Recommendations to Lisa Su, Mark Papermaster, Sharon Zhou, Anush Elangovan and Vamsi Boppana

There has been good progress over the past year on automated testing to improve software quality, but the pace of progress has not been as aggressive as needed and there continues to lack of sense of urgency with respect to providing enough GPU clusters for automated testing CI and for internal software development teams. It is always too little too late.

Every time we meet with y’all (Vamsi, Anush) every couple of months and occasionally meet with Lisa, we always highlight that CI could be better and indeed we can point to specific examples where CI is heading in correct direction, but the overall strategy needs to be aggressively ramped up. For example, the Kubernetes Inferencing Pollara NIC CI still sits at 0% parity with Nvidia’s ConnectX Nightly CI. Kubernetes is the layer that most inference deployments across the world use. The issue is not with AMD engineering not wanting to add support for it, rather they are being blocked by lack of investment in internal CI capacity. The planned ETA of AMD reaching parity on this front by Advancing AI 2026 was missed due to cluster issues.

On the vLLM side, vLLM gating automated test progress has massively regressed due to AMD cluster infra stability issues this week. AMD’s hardcore engineers were making good progress on vLLM gating over the past couple of weeks to reach the goal of attaining at least 90% parity with CUDA on gating by Advancing AI 2026 until AMD leadership started pulling clusters away from AMD’s internal vLLM team to deploy elsewhere given AMD’s internal capacity crunch that overly relies on backstopped temporary clusters.

Gating/blocking tests mean the tests are of the highest quality because PRs cannot merge unless the test is passing. This prevents bugs from being merged. While AMD’s leadership may distract non-technical folks by showing their non-gating pass rate, gating parity and gating pass rate are the metrics that truly matter.

We hope that AMD’s leadership can re-prioritize giving their internal vLLM team stable clusters and update their philosophy on capacity planning to prevent issues like this in the future so that AMD’s internal hardcore vLLM engineers focus on the work of reaching 90%+ parity with CUDA vLLM on gating and have the tools needed to operate at the same velocity as the AMD internal SGLang team.

The chief complaint from most AMD engineers is that leadership still needs to update their viewpoint towards providing stable CI clusters that don’t just randomly need to be migrated and shifted from one CSP to another CSP.

Moreover, there is a constant lack of GPUs available internally for development. For single node aggregated inferencing, there are enough GPUs to go around internally. But in the age of Distributed Multi-node Inferencing optimizations (wideEP and disaggregated PD), there is nowhere near enough GPUs. Even with the additional 2,000 MI355Xs coming online this month and the 6,000 MI325X/MI355X coming online later this year, total capacity will still not be enough and remains more than an order of magnitude less capacity than the stable long term clusters Nvidia has for internal development. This lack of GPU nodes is a problem that has been getting even worse due to the rise of agentic coding. Previously, each human engineer required a couple of nodes to conduct DI inference software development. But now with agentic coding, each agent requires GPUs to test their code against and each human can have dozens of agents running at the same time and each agent in turn can only dozens of sub agents running at the same time too. Thus, without Distributed Inferencing simulation tools like DynoSim, there will be an even greater shortage of internal GPU capacity internally.

Furthermore, unlike Rubin (SM107), which uses a very similar ISA to Blackwell (SM100), MI455 (gfx1250) is a completely different ISA from MI355 (gfx950) and has completely different codepaths and kernels tha will require testing on both gfx950 and gfx1250. This is another factor that will continue to stress capacity. The big audacious goal was to have MI455X open-source vLLM, SGLang nightly automated CI by Advancing AI 2026, but this timeline was missed. We hear that the new timeline has been delayed untill October 2026 but they are still trying to left-shift it back to an August/September 2026 timeline.

We view this slow ramp of internal software development GPU clusters and automated testing of CI clusters as one of the major risks slowing the pace of improvement of software quality and performance and we hope that AMD leadership revisits their capacity planning strategy going forward.

Part 1: AMD MI455 Silicon & Helios Rack

AMD has done it again on silicon engineering. The MI455X is the most advanced chip that comes out of a fab on the silicon front. AMD is the first company to ship 2nm datacenter silicon with both the MI455’s compute tiles as well as the Venice CPU being early adopters of N2. All of the competing accelerator platforms are on N3.

AMD continues to lead with respect to on package integration as well. The MI455 package is the largest CoWoS-L module shipping at 5.5x reticle size. AMD is still the only company adopting TSMC’s SoIC-X hybrid bonding, which allows AMD to scale silicon footprint in the z dimension as well as the x and y dimensions. All this comes together to bring the MI455 to a total of 3,470mm2 of logic silicon in the package, by far the most amount of silicon that is being shipped in a single package.

The package layout borrows heavily from the MI355X. 8 N2 ‘XCDs’ are hybrid bonded atop of 2 reticle-sized base dies that contain SRAM, HBM controllers, as well as the compute fabric, which allows for the XCDs to communicate with each other.

One of the major areas where the MI455X departs from the MI300 family is the addition of two separate I/O dies that house all the PHYs for off-package communications. While the floorplan representation below shows one I/O die for the UALoE scale up fabric, and the other I/O die for the link to the host CPU and NICs for the scale out fabric, the I/O dies are in fact identical. This is a result of the clever implementation of flexible I/O, with each I/O die supporting 72 lanes that are compatible with different protocols at different line speeds including: 212G UALoE, 64G Infinity Fabric, PCIe Gen 6, 128G UALink, xGMI4. With many of these protocols contributed to the open UALink consortium in one form or another, the I/O is designed from the ground up to integrate well with UALink based interconnect solutions.

This silicon spam is what allows AMD to deliver industry-leading peak theoretical dense FLOPs with the MI455X delivering 20PF of FP8 vs Rubin at 17.5 FP8. However, absolute numbers aside, these performance advantages are relatively mild compared to Rubin given how much more silicon content there is. This is down to AMD’s deficit to Nvidia in GPU microarchitecture design, which we will discuss below. Currently, MI455X is lacking the 3 bit LUT tensor cores that Rubin SM107 has. 3 bit LUT tensor cores have potential advantages for reducing relative HBM bandwidth needs. If anything, AMD is forced to be aggressive on silicon to compensate for this deficiency, not to mention its relative weaknesses in software and system design, though its software is getting better. Alternatively, AMD’s product marketing team could find a few more tricks to juice marketed performance numbers.

The large package size also allows AMD to fit 12 stacks of HBM4 for a total of 432GB per package. This is again industry leading compared to Nvidia and Google shipping only 8 stacks adding up to 288GB per package. Each base die is now closer to reticle sized with the HBM-facing edge measuring 32mm to squeeze in 3 cubes per edge, compared to the MI300X AID which was 29mm per edge which isn’t enough to fit 3 cubes.

Along with Nvidia with Rubin, AMD is the only company shipping HBM4 this year. Having 12 stacks also allows the MI455 to win on memory bandwidth at 23.3 TB/s per chip, implying HBM4 pin speeds of 7.6Gbps. This is a slight upgrade from the 19.6TB/s advertised at last year’s Advancing AI. Despite having a 50% wider bus than Rubin, the total bandwidth is barely above Rubin at 22TB/s.

This is because Nvidia aggressively raised its targeted HBM4 pin speeds last year to well above the original JEDEC HBM4 specification. This change was precisely driven by the need to make up for the memory bandwidth deficit to AMD MI455X. At Nvidia’s marketed 22TB/s for Rubin, this is a pin speed of 10.7Gbps: 40% faster than MI455’s HBM4 which means NVIDIA will be forced to use an much higher quality bin than AMD.

As we have mentioned before and covered extensively in our newsletter and Accelerator Model, this change was challenging for memory suppliers to keep up with. Memory suppliers had to rework their HBM4 to deliver the spec Nvidia was targeting. This also pushed out upstream output for Rubin but as of today these issues have been finally resolved. This move proved to be a worthwhile gambit for Nvidia as they could close one of the big shortfalls of Rubin and despite the delays, Vera Rubin will still be the first deliver tokens at scale compared to MI455 Helios. More on that later.

Active LSI

Hidden beneath the surface, we believe that the MI455 is also the known first chip to ship with active Local Silicon Interconnects (“LSI”), which are the bridges that connect the various chiplets within the CoWoS-L assembly. In the brief history of CoWoS-L, the bridges have been passive, containing only wiring as well as capacitors. Active LSIs come with actual circuitry. During TSMC’s aLSI presentation at ISSCC 2026, TSMC demonstrated active bridges with a low-power repeater circuit that regenerates signals mid-channel. The benefit is that because the bridge now shares the burden of maintaining signal integrity, the PHYs on the top dies can shrink meaningfully, at a nearly negligible energy cost, reclaiming leading-edge silicon and shoreline for compute and memory. The give away that this is being shipped in the MI455, and as such it is effectively the “test vehicle”. What TSMC showed is the MI455 interposer: two base dies, 12 HBM4 stacks, and two IO dies.

MI455X Networking

On the network side, MI455X marks the first AMD GPU deployment with switched scale up and scale-up in a rack-scale domain. This a huge upgrade over the 8-GPU point-to-point mesh used from MI300X through MI355X.

AMD’s Helios rack connects 72 MI455X GPUs through 12 102.4T Tomahawk 6 switches in a single tier all-to-all network. Each GPU has 72 lanes of 200G UALoE for the scale up fabric, resulting in total scale-up bandwidth of 1.8 TB/s uni-di per GPU. Each switch uses 432 of its 512 200G lanes, with the over-provisioning a result of AMD using merchant TH6 switches from Broadcom, rather than a proprietary co-designed switch like Nvidia. Scale-out moves in parallel from 400G Pollara to 800G Vulcano, with two NICs delivering 1.6 Tbit/s per GPU with the option of adding a third NIC for each GPU. MI500 is expected to extend the scale-up domain to 256 GPUs across three racks, where copper will likely give way to co-packaged copper or co-packaged/near-packaged optics.

Meta Recsys Infra Strategy an Odd Choice that is Hurting AMD’s chance of success At Meta

3^rd August 2026 Edit: We have updated the configuration for the Meta Custom MI450X SKU based on new information

In short, AMD has worked hard to deliver a chip and system that beats or matches Vera Rubin on major specifications. Oddly, Meta, one of AMD’s largest GPU customers decided to not take advantage of all of AMD’s innovation here. Most of Meta’s orders for the MI455 is a customized variant that is a cut-down version of the full MI455X. This custom Meta MI455X is a cut down version of the full MI450X with half the I/O and only 6 out of the full 8 compute dies/ XCDs. The HBM4 is also downgraded to 8-Hi HBM vs 12-Hi in the standard SKU.

With only one I/O die instead of 2, there are only 72 I/O lanes available instead of 144 with 2 I/O die. With a reduced lane budget, the scale up bandwidth needs to be reduced. We believe it is 36 lanes for scaleup instead of the full 72; this would mean only half the Tomahawk 6 switches will be needed in a rack for Meta’s version. This leaves another 36 lanes for system and scale out: with 16 lanes of UAL 128G to go to the NIC for 1.6T scale out per GPU, as well as 16 lanes of Infinity Fabric for the connection to the host CPU.

As a result of these choices, Meta’s version of MI450X has significantly compromised network capability with only half the scale up bandwidth per chip, leaving it at 900GB/s vs Rubin’s 1,800GB/s. There is also the compute reduction of 25% compared to the regular version. While the reduction in compute is not huge, the cost savings from reducing 2 compute tiles are miniscule in the context of the whole system so it raises the question whether this choice is worth it.

This chip configuration is intended for Recsys workloads and was something that Recsys infrastructure teams decided. However, this decision was made before TBD Lab was formed or could have its say. Given the significant network and compute deficiency for this chip and system versus Rubin, TBD has no interest in this system, nor is it attractive for external customers.

As we specifically stated on our last piece on Meta infra strategy, the decision to go with a cut down custom MI455 is going to nuke AMD’s volume at Meta because TBD will vastly prefer Rubin if the half MI455 design is chosen. AMD needs to step in, put on their big boy pants, and work directly with teams at TBD to make sure they get the normal MI455 instead of the gimped Meta custom version, which is terrible for LLM training and inferencing. The normal MI455 will be competitive with Nvidia’s Vera Rubin.

We do think there is hope though, as after our article, Mark Zuckerberg has already began rapidly exploring changes to their infrastructure strategy org culture.

Helios Rack Explainer Revisit

It has been a year since AMD’s Helios architecture was announced at Advancing AI 2025, and a lot has happened since then. Let’s revisit the Helios architecture and review some of the changes and nuances that have come up over the last year.

Rack Elevation Diagram

The Helios rack features 18 compute trays and 6 scale up switch trays that sit in the middle of the compute trays. Each compute tray will house 4 MI455X GPUs and 1 Venice CPU, adding up to 72 GPUs and 18 CPUs in aggregate. Each scale up switch tray will house 2 102.4T Tomahawk 6 switch from Broadcom adding up to 12 switch ASICs in aggregate. For Meta’s version of MI455x, there will be six 51.2T Minipack Ethernet switches within the rack, and the power supply shelf will be in a separate IT rack.

Compute Tray Layout

The compute tray design has not largely changed since Advancing AI 2025. Each MI455X GPU sits in an EAM module connecting to the scale up links via backplane connectors and the Venice CPU and Vulcano NIC via flyover cables. Notably, the LPDDR5x directly attached on the EAM is not seen - we will explain more on this below. The Venice CPU sits in its own module board with 16 slots of RDIMM and 5 NVMe SSD slots attached. The design is modular, though the use of flyover cables to connect between modules could lead to challenges in production.

Partial Co-design

AMD has presented a wide portfolio of products for their rack scale solutions, including MI455x GPU, Venice CPU, Pensando Vulcano 800G NIC, and Pensando Salina DPU. However, the scale up switch, which enables the true rack scale scale up performance is missing. Unlike Nvidia, AMD has to rely on a partner for an off the shelf merchant scale up switch. This enables a wider ecosystem of scale up switches that are available to their customers, at least in theory. However, this also puts their roadmap at risk of execution of switch partners. Because Broadcom is the only merchant provider shipping a 100T switch with 200G SerDes, Broadcom is AMD’s only choice for the switch. This also complicates logistics and responsibility between vendors, which requires back and forth communication and problem solving together. Due to the lack of a scale up switch product, AMD cannot reach extreme co-design for the complete rack solution.

Memory Despec

One notable omission is there is no longer any direct attached LPDDR running off of each accelerator. Previous roadmaps had up to 1TB of LPDDR as second tier memory for each MI455X attached on the EAM module, however this has now disappeared. We believe this is another consequence of tight memory supply.

Backplane retimed

Another feature of Helios is the addition of Ethernet re-timers on the scale up switch. This is unlike Nvidia’s Oberon backplane, which is completely passive. AMD’s 200G SerDes struggles from the loss on the copper path between the MI455X chip and the scale-up Tomahawk 6 switch. This could be a result of their flexible I/O ambition or due to simply inferior SerDes quality leading to inadequate signal performance in the backplane. For Meta’s MI455X deployment, ~85% of the scale up links will be retimed. The ethernet retimer will be provided by Broadcom and will be placed in the scale up switch tray. This is not an ideal set up as the retimers will add extra cost and power budget to the entire system. Also, it complicates server assembly as it will be a very tedious task tuning all the retimes when bringing up a rack.

Manufacturability and Efficiency Challenge

The MI455X Helios design was AMD’s response to Nvidia’s GB200/GB300 Oberon architecture. AMD referenced a lot of Nvidia’s architecture points for Helios. While the design enables high density single rack scale up, AMD has also borrowed the flyover cable design that made Nvidia suffer. We have discussed the manufacturing challenges regarding the flyover cables in our Vera Rubin architecture deep dive article. The difficulties with flyover cables with compute trays and switch trays for the GB200/GB300 made Nvidia pivot toward the cableless design for Vera Rubin NVL72. By the time this was announced, it was too late for AMD to implement this design philosophy for MI455x Helios.

In the compute tray, the Genesis cable from Molex handles the 128G UALink between the MI455X and Pensando Vulcano NIC. There will be up to 12 Genesis cables per compute tray crammed into the 1U space alongside liquid cooling tubes and power cables. On the scale up switch side, flyover cables are between the backplane connector and the TH6 switch. 1,728 cables will be routed from the backplane connectors to the 16 ports (32 ports per tray) around each TH6 ASIC. All the cables become potential points of failure during assembly and will make manufacturing inefficient.

Compute Tray Topology

Each of the 18 compute tray in the Helios rack holds four MI455X EAMs, each of which support 36 UALoE links, which in turn are built up using two lanes of 200G Ethernet. 72 lanes of 200G ethernet sums up to 14.4Tbit/s uni-di of total scale-up bandwidth per GPU. The MI455X features two N3P based I/O blocks on the North and South of the package – one I/O block hosts these UALoE links for the scale-up network, while the other implements Infinity Fabric for connecting to CPUs as well as UALink128/PCIe Gen7 for connecting to NICs.

Scale Up Topology

On the other end of these UALoE links are 12 Broadcom Tomahawk 6 Ethernet switches across 6 switch trays with two switch ASICs per tray. UALink over Ethernet (UALoE) is the equivalent layer of the stack of Nvidia’s NVLink. Each Tomahawk 6 switch ASIC is built to support 102.4Tbit/s uni-di of aggregate bandwidth across 512 lanes of 200Gbit/s uni-di each. However, only 432 of these 200G lanes are actively used, for a total of 86.4Tbit/s uni-di of bandwidth going from each switch ASIC to the 72 GPUs in the rack. A switch aggregate bandwidth that divides evenly among 72 GPUs would be ideal – 115.2Tbit/s for instance, but AMD is using the only switch that is available today. UALink switches with 115.2T and 57.6T aggregate bandwidth are on the horizon, but not close enough for them to anchor this architecture. In contrast, Nvidia designed the 28.8T NVSwitch to match the needs of a 72 GPU rack, and this divides bandwidth evenly among 72 GPUs, 400Gbit/s uni-di each, with no wastage of bandwidth at all. Each switch tray also includes a small host x86 CPU.

AMD uses a scale-up topology that should by now be familiar to our readers. In this flat one-layer network, each GPU connects to each switch using 6 lanes of 200Gbit/s uni-di per lane, for a total of 1.2Tbit/s uni-di to each switch of the 12 switch ASICs. Each switch ASIC in turn connects to all GPUs in the rack.

The scale-up switch to GPU links is implemented using a copper cable backplane. For each GPU, each of the 72x 200G lanes is carried over two differential pairs (DPs) of copper channels, one for transmit and one for receive. This totals to 144 DPs of copper cables per GPU, or 10,368 differential pairs of copper cables for the entire rack. Each GPU will also need a 144DP male and a 144DP female connector to interface between the backplane cables and the compute tray itself. On the switch side, four banks of connectors will be used per switch tray, with each bank hosting four 108 DP connectors supporting each – a total of 432 DP per bank.

Flyover cables are used to connect the switch ASICs to the connectors on the back of the compute tray. Flyover cables offer better signal integrity than PCB traces, but the disadvantage is that they tend to limit serviceability and thermal efficiency, and add manufacturing challenges on top. Nvidia has flip flopped between the use of flyover cables and PCB – initially aiming to use flyover cables before changing to use PCB traces for better serviceability and airflow.

The total backplane and compute tray content per rack will add up to $68,928, with $44,352 coming from the backplane and the remaining $24,576 attributable to the flyover cables.

With just six switch trays, the UALoE signal travels a shorter distance compared to NVLink in the GB300 NVL72, which uses nine switch trays. In addition, AMD has designed the Helios rack with 9 compute trays on the upper portion and 9 in the lower portion – this allows signals to travel equal distances in both directions.

Scale-out Networking

For scale-out, each MI455X can connect to as many as three AMD Pensando Vulcano 800 AI NICs delivering 2.4 Tbit/s of scale-out bandwidth, with each NIC attached through an x8 UALink128 interface delivering 256 GB/s of bidirectional GPU-to-NIC bandwidth. A separate Pensando Salina 400 DPU handles the front-end network.

We expect the dominant deployment configuration to carry two Vulcano NICs per GPU for 1.6 Tbit/s of scale-out bandwidth.

Outside the rack, Vulcano supports a multi-plane leaf-spine topology that is non-blocking within each plane which is now a common implementation. We have written extensively on how multi-plane architectures are crucial in AI clusters today as it can fit more GPUs on a two-layer network. We elaborated more on the intuition and advantages behind multi-plane fabrics in our Vera Rubin article here.

In a 131k MI455X 8-plane cluster, each 800G Vulcano NIC breaks into four 200G links, giving every GPU one connection into each of the eight independent planes given each GPU is attached to two NICs. Each plane has 512 leaf and 256 spine switches, for 6,144 TH6 switches across the cluster. The configuration requires two 800G DR4 modules at the NICs and three 1.6T DR8 modules per GPU – two for the leaf-side uplink and downlink and one spine-side. We expect the scale-out networking content per MI455X GPU on an eight-plane two-layer configuration to be ~$8,000 per GPU.

AMD’s Pensando Vulcano 800 AI NIC is going on the same direction the industry has went for the past couple of years now. In our “Meta’s Infrastructure Team Needs a Culture Reset” newsletter, we highlighted the issues faced on the data-plane and control-plane level in AI scale-out networks.

In large AI training clusters, there are three main issues: elephant flow, low entropy and suboptimal fabric utilization.

• Elephant Flow: AI workloads tend to have long-duration, heavy-traffic flows that can congest the network and reduce the overall performance of the training batch.

• Low Entropy: Depending on the training job, the number of IP flows can be limited and congest only a few links while the overall fabric still has plenty of capacity.

• Suboptimal Fabric Utilization: Finally, as an overall effect of both elephant flows and low entropy, there is a large skew in the bandwidth utilization of fabric links, which drives how much the fabric needs to be overprovisioned to run smoothly.

RDMA traffic is highly sensitive to packet loss and congestion, and RoCEv2 is built on Ethernet, a lossy protocol, and UDP, which does not allow for recovery mechanism. Meta’s DSF addressed this inside the network with virtual output queues, cell spraying, credit scheduling and deep-buffer Jericho-AI switches but at the cost of a much more complicated data plane, control plane and hardware stack.

Vulcano tries to solve the same problems but through the NIC. Here, the NIC comes with 3 elements, Intelligent Packet Spray, Path Aware Congestion Control, and Out-of-order Packet Handling and In-order Message Delivery, that allows to offload the network complexity from the fabric to the NIC.

Intelligent Packet Spray attacks the low entropy problem at the core. In traditional ECMP fabrics, a flow is hashed onto a single path and stays there. Since AI traffic is composed of a handful of massive, synchronized elephant flows, hash collisions are frequent and some paths get congested while others sit idle. Instead of pinning a flow onto a single path, AMD’s AI NIC sprays the packets of the same flow onto the available paths of the fabric, raising the entropy level to a per-packet level. The result is a 1:1 fabric utilization, and a failure recovery in milliseconds because no link failure can take down the whole flow.

However, spraying packets creates a new set of problems, namely where the two other elements come into play. First, packets coming from different paths can arrive in the wrong order. Traditional RoCEv2 networks force the packets to go back N retransmissions, which involves buffering at the NIC or switch level depending on the approach (scheduled vs non-scheduled fabrics). Here, the NIC handles the out-of-order packet arrival natively and enforces ordering at the message level, removing the need for buffering. Packets are written directly onto the GPU memory as they land, and only lost packets are retransmitted.

Second, spraying packets blindly across paths causes congestion instead of avoiding it. Path Aware Congestion Control closes the gap as the NIC tracks the real time status of each path and shifts traffic away from congested ones before queues builds up. This is aimed specifically at incast, as collective operations like all-reduce tend to have many senders for one receiver, causing head-of-line blocking cascades through the fabric. Because decisions are made at the NIC level, with per-path visibility, ramp up is fast enough to handle the bursty nature of AI workloads. Also, tail latency is minimized here since packets are sprayed intelligently across the fabric.

Taken together, those three mechanisms solve the same problem Nvidia solves with its ConnectX NIC coupled with Spectrum-X adaptive routing. The main difference here is that AMD is adopting the open UEC standard and a multi-vendor fabric rather than a vertically integrated one.

AMD’s AI NIC does not impose the transport model and allows for flexible scale-out networking. It supports switch-based packet-spray, for operators whose fabric already handles load balancing, NIC-based packet-spray, for dumb commodity fabric where the intelligence is offloaded to the NIC, and NIC-based source routing, for operators who want path control from the endpoint. The whole point is that the network behavior adapts to the fabric you already own and the workload you intend to run, not the other way around.

CDNA5 Microarchitecture

Designs Converging with NVIDIA

In the same way that AMD has drawn inspiration from Nvidia’s boring keynote format, AMD’s CDNA 5 in many ways draws huge inspiration from Nvida’s Hopper (SM 90) architecture. First, CDNA 5 reduces the number of threads per wave to 32, matching it with Nvidia’s 32 threads per warp. CDNA 5 also replaces CDNA3/CDNA4 Infinity Cache and small L2 cache with a single large 96 MB L2 cache per FCD (Fabric and Cache Die). This design converges closer to Nvidia’s “global memory -> L2 cache -> shared memory” memory hierarchy. Managing memory latencies across different hierarchies has always been a pain point of AMD kernel writers; we expect simplifying the hierarchy should mitigate this issue.

Increased Staging Memory and MMA Shapes

CDNA 5 has larger memory staging buffers than its Nvidia counterparts. It has 320 KB of LDS (roughly SMEM-equivalent) and 32 KB of VGPR (roughly thread register-equivalent), so each thread has access to 1024 registers. In addition, we have only seen CDNA 5 supporting mostly 16x16xK shapes like its predecessor. Combining these facts, we theorize that the larger staging buffer is for adapting to the increase in wave count: Under the same number of parallel threads, smaller wave size would lead to a larger number of waves. Since the MMA shape didn’t increase, and each thread has access to 4x number of registers, AMD doesn’t have to resort to increasing MMA scope to warp groups like Nvidia does.

Tensor Data Mover

Tensor Data Mover (TDM) is almost identical to Nvidia’s Tensor Memory Accelerator (TMA). TDM moves data from HBM to LDS without register staging. It supports 5-dimensional tiling, out-of-bound checking, and even multi-cast into different work group clusters, the equivalent of Nvidia’s thread block cluster. One difference is that TDM descriptors are loaded from SGPR, unlike Nvidia loading from host to shared memory. Nvidia’s Rubin has just showcased inline TMA descriptor updates to improve user ergonomics, so we look forward to seeing whether AMD will get the design right.

GFX1250 Speaks NVFP4 Natively

The two FP4 formats now competing for inference share the same E2M1 element: one sign bit, two exponent bits, one mantissa bit. But they differ in how they carry the block scale that makes so narrow a type usable. MXFP4, the OCP standard AMD has built its four-bit path around, pairs each 32-element block with a power-of-two E8M0 scale. NVFP4, the format Nvidia introduced with Blackwell, uses a finer 16-element block, an FP8 E4M3 scale per block, and an FP32 per-tensor global scale on top, resulting in a costlier but more accurate arrangement that is fast becoming the default for FP4-quantized checkpoints.

The notable thing about AMD’s gfx1250, the architecture target behind MI455X, is that its matrix engine speaks NVFP4 natively. MLIR-level scale-format enum in ROCm WMMAMatrixScaleFormat, includes e8, e5m3, and e4m3 members.

More concretely, AMD has already shipped a gfx1250-compiled NVFP4 GEMM code object inside AITER and wired it into the runtime dispatch table, where a format discriminator distinguishes NVFP4 from MXFP4 and the dispatch logic selects the NVFP4 assembly kernel on gfx1250. This is a gfx1250-specific capability, not a CDNA4 one. MI355X / gfx950 does four-bit only as MXFP4, through scaled-MFMA with an E8M0 scale over 32-element blocks — no scale-format field, no 16-element blocks, no E4M3 — which is all AMD’s public matrix-core documentation describes.

We also found that CDNA 5 additionally supports unsigned E5M3 (UE5M3) scaling factor format, in contrast to Nvidia Rubin supporting E4M3 and E5M2. UE5M3 repurposes the sign bit, increasing the dynamic range and dropping the minimum non-zero representable absolute value from E4M3’s 2^-9 to 2^-17. UE5M3 is proposed as an alternative to NVFP4’s additional per-tensor FP32 scaling, and future adoption of those formats will show their efficacy.

Conservative Microarchitectural Changes

The microarchitecture evolution from CDNA 4 to 5 shows that AMD believes less in scaling than Nvidia does. Nvidia has bet on models requiring larger multiplications and scaled MMA shapes aggressively every generation, scaling to MMA shapes that need 2 SMs to execute in Blackwell. Since CDNA 5 barely scales MMA shapes, we doubt we will see an equivalent feature in CDNA 5. We also have yet to see innovations on data compression techniques from AMD like Rubin’s 3-bit lookup table weight compression MMA mode.

Part 2: Will AMD Software Defeat the CUDA Moat?

AMD Software Is Improving Quickly, But the Moat Has Moved

AMD's software story is not "ROCm is broken" anymore. It is that ROCm is finally moving with real urgency, but the competitive frontier has moved faster. We said in April 2025 that AMD was "on the right track" after a sharper developer-first push, and later said AMD's software quality had "massively improved" since January 2025.

What Has Actually Improved

CI Is Maturing, But Not Fast Enough

We praised AMD for moving stable ROCm support into upstream vLLM releases in January 2026 and adding nightlies shortly afterward. CI has since made concrete but incomplete progress. A June change added AMD mirrors and gates for eight major test groups: V1 attention, engine, OpenAI API correctness, small-model evaluation, multimodal pooling, and three speculative-decoding paths. A separate July patch cleaned up flaky jobs and prepared existing mirrors to become merge-blocking again. But CUDA parity has not yet been demonstrated: public regression dashboards, AITER accuracy gates, end-to-end disaggregation CI, and automatic performance gating remain roadmap items.

Encouragingly, AMD’s distributed-inference work has begun moving from one-off recipes into upstream CI. SGLang merged a two-node MI355X 1P1D disaggregation nightly in June for DeepSeek-V4 Flash and Pro, in both FP8 and FP4, then added DP-attention, EP8, MTP, and Kimi K2.6 coverage in early July. This is the important shift: disaggregated inference is starting to become a continuously tested upstream feature rather than a demo.

Kubernetes powers most inference deployments across the world, and AMD is a founding partner of llm-d, an open-source distributed inference Kubernetes orchestration engine. But it has yet to have sufficient CI automated testing for the first party Pollara NIC. It isn’t due to engineering not wanting to add it, but they still to this day under-invest in internal CI capacity planning. This results in 0% parity to Nvidia’s ConnectX-7 NIC on llm-d Kubernetes inference nightly testing. The planned ETA of reaching parity by Advancing AI 2026 was missed.

On the vLLM side, gating automated test progress has massively regressed due to AMD cluster infra stability issues this week. AMD’s hardcore engineers were making good progress on vLLM gating over the past couple of weeks to reach the ETA of advancing AI to reach 90% parity with CUDA on gating, until AMD leadership started pulling clusters away from AMD’s internal vLLM team.

Gating/blocking tests uphold the important standards, since PRs cannot merge unless the test is passing. While AMD leadership may distract non-technical folks by showing their non-gating pass rate, gating parity & gating pass rate are what truly matter.

We hope that AMD leadership (Anush, Vamsi, Mark Papermaster) can re-prioritize giving their internal vLLM team stable clusters, so that AMD’s internal hardcore vLLM engineers can do the work of reaching 90%+ parity with CUDA vLLM on gating and have the tools needed to operate at the same velocity as the AMD internal SGLang team.

Single-Node Performance and Reproducibility Are Real Wins

In March, we highlighted an up to 18× improvement in Kimi K2.5 1T MXFP4 interactivity in under 30 days from AITER/vLLM fixes that were already upstreamed into vLLM 0.18. AMD’s own February 2026 technical write-up, “Speed is the Moat: Inference Performance on AMD GPUs”, argues that AITER-driven single-node optimizations deliver a roughly 1.08x–1.2x throughput uplift over baseline framework configurations. That is the correct direction: the baseline open-source frameworks must get faster on AMD, not just AMD-only demos.

AMD’s MiniMax M3 performance has also caught up to B200, with optimizations via their ATOM stack.

Where the story has improved the most is in recipes, documentation, and reproducibility. ROCm now has a much deeper public recipe layer than it did a year ago. The ROCm inference docs expose a full AI-inference section spanning vLLM, SGLang, distributed MoRI, Mooncake, and deployment guides. The vLLM optimization guide covers AITER, attention-backend selection, TP/EP/DP strategies, FP8/FP4 quantization, and single-node to multi-node scaling. The ROCm/MAD repo now publishes blueprints spanning vLLM, SGLang, training stacks, large-EP microbenchmarks, and disaggregated prefill/decode recipes.

The Posture Is Right

This is why we are mostly aligned with Anush on the software diagnosis. The developer-first posture, upstream alignment, day-0 model enablement, and faster release cadence are the right playbook. We praised Anush’s developer-relations push in 2025 and, more recently, credited his team for moving ROCm from a second vLLM fork toward something closer to a first-class upstream experience. The official vLLM blog now says the era of “just porting” AMD support is over, documents seven ROCm attention backends, and shows 1.2x–4.4x throughput gains from the latest AMD/vLLM orchestration work. That is what a credible catch-up path looks like.

George Hotz put it well back in 2025:

AMD’s dysfunction is different. From the beginning they had leadership that can do things (Lisa Su replied to my first e-mail), they just didn’t see the value in investing in software until recently. They sort of had a point if they were only targeting hyperscalers. but it seems like SemiAnalysis got through to them that hyperscalers aren’t going to deal with bad software either. It remains to be seen if they can shift culture to actually deliver good software, but there’s movement in that direction, and if they succeed AMD is so undervalued. Their hardware is good.

-George Hotz

InferenceX: AMD Dev Velocity is Getting Better

In the past, we have seen AMD struggle in efficiently bringing up new models on their hardware in both the aggregated and disaggregated scenarios. We track all iterative performance on InferenceX which allows us to estimate this “velocity” to some extent.

As mentioned in our previous article, the AMD inference team did a great job at rapidly improving its DeepSeek v4 performance over the first month or so, especially in the single node case. Below is a chart labelling all improvements made in the first 47 days of the DeepSeek v4 release to MI355X SGLang single-node config.

More recently, we saw a similar story for MiniMax M3, where the AMD inference team rapidly iterated to achieve competitive performance.

For the disaggregated case, AMD is also doing quite well. This is a huge improvement from ~6 months ago, where the team struggled to achieve parity with Nvidia on DeepSeek R1 for many months. This is a result of a couple of things. First, AMD has made significant concrete improvements to their software stack with MoRI backend for distributed inference, as well as various improvements to Mooncake. More importantly, the AMD distributed inference team, led by Hai Xiao, has been working with a greater sense of urgency to provide better support for distributed inference.

The following video shows that MI355X FP4 disaggregated solution lagged many months behind Nvidia’s.

Note that this was AMD’s first public disagg recipe, posted on InferenceX in January. Take this in stark contrast with the Day 0 progress on MiniMax M3 FP4 disagg shown below, and you can see AMD is in a much better position now to be competitive in terms of distributed inference solutions.

However, going fast isn’t easy. Especially when the competition has a head start. The missing one piece is now real: what AMD’s distributed stack was missing above is now addressed by the engine that comes with ATOMesh. It is a ROCm-native distributed-inference gateway featuring a Rust routing and orchestration layer that handles the cluster-level work disaggregated serving needs, such as prefill/decode-disaggregated routing, cache-aware load balancing, RDMA KV-cache transfer (via MoRI-IO or Mooncake).

It is not a from-scratch build: ATOMesh is derived from SGLang’s sgl-model-gateway, and is substantially reworked around ATOM and AMD hardware. However, it works, as shown in the MiniMax M3 InferenceX results above. Routing policy stays separate from model execution, and it plugs into vLLM and SGLang as naturally as AMD’s own ATOM engine.

Chipping Away at the CUDA Moat: Using Agents to Enable Day 0 Support

The SemiAnalysis team grinded to enable day 0 support for DeepSeek v4 and MiniMax M3, and we will continue to do so for upcoming frontier models such as Kimi K3. This “day 0” perf is an important baseline for showcasing performance over time, which is the north star goal of InferenceX.

Obviously, the 100x cracked engineers at AMD, Nvidia, Inferact, RadixArk, etc are responsible for the hardcore inference engineering work to truly enable day 0 support for these models. However, often times there are bugs to fix or low hanging fruit that the SemiAnalysis team adds in order to enable day 0 sweeps (currently, these are more prevalent with AMD configs). This has become increasingly easy with the rise of capable agents. The flow looks something like this: for a given config (I.e., MiniMax M3 vLLM MI355X FP8), start a new Claude Code/Codex agent to pull the day 0 recipe from the internet, create the necessary plumbing in InferenceX, and then kick off the sweeps. The agent has access to both the GitHub Action as well as direct access to the physical runner to continuously monitor the status. In the case of an engine error, the agent can automatically decipher what the root cause is, and either iterate automatically and re-run or consult the human for intervention. We can execute this pipeline in parallel for multiple configs on various SKUs.

When an error is identified, the current models are quite good at identifying the cause in the upstream engine code (vLLM/SGLang/TRT) and few-shot implementing a fix, with some guidance from the team. Additionally, we use agents to identify low-hanging fruit performance improvements. Some examples of upstream contributions by the SemiAnalysis team + agents:

• TRT: [fix] Fix fused MHC for DeepSeek-V4-Pro hidden size#13710: a Day 0 DeepSeek v4 fused MHC kernel fix

• [Bugfix] Fix NixlConnector handshake block_len validation for GQA-replicated KV heads#45879: MiniMax M3 disagg Day 0 enablement

• [Bug Fix] [MiniMax-M3] Implement EAGLE3 support on the AMD MiniMax M3#45546: enable speculative decoding for AMD MiniMax M3 Day 0

• [Bugfix][ROCm] Fix MiniMax-M3 FP8 KV cache dtype: enable FP8 KV cache support for MI300X and MI325X for Day 0 MiniMax M3

• And Much More...

A shoutout to vLLM community Roger Wang, Hongxia, Michael Goin and other Inferact engineers for the kind mentorship and help on getting these fixes merged.

This kind of rapid iteration would simply not have been possible 3-6 months ago on a team of 2.5 engineers. Current frontier models in a capable harness are legit. They can make meaningful contributions to OSS serving engines and kernels. This may not necessarily be attributed to the “intelligence” of the models, rather the pure “grit” they have when told to achieve a goal. This, combined with the ability to execute many of these tasks in parallel, makes it clear that testing and writing software is not the moat it was last year. This is broadly positive for AMD. By enabling AI agents to take on work previously performed by human engineers, the relevance of the so-called “CUDA moat” is diminished, and this helps AMD offset Nvidia’s advantage in engineering headcount.

AMD seems to be leaning into this thesis heavily, as at Advancing AI 2026, they announced ROCm.ai, a suite of skills, harnesses, and framework integrations that aims to further enable developers to iterate quickly on kernels and performance tuning using agents.

GEAK, Hyperloom, and Optimizing Against InferenceX

ROCm.ai is more than just a keynote slide. Most pieces are already sitting in the open on AMD’s AMD-AGI org, and they map almost one-to-one onto the day-0 loop we described above. At the center is GEAK (”Generating Efficient AI-Centric Kernels”), a mini-SWE-agent-based agent that writes and tunes Triton/HIP and FlyDSL kernels, wrapped by Hyperloom, the orchestrator that profiles a serving workload, finds the bottleneck kernels, throws GEAK and GEMM-tuning agents at them, and gates every candidate on an end-to-end A/B before it counts. Around that sit the supporting cast: Magpie for evaluation, TraceLens for trace analysis, Apex for exporting agent trajectories into an RL-flavored training pipeline, and AgentKernelArena, a head-to-head harness that pits Claude Code, Codex, Cursor, and GEAK against each other on the same kernel tasks under identical scoring.

Hyperloom’s optimizer pulls its performance target straight from InferenceX. On the current main branch, it scraps InferenceX, and target_analyzer writes a competitor target that the agent then tries to clear. An earlier version of their CI went further — computing a % gain over InferenceX, counting how many models “beat” InferenceX, and posting the scoreboard to a Teams/Slack channel — before that machinery was trimmed for the open-source release.

We see this as a good thing. Our vision for InferenceX is to accurately track performance of open-source frameworks, and it definitely serves this role if agents use it as a reward signal and continuously try to improve. These efforts will benefit the ML community with better model serving performance.

The more interesting lesson buried in these repos is one we recognize from our own day-0 grind: generating a kernel is the easy part, trusting the number is not. A meaningful chunk of the engineering is anti-cheat. GEAK had to stop silently scoring the unpatched baseline kernel instead of the agent’s actual patch, and add a GEAK_PROTECT_TEST_FILES mode that strips the agent’s edits to the test harness so it can’t fake correctness by rewriting the reference. Apex ships a tamper detector that flags hardcoded print(”PASS”) and a banned-library list so an agent can’t “optimize” a Triton kernel by quietly routing to a pre-tuned MIOpen or hipBLASLt call. It’s the same grit-over-intelligence story: the models will reward-hack a benchmark the instant you let them, and a surprising share of the work is building the guardrails that keep the speedups real.

And it works. GEAK’s learned notes already log verified end-to-end wins on shipping silicon. ~+21.8% e2e from an MXFP8 decode-bound dense-linear rewrite on MI355X, with honest caveats where grouped-MoE GEMM stalls out around a 1.1x ceiling. None of this defeats the CUDA moat on its own. But it is the clearest signal yet that AMD is industrializing the exact workflow that, a year ago, required a room full of CUDA engineers.

Introducing AgentX, a new InferenceX Agentic Scenario

While the current 8k1k + 1k1k scenarios are a great proxy for evaluating baseline chip performance, they fail to evaluate the entire system (routers, KV cache transfer, KV cache offloading, schedulers, etc) since they are all single-turn, random data requests.

For the past few months, we have been working with industry leaders to develop an agentic benchmark: AgentX. To achieve our goal of making the benchmark as reflective as possible of real world serving, we collected ~3 months of internal SemiAnalysis Claude Code, Codex, etc. traces, which we then replay offline at varying concurrencies using AIPerf. Some of the statistics of the dataset are shown below. Notably, the median ISL and OSL are 140k and 396, respectively (quite different from 8k1k). With a median cache hit rate of 99.2%! Note, this is assuming an infinite cache so is not, of course, actually attainable in most real-world serving.

The dataset also includes realistic subagent usage as well as dynamic workflows, which further stresses the KV cache by continuously introducing uncached context for the server to process. The tool use time / user think time (captured by inter-turn latency) is also reflected in the dataset, further stressing the KV cache and highlighting the efficacy of KV offloading techniques, which have the possibility of increasing KV cache TTL.

We are very excited about this scenario. We have worked with WEKA, Inferact, RadixArk, LMCache, Mooncake, NVIDIA, AMD, and other industry leaders in ensuring it accurately reflects real traffic that may be experienced by a TaaS provider or frontier lab like OAI/Ant.

Results have been rolling in for the past month and are available on InferenceX. We will write an in-depth article on results and methodology soon.

Single-Node is Out, Disaggregated is In

As discussed, the competitive frontier has moved from single node aggregated to multi node disaggregated inference. In our InferenceX v2 article, we discussed the notion of AMD’s “software composability problem” when it came to distributed inference. That is, they had made strides in individual optimizations such as disagg, FP4, WideEP, DP-attention, etc, but combining many of these together broke the stack. While there is still certainly progress to be made, AMD has made significant strides in this area.

Our GTC 2026 recap frames Nvidia’s next moat as disaggregated inference systems, including attention/feed-forward disaggregation (AFD). Since attention is stateful and KV-bound, while FFN is stateless and batch-scalable, these phases can be split and mapped onto the best-suited hardware. Then, inference leadership becomes a distributed-systems problem rather than a single-kernel one. WideEP, disaggregated prefill, KV transport, expert routing, and scheduler/orchestration are all part of that same new moat. AMD can close single-node gaps, but unless those optimizations compose cleanly in vLLM and SGLang, it is still fighting yesterday’s war.

The timeline below puts the gap in perspective. The open CUDA ecosystem has been shipping disagg+WideEP since early 2024, while AMD’s first publicly available PD-disagg + WideEP recipes only landed in January 2026 via InferenceX. AFD’s real payoff requires superpod-class interconnects, which are the kind of rack-scale hardware AMD does not ship until MI455X. And at GTC 2026, Nvidia announced the Groq 3 LPX, an SRAM-based LPU integrated into the Nvidia inference rack specifically for disaggregated decode FFN.

The New Moat: Sparse Models, Disaggregation, and WideEP

Recent Model Trends: Active Experts Stay Tiny While Total Experts Explode

The model landscape makes the composability problem more urgent. Active expert counts stay pinned in the 4–10 range while total experts scale out to 128, 256, 384, and now 512. Many of the newest giant MoEs also collapse the KV cache to the equivalent of one or two heads via MLA-style low-rank designs. That means the frontier is getting sparser, more bandwidth-sensitive, and more dependent on routing, KV movement, WideEP, and distributed scheduling. In that world, one more single-node benchmark win is not the moat. Composable distributed inference is.

Disaggregated Prefill and WideEP

Disaggregated prefill separates compute-heavy prefill from memory-heavy decode onto different nodes, removing the interference between the two phases and letting operators tune prefill vs. decode node counts independently. The trade-off is KV-cache transfer over RDMA. We estimate roughly 290 MB for an 8,192-token DeepSeek-R1 FP8-KV prefill.

Currently on InferenceX, AMD’s disaggregated inference configs often perform either worse or only slightly better than their single node counterparts. For DeepSeek-R1, iso-interactivity throughput for MoRI SGL outperforms the aggregated SGL config by 2×–3×.

WideEP spreads experts across more GPUs and nodes instead of replicating full expert islands. On DeepSeek-R1 (256 routed experts), moving from EP8 on a single 8-GPU node to EP64 across 64 GPUs cuts experts per GPU from 32 to 4, freeing HBM for KV cache, enabling larger concurrent batches (and thus more tokens per expert per GEMM), and scaling aggregate HBM bandwidth with cluster size. The payoff is not subtle: NVIDIA reports up to 2.28× higher per-GPU output throughput from Wide-EP (DeepSeek-V3.2, NVFP4, GB200 NVL72, EP16/EP32 vs EP4/EP8). As we emphasized in InferenceX v2, frontier labs and most TaaS providers are already deploying disaggregated serving with WideEP in production.

Put differently, ATOM and AITER are useful only insofar as they improve the mainstream ecosystem. We have argued that ATOM may help single-node performance but still lacks production features such as NVMe or CPU KV-cache offload, tool parsing, and WideEP. The right strategy is not to build an AMD-only island. It is to upstream the kernels, harden CI, publish recipes, and make FP4 + WideEP + disagg + cache offload work together in the default open-source stacks. That is how the CUDA moat gets eroded now.

Disaggregated Inference Accuracy

Before March 2026, DeepSeekR1 disaggregated configs with DP-attention(DPA) failed accuracy evals with near 0 GSM8K scores, while no-DPA sweeps passed GSM8K at >95%. This was caught with InferenceX eval runs. This shows that no-DPA disaggregated path can preserve correctness, but once DPA is used, the stack breaks, showing AMD’s composition problem again. This issue has, however, been fixed.

One issue that has not been fixed is EP at certain batch sizes in SGLang with the MoRI backend. On DeepSeek-R1, decode with concurrency 64 drops GSM8K to ~80% against a ~94% baseline that holds at every other concurrency. The catastrophic version of this bug, where the same low-concurrency path silently produced fluent-but-wrong output and scored 0 on GSM8K, was traced to an uninitialized reduce buffer in AITER’s FP4 MoE kernel and patched in June; the residual ~80% cliff at concurrency 64 is still open, with AMD attributing it to numerical corner cases in the quantization kernels that only that batch size exercises and declining to prioritize a fix.

Composition Works… Sometimes

AMD has now demonstrated that several of these optimizations can compose: SGLang’s MI355X nightly images cover DeepSeek-V4 disaggregation with DP-attention, EP8, and MTP, while four-node DeepSeek-V4 and Kimi WideEP16 configurations have been validated on hardware. But composition remains model- and topology-specific rather than dependable by default. Kimi’s DP8/EP8 path was excluded after a GPU memory fault, its WideEP16 int4 decode path currently must disable HIP graph capture, and the WideEP16 CI changes remain under review. The gap is no longer that the individual pieces never work together. It is that AMD cannot yet assume they will work together across models, quantization formats, parallelism strategies, speculative decoding, and network topologies without special cases.

Disaggregated Inference Has to Become Everyone’s Job

Disaggregated inference cannot sit in a small specialist corner of AMD. AMD’s own ROCm tutorial says single-node optimization is starting to hit limits, and that distributed inference is becoming more important; ROCm followed that with official Mooncake PD-disaggregation docs in December 2025 and MoRI-based distributed inference docs in January 2026. Once the official stack is publishing xP+yD topologies, RDMA requirements, 1P2D recipes, and KV-transfer frameworks, DI is not a side quest. DI is the product. It must become everyone’s job across ROCm, SGLang integration, CI, recipes, and performance engineering.

That is the AMD software story so far. The single-node path is getting fixed. MoRI is promising. The overlap stack is finally showing up. But the market is already on to SBO, PD disaggregation, load balancing, and wider EP configurations that must compose cleanly under real production constraints. AMD no longer needs one more hero optimization. It needs the entire distributed inference stack to start moving as one system.

Where AMD Stands: The Distributed Stack, Piece by Piece

MoRI: A Real Win, Built by One Tiger Team 🐯

MoRI, to AMD’s credit, is a real win. MoRI is a modular RDMA framework with two dedicated components: MoRI-EP for expert dispatch/combine and MoRI-IO for KV transfer. ROCm has now published official MoRI-based distributed SGLang documentation for MI355X clusters. It is built from first principles by a China-based engineering team. We support the direction, but it needs much more open CI and testing.

AMD's own MoRI roadmap makes the scale problem concrete. The H2 2026 roadmap lays out an ambitious program across three subsystems: a tiered distributed KV cache (HBM→DRAM→NVMe via SPDK/GDS) with scheduler co-design, a next-generation SHMEM v2, EP v2 kernels in FlyDSL/C++ with elastic expert-parallelism and fault tolerance, "mega kernel" codesign with FlyDSL, and rack-level Helios enablement. Topped off with SGLang and vLLM upstreaming. The whole list is owned by the same five or six engineers whose handles dominate MoRI's merged pull requests. It is the right roadmap. It is also, as of publication, entirely still ahead of the team — resting on the same handful of shoulders.

The Newest Silicon Shows the Gap: Helios (gfx1250)

The clearest measure of how far the distributed stack still has to go is AMD’s newest hardware. Across every layer of the Helios (gfx1250 / MI455X) stack the same shape repeats: single-model arch-enablement and KV-transfer plumbing are landing fast, while WideEP, validation, and the wave32-tuned high-value kernels are not there yet.

Start at the foundation. PyTorch’s own gfx1250 support merged in mid-July (ROCm/pytorch #3421, a cherry-pick of an upstream change that landed, was reverted a day later, and re-landed a week after), but it is pure build-time arch plumbing gated behind an unreleased ROCm 7.14+, so it is dormant on every shipping wheel. It ships with no gfx1250 CI, deferring a test runner to a follow-up that does not exist. It is a skeleton: the highest-value paths (Composable Kernel GEMM and SDPA, FP8 grouped GEMM, int4) are filtered out or hard-error, attention is a “Tech Preview,” and wave32 kernels are unwritten. AMD’s code even brands the part “CDNA5” while describing a GFX12.5, wave32, WMMA execution model, the clearest in-tree measure of how far Helios sits from the CDNA wave64 lineage its kernels were written for.

One layer up, the raw MoE kernels are the fastest-moving piece of the stack. AMD’s aiter and FlyDSL libraries have absorbed well over a hundred gfx1250 PRs since spring: wave32/WMMA GEMMs, MXFP4/A8W4 quant, MLA-v4 attention, TDM data-movement atoms. Roughly a quarter of them closed without merging. The centerpiece is the AMD-native “MegaMoE” path: a fused MoE expert-parallel flow whose dispatch/combine op-layer was vendored into aiter and whose gfx1250 grouped-GEMM kernel(global→local expert remap, fused route/scatter, and a masked grouped GEMM tuned for DeepSeek-V4’s 384-expert MoE) is still-open, validated for a8w4 (FP8-activation, MXFP4-weight) only. This is the WideEP MoE primitive the stack needs; it just isn’t finished, and the gpt-oss, R1, and V4 kernels each land on their own track.

The break is at the integration layer, where those kernels meet the frameworks. SGLang’s July 22 gfx1250 nightly only builds and publishes the image. No test job, no accuracy gate, built on a generic runner and merely mirrored through an MI300, because no gfx1250 runner exists upstream. That image builds MoRI but pins a June commit predating gfx1250 support, so MoRI-EP WideEP does not run: the kernels carry no matrix-core or wave64-specific ISA, but MoRI’s build gates only gfx942/gfx950 and the wave32 support AMD has since added lives in a newer expert-parallel path that SGLang’s integration does not yet call. Once AMD wires that into the path SGLang uses, WideEP should work. MoRI-IO’s KV-transfer half is architecture-agnostic host-side RDMA and would move KV cache on gfx1250 today, yet the actual gfx1250 recipes for DeepSeek-V4 and R1 are single-node tensor-parallel with neither WideEP nor disaggregation.

vLLM’s gfx1250 bring-up tells the same story from the other side: four models on the FFM simulator and early silicon with volatile results, dedicated perf work only for gpt-oss, whose “ATOM-parity” tuning pass was itself reverted “until AITER is ready”. MoRI was also removed from the build outright. It is a credible single-node FP4 bring-up; but it is not the distributed stack

Even AMD’s own engine follows the pattern. ATOM’s Helios work is disaggregation-first and expert-parallel-later: ATOM adds DeepSeek-V4 MoRI-IO write-push KV transfer plus a UALink scale-up fabric backend — the Helios-generation interconnect — so the KV-transport half of disaggregation is being built for exactly the model and fabric Helios will run, though its e2e validation ran on eight MI355X nodes and a UALink vPOD rather than MI455x. But it carries no EP dispatch/combine, no MoRI-EP, no WideEP; ATOM’s shipped DeepSeek-V4 recipe still serves the model at TP=8 on a single node, and every one of its MoRI-EP PRs targets last-generation gfx942/gfx950. Bottom to top:

PyTorch, kernels, MoRI, frameworks, ATOM, Helios has arch enablement and the first half of disaggregation, and no WideEP anywhere.

SemiAnalysis Helping AMD upstream into NVIDIA’s KVCache Transfer Library NIXL

At Nvidia’s GTC 2025 NCCL session, we asked if Nvidia would provide support to the AMD communication library fork due to this big refactor in the upcoming Nvidia’s communication library. Nvidia explicitly said it would not help AMD’s communication team adapt to the upcoming NVIDIA library refactor, and that NVIDIA doesn’t participate in AMD’s communication development at all

A year later, at Nvidia’s GTC 2026 Dynamo session, we asked a question: NIXL had already accepted upstream contributions from Trainium’s Neuron fork, so would it accept them from AMD’s RIXL fork too? The maintainers said yes, on the record, in front of a room.

AMD had been carrying RIXL downstream, burning engineering hours maintaining a parallel copy of infrastructure everyone else gets for free. The only thing standing between AMD and upstream was whether anyone would try. NVIDIA had just removed the excuse.

So we spent April making sure AMD knew it. We put the question to Stephen Bates publicly on LinkedIn, offered Anush help to connect AMD directly to the NIXL maintainers, and asked Anush about MoRI/RIXL upstreaming in person during TensorWave’s beyond summit and we helped AMD form the relationship with NVIDIA so that they can upstream the patches.

On May 15, AMD’s Andy Luo opened a PR for ROCm/HIP build support for gfx942 (MI300X, MI325X) and gfx950 (MI350X, MI355X), behind a default-off use_rocm Meson flag. It merged June 4 after several rounds of review with NVIDIA-side maintainers.

Andy’s PR cleared review by proving zero blast radius on the CUDA path. On the AMD side it validated end-to-end VRAM transfers over NIXL+UCX on MI300X and MI355X. The stacked follow-up, #1647, hit 341 Gb/s cross-node RDMA on two MI355X nodes using AMD’s own AINIC RoCE NICs and libionic driver, with no Mellanox NIC in the path. Today, RIXL has been fully ported to NIXL. Dynamo core has since started taking AMD patches too.

NIXL is the transport layer for disaggregated serving, so if AMD wants PD disagg and WideEP to be dependable rather than model-specific, KV transfer has to work on Instinct in the frameworks people actually deploy, not just inside MoRI. Nobody at AMD should have needed us to point that out. The gaps that remain say the same thing: nixlbench HIP support and the ROCm Dockerfile/CI surface landed as separate follow-ups, an AMD GPU CI runner is still an offer rather than a merged workflow, and rocSHMEM is a future plugin.

The Overlap Stack Is Late, and the Market Has Already Moved On

Two-batch overlap (TBO) should have been table stakes a long time ago. SGLang put overlapping two batches and expert parallelism on the 2025 H1 roadmap, and the public docs now expose `--enable-two-batch-overlap` with claims of up to 2× throughput. AMD’s own November 2025 DeepSeek-on-MI300X writeup still described dual-batch overlap as an essential feature that was “still being developed,” with future work centered on dual-batch overlap and expert load balancing. MORI EP support for two-batch overlap only merged into SGLang on February 20, 2026. That is late.

The maturity gap showed up immediately. Days after merging, a maintainer flagged the PR for breaking CI and asked for it to be reverted and reopened after fixes. That is the broader ROCm inference story in miniature: the ingredients are increasingly there, but too many key optimizations still arrive late, land fragile, and need another turn before they become something customers can trust by default.

Part 3: AMD Competitive on Total Cost of Ownership & up to 105% Discount for OpenAI/Meta

Read more

Nvidia has also recently made available their first public release of the Rubin (SM_107) software stack with CUDA13.4 and has upstreamed Rubin PRs to PyTorch, vLLM and OpenAI Triton Compiler. Blackwell was not able to reuse Hopper WGMMA kernels, Rubin is able to reuse Blackwell's kernels, which makes the software bring up process much smoother. For speed of light (SOL) performance, engineers will still need to tune and rewrite kernels but for those that are focused on time to market, Blackwell kernels can be reused. We will also explain Rubin’s new 3 bit programmable LUT tensor core.

NVIDIA has also released on GitHub that Feynman is SM_140. Unlike Blackwell to Rubin, Rubin to Feynman will be an much more complex transition on the kernel front.

The early metrics gathered on VR NVL72 come from CoreWeave. We have not independently verified them. Nvidia has committed to submitting verifiable numbers to InferenceX by Q3 CY2026. Google should submit TPUv7 results in the next couple of months, and AMD has committed to MI455X UALoE72. Once those land, the ecosystem gets an objective comparison across systems.

In this article we break down Nvidia's Rubin claims against several baselines, showing where Rubin clearly leads Blackwell and where the lead is thinner. We will also analyze Rubin’s performance per total cost of ownership using our already existing estimates for Rubin’s total cost of ownership (TCO). The TCO for Rubin and many other systems is sourced from our AI TCO model, which tracks the total cost of ownership of different AI chips, factoring in capex, opex and different other expenses. We also consider performance per watt using our All-in Utility Provisioned Power Estimates from our Datacenter Model.

Finally, we will present a component by component build up of the Bill of Materials (BoM) for the VR NVL72. This is available in our upcoming SemiAnalysis Bill of Materials (BoM) Model.

Another area where Rubin Oberon NVL72 will fare better than Blackwell Oberon NVL72 is in a much faster production ramp period. This is thanks to Rubin’s simpler cableless compute tray design and learnings from Nvidia’s experience with deploying a rack-scale copper backplane, having invested much effort into ironing out issues with Blackwell’s copper backplane. Our Accelerator Model tracks quarter by quarter shipments of Rubin at both the package level and the rack level.

A Brief Breakdown of Rubin Chip-Level Microarchitecture Features

Going through a complete breakdown of Rubin microarchitecture will have to wait until we obtain ssh access to Rubin systems, allowing us to run benchmarks similar to those we carried out when first analyzing Blackwell. However, there are still a few interesting points we can still make.

We expect that Rubin bringup will be much more seamless compared to the transition from Hopper to Blackwell, where engineers expended much effort just to port kernels to Blackwell. This simplicity comes from the fact that Rubin is able to run Blackwell SM100-family kernels across all the important kernel libraries in DeepGEMM, FlashMLA, CUTLASS, among others. Moving from Hopper to Blackwell meant rewriting kernels from scratch. Hopper’s kernels don’t run on Blackwell at all.

Reusing Blackwell SM100 kernels means a clear time to market advantage, but for speed of light (SOL) performance, engineers will still need to tune and rewrite kernels specifically for the Rubin architecture, though kernel reuse buys them time to focus much more on this kernel tuning.

Turning to architectural details, Rubin’s SMEM increased to 328 KiB compared to Blackwell’s 228 KiB. While the default SMEM capacity is 228 KiB, Rubin comes with an oversized shared memory mode that allows increase to 328 KiB. Furthermore, TMEM has been increased to 256 KiB up from 228 KiB in Blackwell as the number of columns increased from 512 to 576. The additional columns will allow stashing block scale factors, while keeping the TMEM region for accumulators disjointed from it. This greatly simplifies block-scaled kernel logic: it saves the kernel writers from carefully pipelining and overlapping MMA matrix and block scaling factor loads.

Rubin's TMA now supports inline descriptor updates. There are tons of use cases for this. For example, in an MoE layer, each expert is a separate weight matrix at its own address in HBM, so the TMA descriptor has to point somewhere new on every expert switch. On Blackwell, that meant rewriting the descriptor in memory and synchronizing before the next load. Now, with Rubin, the per-expert offset is passed inline to the TMA instruction so that one descriptor covers all experts, with no in-memory rewrite between them. This removes overhead during token dispatch and improves decode speed at low batch sizes.

Inline TMA descriptor update corresponds to the ISA feature .override qualifiers. TMA instructions require a tensorMap object that specifies layout and format metadata. By using the .override qualifier, bulk asynchronous copy instructions can reuse the tensorMap object as a template, but replacing certain metadata fields, such as strides. In the case of MoE, expert weights have identical shapes, data types, and properties. By overriding the global address, kernel writers can avoid duplicating or replacing tensorMap objects when loading different experts.

Rubin doubles BF16/FP16 exponential throughput per clock per SM again, which helps overlap Tensor Core work with softmax during attention. FP32 throughput is unchanged from Blackwell Ultra.

Compared to Blackwell NVFP4/MXFP4 Tensor Cores which could only accept UE4M3/UE8M0 block scale factor format, On Rubin Tensor cores, it will now be able to accept UE5M3 8 bit block scale factor too. This additional block scale format will allow for more flexibility and less quantization error is certain cases due to its wider range.

It is also important to note that latency has improved for SM-driven NVLink communications through the use of counted writes, which reduces the number of back and forth messages required to send data between GPUs over the copper backplane. This is a huge deal because Blackwell NVLink latency is multiple times higher than that of TPU and Trainium.

Rubin’s Tensor core delivers twice the throughput for FP8 and FP4 as compared to Blackwell. A key change driving this is doubling the k dimension as this means in theory, a GEMM takes half the number of clock cycles to execute. Additionally, the awkward K=96/3xFP4 instructions from Blackwell Ultra are still present, but alongside a new K=128 variant.

In Blackwell, PDL allowed for overlap at the grid level, where the dependent grid needs to wait for all threadblocks in the previous kernel to complete before starting. This allowed for some overlap and hiding of the ramp-down and ramp-up time between kernels, but didn’t come close to the extremely fine-grained overlap that trendy megakernel authors look to achieve. In Rubin, finer-grained overlap is enabled, where the dependent kernel can synchronize with the previous kernel at the threadblock level instead.

Rubin has 2.8x higher global memory bandwidth than Blackwell Ultra through the use of 3D-stacked HBM4 memory. It is unlikely Rubin delivers any improvement versus Blackwell in memory system latency. Our Accelerator & HBM Model has an complete breakdown of the memory volumes estimates & vendor used in Rubin.

Rubin adds 2:4 sparsity support for activations. In every group of four values, two are kept and two are zeroed. The pattern is regular, so the Tensor Core knows where the survivors are, skips the rest, and runs the MMA at twice the rate. A small metadata field tracks which slots were kept.

Nvidia shipped 2:4 on weights back in Ampere and nobody used it, because it meant pruning the model and retraining. Rubin applies it to activations at runtime, so no retraining is needed. In attention, QK^T runs dense, then the scores get compressed on the way out of Tensor Memory. Softmax processes only the survivors, and the following GEMM against V runs sparse. The output stays dense, so nothing else in the model changes. It works on MLP activations too.

Nvidia has published no accuracy data, and throwing out half the attention scores before softmax is not obviously free. CoreWeave’s DeepSeek R1 results do not appear to use it either, which makes it another Rubin feature with silicon today and no tuned kernels behind it yet.

Lookup Table Weight Decompression in Rubin SM107 Tensor Core

Rubin adds LUT B, a Tensor Core MMA mode that decompresses the weight operand from a lookup table. In this mode, the B operand is a compressed matrix of indices. In a standard inference GEMM, the B operand holds the weights. The weight values live in a lookup table in Tensor Memory. The Tensor Core reads each index and reconstructs the weight value inside the MMA. There is no separate dequantization pass. After lookup, the multiply runs at FP8.

In LUT B, every weight position stores a 3-bit index rather than a complete numerical value. The index selects one of eight E4M3 values in the lookup table shared by that weight’s 8×64 block. For example, if the stored index is 5, the Tensor Core uses entry 5 from that block’s lookup table as the weight. The lookup happens inside the MMA, so the kernel never has to construct a separate decompressed weight matrix.

The lookup table (LUT) does not have to follow the spacing of a conventional uniform or floating-point grid. A quantization algorithm can therefore place more values around dense clusters of weights, use uneven spacing for long tails, or choose an asymmetric codebook when positive and negative weights have different distributions.

This flexibility creates the possibility of better accuracy per stored bit. Rubin LUT B has fewer individual codes than FP4, but it can place those codes where a particular weight group needs them rather than accepting the fixed ratios of E2M1. It is not automatically more accurate than MXFP4 or NVFP4, however. One codebook is shared across 512 weights, whereas NVFP4 adapts its scale over much smaller groups of 16. The result will depend on the codebook-fitting algorithm, calibration data, quantization-aware training and whether sensitive layers remain at higher precision.

Each index is 3 bits while the lookup table has 8 entries. Each entry is one byte, an E4M3 8-bit float in the reference kernel. This results in 3.125 bits per weight: 3 bits for the index, plus 64 bits of codebook spread across 512 weights. The codebook sits in HBM with the indices, so 3.125 bits per weight is the full stored footprint.

The instruction loads the compressed weights into the collector buffer. The Tensor Core can hold them there and reuse them across a run of activation tiles. This is a weight stationary pattern. The mode also has limits as it does not support transpose of the B matrix.

The block-scaled formats, NVFP4 and MXFP, also decompress inside the MMA. But they apply one uniform scale per block, not a codebook. Software methods like AWQ reach low bit counts by running a separate dequantization step before the matmul, Rubin is the first NVIDIA Tensor Core input format that reconstructs a non-uniform codebook inside the MMA.

A lower bit rate cuts the HBM capacity that the weights need. It also cuts the bytes that the GPU reads for each weight. At low batch size, weight bandwidth limits the decode step. Fewer bytes per weight then raise decode throughput. A non-uniform codebook also holds accuracy better than uniform rounding at the same bit count. This feature should also have an impact on power efficiency, as fewer bits will need to move through the memory system for each flop.

Using Kimi K3 2.8T as an example, at about 4.5 bits per weight, MXFP4 stores 2.8e12 x 4.25 / 8 = about 1,4875 GB, where GB = 1e9 bytes. At 3.125 bits per weight, the Rubin lookup-table format stores 2.8e12 x 3.125 / 8 = about 1,094 GB (about 1.09 TB). The difference is about 393.5 GB. These figures cover the raw weight payload only, and exclude the KV cache, activations, and any parallelism replication. At 288 GB of HBM4 per Rubin package, the weights alone need about 6 packages in NVFP4 and about 4 packages in the new Rubin format.

Feynman Architecture Sneak Peek

From Blackwell (SM100)/Blackwell Ultra to Rubin (SM107), the jump is relatively small in terms of the microarchitecture, so Rubin can be thought of as a Blackwell kicker architecture. In comparison, Feynman (sm_140) is a completely new architecture family. This will require rewriting lots of kernels from Rubin to Feynman, which is similar to what happened from Hopper WGMMA to Blackwell tcgen05. Our Accelerator & HBM Model provides a full breakdown of Feynman quarter by quarter volume estimates.

Feynman’s 3D stacking will be similar to what AMD has been doing with 3D stacking since their MI300X with CDNA3.

One of the new features of the Feynman architecture is that it will contain sparsity aware data movement ops. These can be used in sparse GEMMs to improve performance by avoiding pointless loads, stores and FMAs.

Nuances of CoreWeave VR NVL72 Results

Yesterday, CoreWeave published their benchmarked Vera Rubin NVL72 Inference results expressed in units of performance (tokens/sec) per power used (MW). We will break down the nuances of their data and compare their results against Blackwell’s performance using our own InferenceX July 2026 results as a baseline.

The first notable claim on the CoreWeave-Nvidia chart is that VR NVL72 achieves 10x better token throughput per megawatt than GB200 NVL72 at the iso-interactivity of ~150 tok/s/user. This is about 50% faster than today’s “fast mode” on frontier models.

Three things about their chart. The benchmark is single-turn, 8k in and 1k out. The y-axis is output token throughput per megawatt, not total throughput. And their power number covers both prefill and decode GPUs, even though only output tokens are counted. InferenceX measures output throughput against decode GPU watts only, so we renormalized our data to match theirs for this comparison.

It is important to point out that CoreWeave claims to have enabled all of the following inference optimizations on both their baseline GB200 NVL72 and Rubin NVL72 performance results, including but not limited to:

• NVFP4 Precision

• Speculative Decoding (Using MTP)

• Disaggregated Serving (Using Dynamo)

• Wide Expert Parallelism

• via TensorRT-LLM

The above results seem to suggest that Rubin comes to market with a strong performance gain vs Blackwell out of the gate. However, there are a few nuances that are worth unpacking.

First, attentive readers will note that CoreWeave is comparing Rubin against a GB200 NVL72 2025 baseline. In some ways, comparing performance at the early stages of GB200 NVL72’s lifecycle is fair, since Rubin performance is expected to massively improve from this early stage in its own lifecycle. Our analysis will also use the GB200 NVL72 early performance results from 2025, but we also compare how GB200 NVL72 did by 2026 as well as the most current GPU worth comparing to: GB300 NVL72. We will directly compare GB300 NVL72 performance from early in 2026 with Rubin’s comparable early lifecycle performance.

The second nuance in CoreWeave’s performance results is that they are using DeepSeek R1 671B, a model that is not widely used anymore. One would perhaps wish that CoreWeave used a more modern model like GLM5.2, Kimi K2.5, Qwen3.5, or DeepSeek V4. Even better would be Kimi K3 or Qwen3.8, both of which are coming soon to InferenceX! At least CoreWeave is not using GPTOSS 120B in Summer 2026 like AMD is for MI455X UALoE72 performance metrics. We expect that the fog of war created by benchmarking old models will be cleared up once Nvidia starts benchmarking Rubin on more modern model architectures with InferenceX in Q3 CY2026.

Oddly enough, CoreWeave’s choice of using DeepSeek R1 671B is theoretically more favourable towards the Blackwell baseline, and not Rubin. Rubin’s main advantages lie in a higher HBM capacity, higher CPU DRAM capacity, and greater HBM bandwidth, meaning that Rubin is more optimized for multi-trillion parameter models like Fable 5, Gemini Pro, Kimi K3, and Qwen3.8 2.4T.

The third noteworthy item is that CoreWeave uses only single turn 8k/1k input/output tokens. Theoretically, multi-turn long context workloads like Agentic Coding should do better on Rubin, due to Rubin’s higher HBM capacity and bandwidth, but this would not be captured on a simple single-turn benchmark. Our upcoming AgentX benchmark scenario created in collaboration with Weka, LMCache, the vLLM/SGLang community, Nvidia, AMD, and many others in the community will provide a realistic agentic workload to benchmark inference performance. We encourage everyone to adopt this inference benchmark!

Finally, we note that CoreWeave’s testing was done on a pre-production rack without a scale-out fabric. Specifically, CoreWeave used a Dell Engineering Sample (ES) rack. We do believe these results are valuable as they use wide EP and PD disagg, which uses the NVL72 scale-up backplane and proves that it is working well. This backplane faced many reliability challenges during the ramp of GB200 NVL72 Oberon, as we have noted in our Accelerator model.

Rubin Versus Blackwell Performance per MegaWatt

The metric Nvidia chose to lead with was “output tokens per second per all-in utility megawatt”, counting every GPU in the system. To compare apples-to-apples, we renormalize our own InferenceX benchmark data onto the same total-GPU basis. Below, we put VR NVL72 up against our official GB200 and GB300 July 2026 benchmarks, as well as CoreWeave’s 2025 GB200 baseline.

The eye-catching multiples in Nvidia’s charts all come from the 2025 baseline. When comparing benchmark data, we believe we should use figures from the same time period, so the July 2026 GB200 and GB300 benchmarks are the more useful comparison.

In theory, datacenter PUE can be lower for Vera Rubin since Vera Rubin can operate with 45 degrees Celsius coolant temperatures in custom datacenters without chillers. However, for our comparisons, since most datacenters are designed to accommodate a wide variety of systems, we use the same PUE across the DLC cooled chips.

The following pareto curves plot output throughput per total-GPU megawatt against interactivity. Each line stops where its recipe’s frontier ends.

Next, we provide the same data in table form. When a cell says "impossible," we mean that the interactivity is past that recipe's frontier, simply not allowing the configuration to serve that workload at that speed.

Here is the same frontier as bars, across the 100 to 300 tok/s/user band. All four recipes have data through 250 tok/s/user, and only Rubin and GB300 make it to 300 tok/s/user.

Let’s first compare Rubin against the July 2026 GB300 NVL72 baseline. Rubin’s lead is smallest at low interactivity and continues growing through the middle of the interactivity curve. Rubin sits at near 2x the throughput as Blackwell up through 100 tok/s/user, then widens to roughly 4x around 200 tok/s/user, where the gap peaks. Then, the gap begins to narrow again. The headline 5.4x performance gain over GB300 at 300 tok/s/user isn’t Rubin pulling further ahead. Rather, it is GB300 running the last, barely viable point on its frontier, which causes the ratio to balloon. GB200 can’t reach 300 tok/s/user at all. Blackwell’s per-GPU throughput drops off fast as the batch shrinks at high interactivity, while Rubin is still on a flatter part of its frontier.

Comparing Rubin against the 2025 GB200 NVL72 baseline is different, showing the biggest lead in the middle of the curve. The gap starts at under 3x at low speeds, but increasings to about 10x at 150 tok/s/user (the point Nvidia highlights in their chart), before falling back to 6x at around 200 tok/s/user. The data from that line is accurate, but as we have mentioned it use a software stack that is a year-old, not the GB200 you would run today.

At the very top of the interactivity range, the Blackwell curves drop off. By 350 tok/s/user, neither GB200 nor GB300 can serve the workloads at all, leaving only Rubin with an actual curve, delivering 96,446 tok/s/MW at 300 tok/s/user and 70,703 at 350 tok/s/user.

Clearly, Rubin is going to give us a lot more “fast mode” than Blackwell.

Rubin Versus Blackwell Performance per TCO

Per-megawatt performance only counts performance against power. Cost per million output tokens folds in the hardware’s total cost of ownership (TCO) including IT capital costs as well as electricity and datacenter costs. Our TCO model breaks this down comprehensively, providing capital costs and operating costs across server generations. Here, we divide each SKU's all-in TCO by that same renormalized output throughput, so lower is better. Rubin carries a higher TCO per GPU than Blackwell, $3.57 per GPU-hour against $1.84 for GB200 and $2.36 for GB300 in the operator ownership scenario (not rental prices). The charts and tables below will show how Rubin’s $ per token lead comes out a little smaller than its per-megawatt lead.

As with the per-MW analysis, the 2025 GB200 baseline produces the largest gains in performance for Rubin, but the July 2026 GB200 and GB300 numbers are the more relevant baseline for comparison for anyone buying capacity today.

The following pareto curves plot cost per million output tokens against interactivity. Each line stops where its recipe’s frontier ends.

Next, we provide the same data in table form, with the ratio showing how many times cheaper Rubin is at each interactivity. Again, a cell marked "impossible" is a speed that the recipe's frontier can't reach.

The chart below plots the frontier as bars across the 100 to 300 tok/s/user band. All four recipes have data through 250 tok/s/user, and only Rubin and GB300 reach 300 tok/s/user.

Against July 2026 GB200 & GB300, Rubin is cheaper at every interactivity, and the gap widens as you climb. The gap starts at about 1.5x cheaper than GB200 through 100 tok/s/user, and improves to 3x by 200 tok/s/user through to 250 tok/s/user. The 5x edge over GB300 at 300 tok/s/user is the same as the per-MW view, where GB300 can barely serve tokens and GB200 can’t serve at this interactivity level at all.

The 2025 GB200 NVL72 baseline is once again the more dramatic one, cresting in the middle of the curve. Rubin is a little over 2x cheaper at low speeds, peaks near 8x at 150 tok/s/user, and then moves back to 5x by 200 tok/s/user. Same comment as on the per-MW version: the 2025 GB200 baseline measures a year-old software stack, not the GB200 you would run today.

At the very top of the range, things are the same. GB200 has no operating point past 250 tok/s/user and GB300 has none past 300 tok/s/user, so by 350 tok/s/user only Rubin can serve at all, delivering a cost of $4.18 per million output tokens.

Below the paywall, we will continue breaking down Rubin’s performance as compared to the best-known publicly available MI355X distributed inferencing performance. We also briefly analyze how the Triton Compiler, PyTorch, vLLM, and Dynamo software will function on Rubin.

Read more

We believe that Meta Infrastructure needs a cultural reset to better serve the Meta AI organization, especially the world class researchers who are at MSL. This is even more important as Meta embarks on the path of selling its compute to outside customers, not just serving captive internal users.

Meta Infrastructure has become bloated, with middle managers expending resources on over-engineered technology solutions that lose sight of broader organizational needs. The company appears burdened by far too many disparate groups that are over-optimizing for certain metrics as opposed to delivering usable technology for the company as a whole. Middle managers will do everything to justify their proposals to protect their positions within Meta, which has become an extremely political organization.

One big issue is Meta’s six-month performance review cycle, in which the bottom 10% to 15% are cut every review round. The result is an organization of employees that optimize for short-term wins rather than long-term strategies. Some managers push for highly visible projects that can be delivered quickly, a practice known as “window washing” and then promptly pivot or abandon them. Few openly challenge leadership, which leads to bad decisions going uncorrected. The whole system discourages long-term thinking and leads to risk-adverse behavior.

Within Meta Infrastructure, supply chain teams also have little say over engineering teams. The result is technology decisions driven by political motivations rather than thoughtful software/hardware co-design for the broader company.

Frequent pivots are also common. And because Meta has a reputation for throwing money at problems and executing at high speed, these U-turns end up becoming more costly versus other companies that take a more disciplined or conservative approach. Suppliers also lose faith when given design wins are later cancelled. This has lead to less supply chain prioritization on new designs. Some suppliers favor focusing on Amazon or Google designs due to Meta’s frequent reshuffling.

A lot of Meta’s issues come from a lack of financial discipline, with managers creating new projects and headcount to fill out and justify the massive budgets given for initiatives like AI. It’s a repeat of what happened to Meta’s Reality Labs division, where billions of dollars were spent on engineers and R&D until layoffs, partly brought on by the company’s pivot to AI, slashed the team and their related projects starting in 2022 and continuing through this year.

Rivos Acquisition

First, one of Meta’s most recent missteps has been the more than $2.5 billion it sunk last year into the acquisition of Rivos, a chip startup.

Few inside Meta’s chip division have a full understanding of why the company bought Rivos in the first place, and those who championed the deal internally have since gone quiet. The prevailing theory is that Meta had the money, the custom silicon space was heating up, and it was already licensing Rivos’ IP for a future chip, so leadership figured it might as well own Rivos’ technology outright than let anyone else have it.

With the Rivos acquisition, Meta also gained the ability to bypass partners like Broadcom to manage the manufacturing and testing of their own custom chips, a practice known as customer-owned tooling. However, paying more than $2.5 billion for that privilege doesn’t make much sense either given that a COT team could be built from scratch for maybe $100 million a year or more.

Although Meta wanted only Rivos’ accelerator and GPU team, the startup’s founders insisted on an all-or-nothing deal. Meta bought the entire company and then heavily cut employees in the parts it didn’t want.

Some former Meta chip employees say the acquisition was led by Meta’s silicon chief, Yee Jiun Song, who pushed for the deal against the wishes of some of those under him but has since lost interest. The result is an organization that has rejected the transplant, leaving Rivos staff stuck to fend for themselves. Existing Meta chip managers treated the acquisition as a pool of free headcount, grabbing Rivos engineers to build out their own empires and pulling the original startup apart in different directions until little of it remained intact.

The technology reasons behind the deal have evaporated just as quickly. Part of Meta’s original interest was that Rivos had SIMT core IP which was closer to a Nvidia-style GPU architecture in terms of programmability, whereas Meta’s own MTIA is SIMD. Yet after the deal closed, Meta cancelled the chip known as Olympus that was meant to use Rivos’ GPU IP, because the design was too aggressive in terms of system and package architecture as well as the software being far from ready. Instead, Meta is sticking with its existing chip architecture through MTIA 600.

Meta still wants to develop a chip that incorporates Rivos technology, and it has created a new chip project known as Phoebe that is now scheduled to tape out in 2028. But not everyone inside Meta is optimistic about the project, and some believe it could eventually be cancelled as well. We have been updating our subscribers of the Accelerator Model about all the different architectures, floorplans, and volumes of various custom silicon.

Meanwhile, incoming Rivos employees have gotten a crash course in Meta’s internal culture. Unlike more mature hardware organizations like Apple, Meta’s hardware roadmaps can change just as quickly as its software roadmaps, with constant pivots, meaning that something you are working on now could be pointless in six months. Many within Meta’s chip division say there is no clear org structure, no clear decision-making, and no clear ownership of which group does what.

A funny example of the disfunction that is approachable to those not in the weeds, Meta doesn’t give most employees assigned desks; people who have been there over a year still scrounge daily for a workspace, where sometimes the desk has a monitor, sometimes the monitor has the wrong cable, and sometimes there is no monitor at all. Employees have formally applied for an assigned desk and been denied, even though desks sit empty everywhere.

Finally, the Rivos acquisition has created dissatisfaction among the pre-existing Meta silicon team. Many Rivos engineers joined with higher compensation and titles without having the requisite responsibilities to justify such perks, and they were placed under existing Meta silicon employees. This resulted in decreased morale within the pre-existing team and also with incoming Rivos staff, who lack the power to make decisions. Since the beginning of this year, a number of Meta’s silicon engineers have been leaving for startups or other well established companies like Arm and Nvidia.

As we first reported for clients, around 30% of the Rivos engineers who joined Meta as part of the acquisition were let go in recent layoffs. Rivos co-founder Mark Hayter has already left. A number of former Rivos people have since departed Meta to join Gerard Williams’s new chip startup, Nuvacore, after their first tranche of RSUs vested in May. We believe that Rivos CEO and co-founder, Puneet Kumar, is also eyeing up an exit in a year or two when his shares in Meta fully vest. There is speculation he could join Rosaic Labs, a new chip startup co-founded by Amarjit Gill, an investor in Rivos and longtime collaborator with Puneet Kumar at SiByte, P.A. Semi, Apple and Agnilux.

Grand Teton

Now let’s talk about Meta’s AI server designs. Meta’s hardware decisions are largely driven by a TCO analysis comparing a matrix of workloads such as ranking-recommendation and GenAI against what vendors like Nvidia, AMD and their own custom silicon offering provide. Meta CEO Mark Zuckerberg has issued an edict that the systems being designed have to work for the entire business, meaning they have to support not only Gen AI workloads but core ranking and recommendation systems as well.

However, Meta has still opted for several “optimizations” for its GPU servers, all of which are strange choices that are often worse than the standard configurations that other hyperscalers purchase. Part of the issue stems from the company’s server teams being in completely separate organizations than their networking teams, with different goals and opinions. Meta engineers are also motivated by the idea of owning their own network operating system and hardware rather than rely on other companies.

To recap, Meta’s H100 HGX server was called “Teton Grand” a design that has been contributed to OCP. The major difference from the standard HGX server is the addition of a switch tray in addition to the standard GPU and CPU head-node configuration. This switch tray houses four Broadcom PCIe switches, 16 SSDs and eight NICs. The function is to provide extra PCIe lanes so that the Grand Teton can add more SSDs per server.