Furthermore, because subscription plans are so heavily subsidized, they can also have a large impact on margins and revenue per MW despite only being a small portion of total revenue. Consider the following rough numbers for Anthropic.

Despite being just 10% of overall revenue, subscriptions can take up over 40% of inference compute and lower blended revenue per MW by ~$36M. Subscriptions are even more important for OpenAI, as they make up a larger portion of their total revenue. For specific numbers, see our Tokenomics Model.

In other words, if you want to accurately model AI lab financials, you need to understand their subscription limits.

So how do limits actually work? The way to think about subscriptions is that your monthly payment grants you some numbers of “credits”. Each (model, token type) combo consumes a different amount of credits. Because credit cost ratios can differ dramatically from API price ratios, the “value” of the same plan changes depending on what model and workload you’re running. Put differently, it doesn’t make sense to say a plan is worth $X in isolation—you need to consider the full (plan, model, workload) tuple.

Here’s how the API-equivalent value of the same $200/month Claude plan changes depending on the model and workload.

A single static snapshot isn’t enough either. Labs publicly change their limits all the time with promos and new model releases. They can also silently change limits whenever they want by tweaking credit costs. Ideally, you’d re-check the cost of each (plan, model, token type) daily so you can surface any changes in real-time.

This is exactly what SemiAnalysis has done with our new Subscriptions Dashboard available exclusively to Tokenomics Model subscribers. Besides every OpenAI and Anthropic subscription, we also track Meta, SpaceXAI, Cursor, Cognition, Z.ai, MiniMax, and Moonshot.

New providers, plans, and models will be added as soon as they’re released. The rest of the article will give an overview of everyone’s current limits. All token and dollar amounts below assume agentic usage unless otherwise specified.

Methodology

Tokens are generally priced per MTok (million tokens) across the following types:

• Input: Fresh tokens added to the LLM’s context window that aren’t cached

• Cache write: Input tokens that are cached for multi-turn conversations. Generally slightly more expensive than regular input tokens.

• Cache read: Tokens from previous turns that are already cached in the conversation. Generally extremely cheap compared to uncached tokens.

• Output: Tokens generated by the model. The most expensive token type.

Additionally, some models charge more for tokens above a certain context window. Those models typically compact before that expensive context window is reached, so we also keep our measurements below it.

Subscription plans don’t expose this fine grained pricing. Instead, they provide a simple 0 to 100% usage meter across 5-hour and 7-day windows, and sometimes a separate meter for a model like Fable. Subscription tiers are then differentiated by usage multipliers relative only to the provider’s other plans. OpenAI for example used to advertise “Expanded Codex usage” in their $20 Plus plan, “5x more usage than Plus” in the $100 Pro plan, and “20x more usage” in the $200 Pro plan until they cut their $200 plan usage in half and removed all relative usage from their pricing page.

Setup

We compute the subscription-rate of each (plan, model, token type) triple by running experiments that isolate one token type at a time and watching how far the model provider’s meter moves. An experiment is some number of repeated calls using a specific prompt that maximizes one token type while minimizing all the others.

Input, cache writes, and cache reads share the same prompt template. We use a portion of War and Peace since some models will refuse to respond if given large blocks of gibberish. For the input token experiments, we use a random tag on every call to ensure nothing gets cached. The cache write experiments run the same way but with the prompt marked for caching, so each new tag forces a new cache entry. The cache read experiments use a fixed tag, so the first call writes the cache and every repeat reads it.

For the output token experiments, we used a technical essay to force long outputs because models will refuse mechanical prompts like “repeat SemiAnalysis 100,000 times”.

Computing the Results

For every call we record two things: how many tokens of each type the provider billed, and what the usage meter read. Turning those into a price takes three steps.

1. Measure the rate of each token type

Providers generally report usage on a meter that moves in fixed amounts. This could be a whole percentage point, a credit, a cent etc... A single request often does not move the meter at all, so the cost of one request cannot be measured directly.

Instead, we measure in steps. As requests run, we keep a running total of the tokens used. Each time the meter rises, we store that total. The tokens used between two meter moves make up one step, which is the cost of moving the meter by one unit.

We drop two partial steps. When a run starts, the meter is already part of the way to its next move, so the tokens before the first move are less than a full step and would make the rate look higher than it is. The tokens after the last move never finish a step, so we drop those as well.

To get a rate, we add up how far the meter moved over the complete steps and divide by the tokens used in them. Providers charge differently for input, output, and cached tokens, so we calculate a separate rate for each.

Caveats:

• Because we see the meter only once per request, a single step can be off by up to one request. These errors don’t add up with more consecutive steps, so we track a range instead of a single number, and the range gets narrower the more steps we count. We keep adding steps until the range is within ±5%, then report the rate across all of them.

• No request contains only one type of token. Some plans, for example, require a set of instructions on every request. We subtract that extra amount using the prices we measured for the other types.

• Reading from the cache costs almost nothing on some plans and may not move the meter at all. If the meter does not move after 500M cache read tokens, we assume that cache reads are free.

2. Convert rates into tokens per window and per month

We read every meter a plan shows after every request. This could be a 5-hour limit, a weekly limit, and for models like Fable, its own weekly limit. So each token type gets a price on each meter. This is how much of that limit a million tokens uses. From each price we work out how many tokens fit e.g. if a million tokens use 5% of a limit, the whole limit holds 20 million.

The 5-hour limit resets many times within a week, so a month is capped by the weekly limits.

3. Pricing

Lastly, we compute the API-equivalent value by converting the limit % consumed by a million tokens of each type into a dollar amount. For example, if you consume 1% of a $200 plan’s monthly limit, that’s $2.

Then, we assume a workload shape, and compute how many total tokens you can get from each plan given this ratio. For the agentic workload, we use our own usage ratios in September as reported in our Tokenomics Model.

Finally, we multiply by the blended price per MTok at API prices to get the API-equivalent value.

Catching a Provider A/B Test

While refining our methodology, we ran into a very confusing situation where just 1 of 3 of the same subscription we tested for a particular provider had ~20% lower limits than the other 2. This unlucky account happened to be significantly older as well, and we were worried that the provider had some cursed setup where they changed subscription limits depending on the age of the account.

Fortunately, after reaching out to the provider, we were able to confirm this was not the case and that our unlucky account was part of an “extremely tiny” A/B test they were running on limits. Additionally, they wanted to emphasize that they “didn’t just decrease limits wholesale” and were instead testing “how to better balance when people hit limits”.

This conclusion is interesting for two reasons:

1. It proves providers can silently change subscription limits at any time

2. Our method is sensitive enough to detect these subtle changes!

OpenAI vs Anthropic

The biggest question in AI subscription land is how Anthropic’s limits compare to OpenAI’s. Note that OpenAI just made some massive changes to their subscriptions last week, halving the API-equivalent value for the $200 plan and introducing a new $500 tier. The following numbers describe the current state of OAI vs ANT. Afterwards, we will examine OpenAI’s recent changes in depth.

Limits are quite similar across the board for GPT-6 Astra vs Fable 5.1, though remember that Fable can only be used for 50% of your limit. In other words, your $200 ANT plan would still have 50% left after consuming $2,485 worth of Fable 5.1, whereas the equivalent OAI plan would be fully exhausted after $2,897 of Astra.

This extra 50% starts becoming very relevant when we compare Opus 5.5 vs GPT 6.1 Sol.

At this mid-tier of models—which both companies market as the intended daily driver for most users—Anthropic is an overwhelmingly better deal, offering ~5x the API-equivalent value across the board. You could argue that this is unfair for OAI because 6.1 Sol is much cheaper per token than Opus 5.5, but the gap is still massive even if you switch to comparing the number of tokens.

Overall, we believe API-equivalent value is the single best number for capturing the “value” of a subscription plan, but there are times where raw token volumes can be more appropriate. For example, if a model is a particularly bad (or good) deal at API prices, then the API-equivalent value will of course be misleading.

Token efficiency is also an extremely relevant factor, but the industry unfortunately lacks reliable data here. Many people like to cite this chart from Artificial Analysis, but we do not believe the benchmark tasks in the AA Intelligence Index are at all representative of real work people do with LLMs.

OpenAI’s new $500 plan and massive limit reduction

Our tracking confirmed the 50% value reduction for OpenAI’s $200 plan announced by Tibo. $200 plans purchased before the cut will keep the old higher limits until October 29th, but newly purchased plans immediately start with the lower limits.

As you can see, OpenAI specifically halved the number of tokens you get for each model tier. This caused the API-equivalent value for Sol class models to drop by over 50% because OAI also cut cached input token pricing for 6.1 Sol.

The new $500 plan only offers 21% more Astra than the old $200 plan. Interestingly, the API-equivalent value for Sol class models actually decreased due to the aforementioned 6.1 Sol price cut.

Of course, 300 TPS Ultrafast was the real headline feature for the $500 plan and we are currently testing its limits. Results will be published to Tokenomics Model subscribers as soon as they’re available. We are also actively testing the limits when using your ChatGPT subscription on third party apps like Devin.

The $200/month plan also used to be significantly more subsidized than OpenAI’s other subscriptions. If we normalize each (plan, model) combo to the API-equivalent value and tokens per dollar, we can see that Pro 100 offered ~2x more per-dollar value than Plus on Astra and Pro 200 was another ~2x on top of Pro 100 across all models.

With last week’s cut, Pro 100, 200, and 500 all offer the same tokens per dollar for every model. Plus is also comparable for Sol but offers a relatively worse deal on Astra.

Anthropic, on the other hand, already offered the same per-dollar value for all of their subscription tiers and currently crushes OpenAI.

OpenAI used to be lauded by indie developers for their generous subscription limits compared to Anthropic, but this is simply no longer true today. Opus 5.5 on any Claude subscription offers far better value than anything from OpenAI.

The only counter argument for OpenAI is that none of their Pro plans have a 5 hour limit, which makes it easier to consume a higher % of your total monthly limit. However, we don’t think this offsets the ~4x higher API-equivalent value offered by Opus 5.5 on Claude plans.

Do labs adjust subscription limits based on API pricing changes?

We’ve seen a number of significant price cuts recently to Anthropic and OpenAI models.

• Fable 5.1 cut cache reads by 75% vs Fable 5

• Opus 5.5 cut input and output token pricing by 20% vs Opus 5 and cache reads by 60%.

• GPT 6.1 Sol cut cache reads by 50% vs GPT 6 sol. This was after making GPT 6 Sol 60-67% cheaper vs 5.6 Sol.

One important question is whether the labs adjust token limits after these price cuts, such that the API-equivalent value of your subscription stays constant.

We already showed earlier that OpenAI didn’t change Sol token limits on the $200/month plan when they released 6.1 Sol. This caused the API-equivalent value to drop ~30%.

Anthropic also didn’t increase Fable token limits at all with the release of 5.1, but they did up Opus token limits with the 5.5 price cut.

However, the ~20% token increase for Max plans and ~50% increase for the Pro plan weren’t enough to fully offset the API price cut.

Two different approaches to increasing subscription margins

As we explained at the start of this article, subscription gross margins are way lower than API and meaningfully reduce revenue per MW for both OpenAI and Anthropic. Both companies are well aware of this fact but have chosen two different strategies for reducing subscriber subsidies.

For a given plan, Anthropic decreases the API-equivalent value for more premium models. The drop off is small between Sonnet 5.5 and Opus 5.5, but becomes very meaningful with Fable 5.1.

Reducing the API-equivalent value as you release more powerful models at higher list prices is the more subtle way to stop subsidizing subscriptions. Assuming 100% utilization and 92% API gross margins, maxing out Opus 5.5 vs Fable 5.1 usage corresponds to -369% and 1% gross margins respectively. If we were to use a more realistic average utilization of 20%, then gross margins rise to 6% and 80% respectively.

In other words, Anthropic would likely already have software like margins on their subscription plans if everyone only used Fable. Opus class models will continue to become cheaper to serve over time as Anthropic figures out how to make the model smaller, and new model tiers above Fable will have even lower subscription limits (assuming they’re included at all).

OpenAI, on the other hand, picked the nuclear option of just immediately cutting to Fable-level limits across the board.

The downside of this approach is supposed to be public backlash, but OpenAI has somehow managed to more or less avoid this entirely. It’s likely some combination of all the good DevDay PR drowning out the negative news from the day prior and grandfathering existing plans for an additional month.

Chinese subscription plans are still a good deal

Despite being compute poor, the Chinese labs still offer subsidized subscription plans, but the level of subsidization varies dramatically between models.

Here’s how they compare to OpenAI and Anthropic after normalizing per dollar.

Value per dollar increases as you buy higher tier subscriptions from the Chinese labs. On average, the API-equivalent value per dollar is a little less than the ~12x you get from OpenAI.

Third party plans are worse than first party

The final comparison we’ll consider is OpenAI and Anthropic models on their own first party plans vs third party wrappers like Cursor and Cognition (Devin).

Read more

How does sparse attention affect the TAM of memory, including HBM and NAND? Sparse attention selects top-k most relevant tokens to attend to, reducing the memory consumption and bandwidth requirements during the core Scaled Dot-Product Attention (SDPA) operation. However, the efficiency improvement doesn’t directly translate to overall memory savings in practice. Concretely, the top-k selection operation typically requires the full context to be in HBM, so sparse attention doesn’t eliminate the memory capacity bottleneck.

To overcome this limitation, the SGLang team designed HiSparse, a hierarchical memory system that proactively offloads KV cache entries from device HBM to host DRAM. HiSparse behaves like an LRU (Least Recently Used) cache, where it loads tokens from DRAM to HBM upon top-k selection cache miss, and it offloads tokens from HBM to DRAM based on the LRU eviction policy. To reduce the cache miss latency, HiSparse overlaps KV cache loading of layer N with the execution of layer N-1 (layer-wise overlapping), introduced in HiSparse’s prior work HiCache.

With HiSparse, SGLang greatly boosts throughput at high concurrency and long context scenarios, at the cost of top-k cache miss I/O overhead.

Sparse attention reduces KV cache memory and bandwidth requirements at the SDPA operation, but it does not reduce the overall memory capacity usage. In addition, HiSparse shows that system optimizations can overcome sparse attention’s memory capacity limitations, so sparse attention memory profile alone cannot sufficiently portray the full picture of system KV cache efficiency. To provide a more holistic picture of serving sparse attention models, here we explain the design of Z.ai’s GLM-5 model series, and how they are served in practice.

GLM5.3 Agentic Inference Serving

In our realtime benchmark board InferenceX, GB300 delivers the lowest modeled serving cost in our comparison at a response speed of 150 tokens per second. GB300 handles more tokens per GPU, and its higher hourly cost still leaves it ahead of GB200 on cost efficiency. The GB300 result uses Dynamo-TRT-LLM, while the GB200 result uses Dynamo-SGLang.

We use the September 28 AgentX snapshot results to help illustrate the serving implications of GLM-5.3. We estimate costs using InferenceX’s model of owning and operating infrastructure at hyperscaler scale. Total tokens include input and generated output, including cached input history reused across turns. Streaming speed is measured using p90 interactivity; time to first token is evaluated separately.

At 150 tokens per second, GB200 costs approximately $0.044 per million total tokens, compared with $0.049 for MI355X running ATOM. That is about 12% lower cost at the same streaming-speed target. These values are estimated from the measured curves, rather than from a separate test at exactly 150 tokens per second.

For this comparison, we use InferenceX’s estimated cost of owning and operating the infrastructure at hyperscaler scale. Total tokens include input and generated output, including input history reused from cache.

Among the Dynamo-SGLang results, GB300 serves roughly 13950 total tokens per second per GPU here, a 17.5% throughput advantage versus 11873 total tokens for GB200. However, we assume $2.31 per GB300 GPU hour versus $1.86 for GB200. The higher hourly cost more than offsets GB300’s throughput lead at this target.

In comparison with AMD, ATOM costs about 13% less than GB200 at 100 tokens per second. GB200 costs about 5% less at 125 and 12% less at 150. Across these three targets, neither system has a uniform cost advantage.

Counting only generated output, GB200 costs $5.92 per million output tokens at a response speed of 150 tokens per second, which is about 11% less than $6.68 for MI355X ATOM. This metric is useful to analyze cost under agentic workloads, where agents repeatedly reuse long input histories while generating relatively little new text.

The GB200 measurements on either side of the 150-token-per-second target have p90 TTFT (time to first token) of roughly 14–19 seconds, compared with about 1.1–1.2 seconds for MI355X ATOM. Some of the GB200 lowest-cost runs have much longer waits for the first token.

We therefore make a second comparison using only configurations that were actually tested and achieved at least 150 tokens per second with p90 TTFT at most two seconds. The chart below shows MI355X with ATOM at $0.0607 per million total tokens versus $0.0666 for B200 with Dynamo-SGLang. ATOM costs about 9% less with the two-second first-token limit. If the limit is relaxed to ten seconds, GB300 with Dynamo-TRT-LLM becomes eligible at $0.0451, about 26% below ATOM’s $0.0607.

Optimizations

In GLM’s 5.2 / 5.3 model, its attention architecture reduces memory demands, and long running agents still need to retain earlier conversation history. GLM reduces the cost of handling this history in two ways: KV compression reduces the cached state stored per token, and sparse attention reduces how much of that state each attention operation reads. Then the serving engine would determine where to store the cache and how to retrieve it efficiently.

These B200 results show how much reuse can happen outside of GPU memory. When concurrency increases from 8 to 16 requests, the share of prompt tokens reused from GPU memory falls from 90.3% to 54.8%. Much of that reuse shifts to host memory, which rose from 6.0% to 40.3%. Much of the decline in GPU cache is offset by reuse from host memory, keeping the overall cache hit rate above 95% at all concurrency levels.

Beyond cache management, changes to how the serving engine executes prefill and decoding can also change performance. Two separate GLM-5.2 studies illustrate this:

• vLLM keeping the first decoding step on a consistent CUDA graph execution path reduced average TPOT (time per output token) from about 40 ms to 22 ms in an NVFP4 study.

• On a different 8 MI355X long context workload, ATOM engine processing prefill in chunks across pipeline stages delivered 98% higher total throughput and reduced median TTFT from 28.6 to 8.7 seconds.

TileRT

For GLM5.3 on TileRT, AMD MI355X was supported first. TileRT is a low-latency LLM inference engine for that compiles decoding into a single persistent kernel, reducing launch overhead and overlapping computation, memory access, and communication.

It prioritizes faster token generation per user, rather than maximum batched throughput, while vLLM handle prefill.

On AgentX, FP8 TileRT MI355X achieves 2x the P90 interactivity than the best FP4 MI335X config. Comparing to GB300 NVL72, it brings a 40% uplift in interactivity. These are serving metrics that used to require specialized hardware.

However, TTFT is suboptimal and there is still room for development, such as optimizing KV transfer, FP4 support, and larger batch sizes.

DeepSeek Sparse Attention

GLM-5 (and GLM-5.x models) is a 744B total, 40B active-parameter mixture-of-experts model. For every token, it has 1 shared expert and is routed through 8 out of 256 experts, which is sparsity 32. It features DeepSeek Sparse Attention, which we will discuss in this section.

DeepSeek Sparse Attention (DSA), introduced in DeepSeek V3.2, consists of two components: A lightning indexer that selects top K tokens, and a sparse Multi-Latent Attention (MLA).

Lightning Indexer

A lightning indexer is functionally similar to a lightweight attention mechanism. Queries and keys are down-projected to lower dimensions, with indexer queries being multi-headed and indexer keys single-headed. Since we only need a query and key relation score (logits) for selecting top K tokens, the indexer doesn’t need value embeddings or the softmax operation for normalized logits. Instead, it computes a query-key dot product, followed by a ReLU non-linearity. The indexer then computes each token’s indexer score as a weighted sum of logits across all query heads, which we use to select the top K tokens with. This means despite the indexer having multiple heads, token selection is shared across multiple attention heads. When selecting top K tokens, we keep queries that attend to less than K tokens as dense attention, and we select top K otherwise.

Sparse MLA

As we explained in our Kimi K3 article, MLA operates in two modes: Multi-Head Attention (MHA) mode and Multi-Query Attention (MQA) mode. Both come with trade-offs: MHA mode has lower FLOPs but 42x higher memory cost, MQA mode has lower memory cost but up to 3.4x higher FLOPs. DSA uses MQA mode, since sparse attention attends to less tokens,mitigating the higher FLOP usage. However, in practice, there is a sequence length threshold where under it, MHA mode is actually more efficient that MQA mode. The intuition is that at shorter sequence length, the memory loading time isn’t as dominating, and MHA mode may have an edge for being lower FLOPs. vLLM implements this feature, and configures MHA mode for sequence length 2K to about 5K for data parallel, and 2K to about 77K for tensor parallel. The 2K minimum is due to top K being 2048, and as mentioned above, attention below 2K is dense.

MLA Modifications

Comparing the MLA dimension configurations of GLM-5 and DeepSeek V3.2, we see that the most notable differences are the number of query heads and the query key head dimension.

GLM-5 tech report mentions DeepSeek chose the number of query heads according to the roofline of H800. This is likely referring to the fact that MLA’s SDPA arithmetic intensity during decode is roughly at the ridge point of H800. Derivation as follows. Assume

• L: Sequence length

• d: Head dimension

• r: RoPE dimension

• H: Number of heads

• b: Number of bytes per parameter

In MQA mode, the FLOPs are

2 * H * L * (d+r)  # P = Q [H, d+r] @ K.T [d+r, L]
2 * H * L * d      # O = P [H, L] @ V [L * d]

and the memory loaded is

b * H * (d+r)  # Q
b * L * (d+r)  # K and V

So the arithmetic intensity is

(2 * H * L * (2 * d + r)) / (b * (d+r) * (L + H))
= (2 * H * L * (2 * d + r)) / (b * L * (d + r))  # L >> H
= (2 * H * (2 * d + r)) / (b * (d + r))

If we plug in the DeepSeek V3.2 configuration d = 512, r = 64, b = 2), we get

(2 * H * (2 * 512 + 64) / (2 * (512 + 64)) ~= 2 * H

H800’s practical arithmetic intensity is 258.2 FLOP/B (Practical peak 865 TFLOP/s, 3.35 TB/s bandwidth, according to DeepSeek), and plugging in the formula gives us H ~= 128.

The fact that GLM-5 has H = 64, half the number of query heads, implies that GLM-5 may be designed for different hardware. If we plug in GLM-5’s configuration (d = 512, r = 64, b = 2, H = 64), we get an arithmetic intensity of 120.8 FLOP/B. Thus, we suspect GLM-5 is optimized for Moore Threads MTT S4000, which is about 128 FLOP/B. Moore Threads’ collaboration with Z.ai, such as GLM-5.3-Flash day 0 support, corroborates our theory.

Separately, GLM-5 increases the QK NoPE dimension from 128 to 192, as ablations show the configuration is superior under iso-FLOP and iso-parameter constraints. This increases the SDPA head dimension from 192 to 256, which is still a 33% FLOP reduction after considering the head number reduction.

IndexShare

The latency cost of the indexer becomes non-negligible as context grows. To mitigate the cost, Z.ai proposes IndexShare (aka IndexCache) in GLM-5.2, where every 4 DSA layers share one indexer.

IndexShare introduces complications in training and inference. Standard DSA training involves two stages: A dense warm-up stage where all model weights except the indexer are frozen and attention computation is dense, and then a sparse adaptation stage where the full model is trained with sparse attention.

In terms of training objective, the indexer is trained to align with the sum of attention head scores, using a KL divergence objective. IndexShare adapts the training objective to aligning the indexer logit distribution with the average distribution of attention scores across shared layers.

For inference, typically the indexer caches keys, like attention caches KV. Since multiple layers share one indexer, we need to additionally cache the top K indices at the full layer, in order to reuse the top K selection information at shared layers.

This design reduces indexer cache by 75%, reduces indexer FLOPs by 75%, and boosts throughput by 1.5x to 1.8x at different context lengths and prefill/decode.

Post-Training Design

Pipeline

After pre-training and mid-training, GLM-5’s post-training pipeline starts with supervised fine-tuning (SFT), continues with three RL training stages (reasoning RL, agentic RL, and general RL), and ends with an on-policy cross-stage distillation, distilling reasoning and general RL back into the SFT checkpoint.

Here we highlight two details. First, the tech report states that the reasoning RL model is trained entirely on-policy, implying that it is trained synchronously. Synchronous RL training is low system efficiency but has higher RL training stability. Second, on-policy cross-stage distillation is similar to Multi-teacher On-Policy Distillation (MOPD), where a student model is RL trained with on-policy distillation over multiple teacher models. However, unlike MOPD, whose goal is to merge expert teacher models into one model, GLM-5 here seems to be more about recovering old capability loss when the model learns new capabilities. Following this assumption, we are not sure why the Z.ai only uses reasoning and general RL checkpoints as teachers, instead of using all three RL checkpoints.

Based on the announcement blog post, GLM-5.2 may have replaced the on-policy cross-stage distillation with a standard MOPD pipeline, which Z.ai named parallel OPD training. Z.ai touted its scale and training efficiency: they merged more than ten expert models into a final model by training for about two days.

RL Training Algorithm

Z.ai adopts different RL algorithms for synchronous and asynchronous RL stages. Here we compare the major differences in the RL objectives.

In the synchronous RL stage, Z.ai adopts Group Relative Policy Optimization (GRPO)-based RL algorithm with modern adaptations, including:

• Remove KL regularization term to improve system efficiency and allow more aggressive gradient updates

• IcePop: Apply token-level masking based on the training-inference mismatch ratio

• Clip-Higher: Use a higher max threshold for importance sampling ratio to encourage exploration

In contrast, Z.ai employs a REINFORCE-like RL algorithm for asynchronous RL stage, adding Direct Double-Sided Importance Sampling (DIS), an IcePop-like clipping mechanism. Below we compare the token-level objectives for the two RL stages:

The importance sampling (IS) ratio compares the probability of each sampled action under the current policy and the behavior policy, scaling each action’s contribution to the training objective. For IS ratio, the synchronous RL stage utilizes the trainer policy of the previous iteration for the behavior policy, whereas the asynchronous RL stage utilizes the inference policy for that. In an asynchronous training setting, policy updates happen multiple times during rollouts, so we will need every checkpoint version in between to calculate the IS ratio. However, this is infeasible due to the high memory and latency overhead. As a result, Z.ai opts for using inference policy for efficiency and simplicity, at the cost of numeric stability challenges.

Finally, we highlight the major differences in advantage calculation in RL training, cross-distillation, and long-horizon RL training. RL training employs the GRPO standard group-normalized advantage, and cross-distillation uses the reverse KL divergence.

Single-Rollout Asynchronous Optimization

Long horizon tasks produce extremely long trajectories, and in the context of GRPO, the high variance in trajectory length produces unstable training signals that may skew towards longer trajectories. What’s worse, since RL rollout workload is sensitive to end-to-end latency, the tail-latency of a straggler rollout worsens for long horizon tasks. To mitigate this issue, Z.ai introduced Single-rollout Asynchronous Optimization (SAO) for long-horizon RL training in GLM-5.2. SAO replaces group-normalized advantage with a modified Generalized Advantage Estimation (GAE), inspired by Proximal Policy Optimization (PPO). GAE only needs a single rollout to compute the advantage. This removes the need to do multiple rollouts per prompt, mitigating GRPO’s short-come.

GAE is computed with values (In RL terms, expected returns of the current state), estimated by a value model. Concretely, the value model takes the rollout as input and produces a value estimation for every token. SAO adapts GAE to agentic RL training by removing the observation tokens, i.e. tool call results, from the optimization. Previously in GLM-5, skipping the tokens is sufficient; in SAO, we modify the advantage estimation formula to bypass the observation tokens. We refer readers to the SAO paper and its appendix A.1 for more intuition on the Bellman target and the ablations.

In SAO, the value model is a separate LLM concurrently trained with the policy model. This means early stages of training are susceptible to high variance of advantage calculations, due to a weak value model. To combat this, SAO proposes multiple techniques for training value models:

• Two Time-scale Update Rule: Updating the value model twice as frequently as the policy model. This means training on each data batch twice at every policy training step

• Attention Parameter Freezing: Freezing the value model’s full attention layers to improve training stability

• Scaling Pre-training: Scaling the value model pre-training corpus to initialize the value model, so that the value model can robustly estimate values in the early stages of training. Unfortunately, Z.ai didn’t disclose the scaling information of pre-training corpus

At the cost of computational overhead and doubling memory footprint, SAO replaces group-wise sampling with a value model, mitigating the latency issues of GRPO.

Post-Training Infrastructure

Z.ai uses slime, one of the best open-source RL training frameworks, as the post-training infrastructure throughout all GLM-5 models. We refer readers to the slime GitHub repo to learn more about its design and features, and here we highlight two features.

Performance characteristics of RL rollouts are highly task-dependent, and each task comes with different tools and reward functions. To ensure system efficiency, Z.ai designed a server-based multi-task rollout orchestrator. Each task acts as an independent microservice server, controlling rollout and reward logic. The orchestrator balances the overall performance, controlling the per-task rollout ratio and generation speed. Centralization also allows dynamic ratio adjustments across tasks, fine-grained progress monitoring, and many more. Z.ai claims the orchestrator supports 1000 concurrent rollouts when training GLM-5.

In the GLM-5.3 announcement, Z.ai claims to use local storage as a cache layer in addition to host memory, which enables dynamic teacher model switching and prefetching for MOPD. slime PR#1538 implements teacher switching by introducing a TensorBackuper class, which stores teacher weights as PyTorch tensors pinned to CPU memory. When a rollout is routed to a teacher, a TensorBackuper object copies the teacher model back to GPU memory to perform a forward pass (implementation here). However, this is CPU-backed weight swapping, and we have yet to see storage-backed implementation.

In our paywall section, we present our analysis of GLM-5.3’s cybersecurity capabilities.

Read more

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Our teardown traces 18A from its four-sheet RibbonFETs (Intel’s marketing name for GAAFETs) and gate stacks through contacts, frontside and backside wiring, and the bonded carrier. We explain how these material and integration choices improve gate control and reduce resistance, while adding capacitance, thermal resistance, and process complexity. Our measurements put Panther Lake’s 18A compute logic and TSMC N3E GPU logic at similar logic density. However, 18A does not lead TSMC N3P, N2 or Samsung SF2 in peak density. Panther Lake’s CPU cores are incremental updates, and the high-end GPU still uses TSMC N3E.

Panther Lake assembles one compute tile, one GPU tile, and one I/O tile atop a passive base tile using Intel’s Foveros-S advanced packaging. Both compute tile variants use Intel 18A. The Xe3 GPU options are a 4-core GT1 tile on Intel 3 and a larger 12-core GT2 tile on TSMC N3E. Both I/O tile variants use TSMC N6. [1], [2]

Our analysis centers on the PTL-U compute tile, both the 4-core and 12-core GPU tiles, as well as the 12-lane I/O tile.

PowerVia

In conventional chips, power and signal are routed through the same frontside metal stack towards the device frontend. Power rails consume scarce routing resources near the transistors, while tall via stacks carry VDD and VSS from the coarse upper wires to local rails. Backside power delivery network (BSPDN) moves the main power network behind the transistor layer, to the backside, separating it from frontside signal routing. We covered BSPDN and its impacts in 2024. [3], [4], [5] Intel’s BSPDN implementation, branded as “PowerVia”, routes power through dedicated backside metals to nano-TSVs, which connect those rails to local source/drain (S/D) contacts.

Implementing that separation requires Intel to build the interconnect stacks from both sides of the wafer. The frontside comprises the M0-M14 signal stack, while the backside comprises the BM0-BM5 power stack. M0 and BM0 are closest to the transistors.

The nano-TSVs connect the two sides, but Intel patterns and etches each via from the front after forming the contacts. A narrow via runs from the side of the contact deep into the silicon substrate. Intel then completes the frontside signal metal stack, bonds the wafer to a carrier, flips it and removes the original substrate until the buried via tips are exposed. The backside metal stack is then deposited directly on the revealed vias.

The nano-TSV and backside-via profiles taper in opposite directions because Intel forms them from opposite sides of the wafer. The transistor structures form the FEOL. Local contacts and nano-TSVs connect them to the wiring. M0 begins the frontside interconnect stack.

The silicon carrier remains attached above the frontside interconnects. It supports the device wafer during substrate removal and backside processing and remains part of the finished chip’s thermal path.

PowerVia removes the main power distribution from the congested frontside metals, routing supply through shorter and wider backside wires. Its lateral landing still occupies area in the standard cell, so it recovers less cell area than a direct backside contact. [3] Nano-TSVs beside the logic devices carry VDD or VSS from the backside power network, while signal connections continue upward through the frontside metals.

Backside Interconnects

Samsung SF2 data is included for comparison to Panther Lake’s within this article. SF2 is the incumbent GAA foundry node but lacks BSPDN, serving as a useful reference to evaluate 18A. A full teardown of Samsung’s S26 products, processed on SF2, will be shared soon.

The PowerVia supply path runs from the backside Cu rails through Mo-lined W nano-TSVs to the local transistor contacts. In this cross section, the tapered connection spans roughly 150 nm from the contact level to BM0. The Ta liner confines Cu and promotes adhesion to the surrounding stack; the AlOₓ etch stop controls the next dielectric etch above the rail. Dielectric beneath the ribbons electrically separates the devices from the backside wiring and removes the conducting silicon body below the channel. [6]

AlOₓ serves as an etchstop (ES), enabling endpointing and protecting the underlying layers. Low-volatility aluminum fluoride reaction products resist the fluorinated plasma, allowing a thin AlOₓ film to protect the metal while the surrounding low-k dielectric is removed. [6], [7]. While the BM0 and layers above the M1 lines show double AlO_x layers, Our SMIC N+3 teardown showed single AlOₓ layers. SMIC uses a simpler local AlOₓ substack, while the remaining cap and etch sequence provide the required landing protection.

So why double layers? The closely spaced AlOₓ doublets provide two protected endpoints in the etch sequence. Intel documents an AlOₓ/SiN/AlOₓ stack that explains the benefit. The main dielectric plasma etch stops on the first AlOₓ film; a selective wet clear opens that film; a second plasma etch removes the intermediate SiN and stops on the second AlOₓ film. The final wet clear exposes the metal landing surface. SiN is the intermediate dielectric in Intel’s published example. [8]

The second stop protects the metal through a cap breakthrough. Wide openings can etch faster than narrow ones, and etch depth varies across the wafer. Metal under an early-clearing opening would otherwise be exposed while other openings still need more etching. Staged protection widens the process window and reduces metal erosion, corrosion and void formation. [8]

TSMC documents AlN/AlOₓ/SiOC/AlOₓ above Cu, with AlN blocking Cu diffusion, and a simpler AlN/SiOC/AlOₓ variant that omits one AlOx film. [9] Levels with different opening sizes, aspect ratios, pattern densities and cap materials need different etch margins. A double AlOx stop is useful where another protected endpoint justifies the added processing.

The extra film adds formation, selective opening and cleaning steps, plus another set of interfaces to control adhesion, moisture, and stress. These blanket films are opened through the existing via pattern, so each film does not require another lithography mask. AlOₓ adds parasitic capacitance when it replaces lower-k dielectric; two thin AlOₓ films can nevertheless contain less AlOₓ than one thick film. Total thickness, placement, and theintermediate dielectric determine the electrical cost. Deposition chemistry also changes AlOx permittivity and residual hydroxyl content, which can oxidize the underlying metal. [7], [8], [10], [11]

The backside stack separates into relatively fine BM0-BM2 wiring near the devices and coarser BM3-BM5 power distribution. The largest pitch increase occurs between BM2 and BM3. BM0’s pitch closely matches the logic-row height, fitting local power delivery to the cell rows. Higher levels aggregate current through larger conductors: routing density becomes less important than low resistance and current capacity as the network approaches the package. This hierarchy provides wide power wiring for the power delivery network without consuming scarce frontside signal-routing resources. [3]

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Frontside Interconnects

Intel 18A combines Mo-lined W contacts and nano-TSVs with a separate backside Cu power network. Samsung SF2 keeps power on the frontside, using Ti-based contact interfaces and Ta-based barriers and Co liners around Cu wiring.

In 18A standard-cell rows, backside power rails supply the devices through nano-TSVs within the cells, freeing frontside routing resources. Samsung’s M0 accommodates both power and signal connections.

From the device toward M0, the connection runs through a Ti-based S/D interface, W contact fill, a Mo-lined W via, and the Cu M0 wire. Mo supplies a conductive nucleation and adhesion layer for W, replacing the resistive TiN liner used in conventional W integration. This increases the effective conduction volume within the feature while retaining W fill and its established polishing, cleaning and etching processes. Intel’s Mo/W patent describes this integration tradeoff. The nano-TSV uses the same Mo-lined W construction in the backside supply path. [12]

The move from TiN to Mo is an incremental change. While a full Co or Mo fill can also reduce the volume lost to liners in very small features, it requires new integration schemes that increase complexity and risk. Cu remains attractive for wider wires due to its low resistance. As wires and vias shrink, the diffusion barrier consumes an increasing fraction of their cross-section. [12], [12], [14]

Intel uses Co/Ru liners at M0-M1, Co at M2-M4, and Nb at M5-M9. The lower-level liners help Cu adhere and reduce void formation during trench fills. Applied Materials’ Endura has new thermal control that facilitate wetting process, so the thin film continuity is good enough that good capillary pressure will drive Cu atoms to the via bottom without voiding.

Intel’s choice to use Nb is particularly interesting. Intel’s Nb patent describes a conductive diffusion barrier intended to reduce the barrier’s contribution to resistance relative to conventional Ta-based barriers, particularly at via bottoms where all current crosses the barrier. The patent pairs Nb in coarser levels with the option of lower-cost PVD processing. [15], [16]

The upper metal layers support thicker barriers formed through physical vapor deposition (PVD) despite its worse coverage and uniformity. Meanwhile, the lower metal layers require thinner barriers deposited through conformal atomic layer deposition (ALD). Co/Ru adds another material interface and requires controlled deposition and Cu fill. Changing liners and barriers by metal layer allows Intel to optimize interconnect resistance, process complexity, and reliability. [15, 16]

RibbonFET

RibbonFET, Intel’s name for its gate-all-around FETs (GAAFETs), replaces the FinFET’s vertical fins with four stacked horizontal silicon nanosheets, allowing the gate to surround the channel on every side. The path to GAAFET begins with the planar transistor.

A planar MOSFET places the gate above the channel between its source and drain. Pairing an NMOS with a PMOS transistor creates a CMOS inverter, in which the NMOS pulls the output low for a high input, and the PMOS pulls it high for a low input. The gate must retain electrostatic control of the channel to ensure clean switching. As gate lengths shrank, the drain began to compete with the gate for that control, increasing off-state leakage.

Electrostatic control was restored through an architectural evolution that raised the channel into a vertical fin and wrapping the gate around three sides. Called “FinFET”, this new architecture packed more effective channel width into a smaller footprint. Further scaling made it harder to maintain both drive current and leakage within smaller cells, and reintroduced the same problems planar MOSFETs faced. Nanosheet GAAFETs close the fourth side by replacing the vertical fin with a stack of horizontal nanosheets, each surrounded by the gate. The tighter electrostatic control suppresses leakage at shorter gate lengths while stacking adds effective channel width within the cell footprint.

In a FinFET process, channel width changes in discrete steps as designers must add or remove whole fins. Nanosheet width can instead be adjusted continuously within the process’s design rules. Wider sheets increase drive current, while narrower sheets reduce capacitance at the cost of drive current. Intel 18A uses stacks of four nanosheets each and varies their widths across logic and SRAM. At the process level, adding more sheets to each stack increases effective channel width and drive current, but complicates fabrication.

RibbonFET vs MBCFET

Samsung began GAAFET production in 2022 with SF3E, following with SF3 and now SF2. Its ‘MBCFET’ provides a useful structural comparison with Intel’s first RibbonFET implementation. [17] STEEL is digging deeper into SF2, used in the Exynos 2600, and TSMC’s GAAFET N2, used in Apple’s A20 Pro, in upcoming newsletter articles. We’re throwing some teasers on X. Let’s compare Samsung SF2’s MBCFET with Intel 18A’s RibbonFET.

Subscribe now

Even to the untrained eye, Intel’s extra nanosheet is obvious. Intel stacks four ribbons to Samsung’s three. Samsung’s sheets are much wider in these fields, so both sheet count and width matter to the available channel perimeter. Sheet width also changes which silicon surfaces carry current. On conventional (001) silicon, wide nanosheets emphasize the broad top and bottom surfaces, favoring electron transport; the larger sidewall contribution in a narrow sheet favors hole transport. Thinner sheets improve gate control but increase confinement and scattering. This makes width and thickness part of the NMOS/PMOS balance, alongside strain and threshold voltage. [18], [19]

GAAFET designs like 18A use different work-function-metal (WFM) stacks for NMOS and PMOS. Around each ribbon, a thin SiOx interfacial layer separates the silicon channel from the HfOx high-k dielectric, with La providing dipole tuning and the WFM wrapping the dielectric. NMOS uses a TiAl-based stack, while PMOS uses TiN WFM. W fills the remaining gate trench, providing a lower-resistivity path where the work-function layers are no longer needed. In this field, the PMOS stacks leave room for W between ribbons, while the NMOS stacks occupy more of those gaps. A silicon-based dielectric marks the P/N boundary, allowing the PMOS and NMOS gates, sharing the same gate trench, to be processed sequentially.

Fast logic paths, retention circuits, and SRAM need a family of threshold options. Changing threshold without substantially changing device dimensions, capacitance or fabrication complexity is valuable. FinFET processes typically use different work-function-metal stacks. In a four-ribbon GAA stack, the narrow sheet-to-sheet gap limits how much WFM can fit around each channel. La in the gate dielectric creates interfacial dipoles at the SiOx/HfOx boundary, shifting effective work function and tuning threshold voltage. This gives Intel another control alongside its NMOS and PMOS WFM stacks. Low-threshold devices improve critical-path drive; higher thresholds reduce leakage elsewhere. Dipole tuning is especially useful in GAA because it changes threshold without consuming the narrow intersheet gap with thicker WFM.

Precise control of La incorporation, diffusion and interface quality has long been a challenge, limiting viability in high volume production but is now seen from every leading-edge foundry. Intel’s patent describes depositing a dipole-forming oxide above HfOx and annealing it toward the interfacial oxide before completing the work-function and fill metals. This separates threshold tuning from the space available for metal. Newer research addresses the thermal cost: imec’s 2026 dipole-middle research inserts the shifter between two HfO_x depositions, shortening the diffusion path while protecting SiO_x during patterning. [20], [21]Matched-cut EDS, Intel 18A (left) vs. Samsung SF2 (right).

Intel retains raised source/drain epi beneath its contacts, while Samsung recesses W deep into the epi to form a V-shaped Ti-lined interface. The deeper contact increases metal-to-semiconductor area and shortens the current path from the lower sheets, reducing contact and spreading resistance. It also removes epi volume and brings the contact etch closer to the channel ends. Retaining more epi preserves the material available for strain transfer, especially from SiGe into PMOS. These geometries balance contact access against stress engineering and etch margin. [22], [23]

Samsung stacks three sheets to Intel’s four ribbons, and both processes use sheet width to tune drive strength. In our Samsung cross-sections, widths range roughly from 19 to 30 nm in the NPU rows and 37 to 50 nm in the CU cell. The Samsung nanosheets taper, with the widest sheet at the bottom and the narrowest at the top. Both processes use HfOx gate dielectric and Ti-based work-function stacks, with Al in the NMOS stack. In the Samsung devices shown here, the dielectric and WFM occupy the intersheet gaps, leaving W above the top sheet. Intel’s PMOS stack leaves more room between ribbons, and W fills those gaps while the thicker NMOS stack leaves W mainly in the upper trench. Gate-stack EDS maps.

The W between Intel’s PMOS ribbons provides a conductive path close to the lower gates. Where WFM fills the entire gap, the gate still surrounds the channel, but voltage reaches it through the more resistive work-function films. Thinner WFM and dipole tuning preserve room for low-resistivity fill; Mo and Ru are alternative fill metals being developed for further scaling. [24]

A masked, sequential WFM flow explains the different gate heights and inter-nanosheet fill. The proposed sequence below shows how separate NMOS and PMOS work-function steps produce that geometry.

Enabled by the BSPDN process, Intel replaces the dense-logic silicon subfin with dielectric, removing the parasitic conduction path below the ribbons and reducing substrate-related capacitance. A retained silicon body as in classical, non-SOI, planar and FinFET designs needs junction and punchthrough-stop engineering to suppress leakage. Dielectric isolation makes that leakage less sensitive to the subfin doping profile but adds removal and fill steps. It also weakens the direct thermal path through silicon, making the contacts, metal stacks and package more important for heat extraction. [24], [26]

Fluorine is concentrated around selected Intel device structures in the maps. WF₆ is a standard precursor for W nucleation and fill, while barrier films protect adjacent dielectrics from fluorine attack. Low-fluorine W processes reduce the residual-F burden. Chloride-based precursors avoid introducing F during W deposition, but require control of chlorine attack, nucleation and fill quality. The integration target is a continuous, low-resistance W path with a thin protective liner and minimal chemical damage to the surrounding stack. [20], [27], [28]

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Library analysis

We measured cell height, gate pitch, metal geometry, and ribbon dimensions at the XTEM sites shown below. The tables group these dimensions by site and device polarity. Our “sheet cuts” cross the silicon channel and show the ribbons end-on. “gate cuts” run along the channel through successive gates.

The 18A logic cell dimensions point to a five-track logic library while the N3E and Intel 3 cell dimensions evidence a seven-track logic library. The DDR-PHY uses wider M0 wires and much larger spacing than core logic. That trades routing density for lower wire resistance and weaker coupling between neighboring nets. The geometry suits the current delivery and coupling requirements of analog, clock, and I/O circuitry.

PowerVia lets 18A combine a compact cell height with wider M0 geometry by moving the main power rails off the signal-routing tracks. That relaxes local wire scaling while preserving a small cell footprint. Cell height and gate pitch set the geometric density; pin access and routability determine how much of it a real block can use. [29]

The biggest takeaway from our gate-pitch measurements is that Intel 18A compute logic and TSMC N3E GPU logic have similar density in the Bohr representative-cell model. The 18A example is 18.6% denser than the Intel 3 GPU example. Gate pitches are nearly identical across the three sites, so cell height drives most of the difference.

The Bohr model combines a four-transistor NAND2 spanning three gate pitches and a 32-transistor scan flip-flop (SFF) spanning nineteen pitches, weighting their densities 60:40. The sensitivity column shows how independently changing cell height and gate pitch by ±1 nm changes the result. This compares representative cell geometries; whole-die density also depends on cell mix and placement.

The 18A P-core gives M0 substantially more metal cross section than the N3E vector engine. Treating each profile as a trapezoid gives 2.63 times the area per line and 1.84 times the area after normalization by routing pitch. The larger section reduces the geometric contribution to line resistance and lowers current density for a given current. Taller and wider wires also add capacitance, so circuit delay depends on the balance of resistance and capacitance. The DDR-PHY has less metal area per routing width than the 18A core fields, while remaining above N3E. [30]

Area = height × (top CD + bottom CD) / 2, including liners. Area/pitch normalizes by routing width. Taper is the symmetric sidewall angle from vertical, with the largest angle belonging to the DDR-PHY.

Compute tile

The measurements show how ribbon dimensions and gate-stack geometry vary across the compute tile and between NMOS and PMOS to balance channel drive, gate load and the space needed for the dielectric/WFM stack across logic, SRAM and the DDR-PHY. Width mainly changes available channel perimeter; thickness also changes electrostatic control and carrier confinement. Gate-stack thickness then determines the space left for low-resistivity fill

P-core and LP E-core logic

Both the P-core and LP E-core use multiple nanosheet widths. Widths are measured on high-magnification XTEMs while wider-field images demonstrate additional width choices within the LP E-core. Multiple widths are expected even within an LP E-core. Timing-critical paths, buffers and cells with different fanout need different drive strengths. The lower-magnification fields show this width diversity beyond the sites quantified in the table.

L2 and L3 SRAM

GAA gives SRAM designers another way to balance the pull-up (PU), pass-gate (PG), and pull-down (PD) transistors. FinFET bitcells set device strength through fin count while GAA adds nanosheet width as a sizing knob. In a 6T SRAM cell, a strong pull-down relative to the pass-gate limits read disturbance, while a strong pass-gate relative to the pull-up improves writability. During a write, the pass-gate and write driver pull the node storing “1” below the inverter trip point. During a read, the pull-down holds the node storing “0” low. Bias, threshold voltage, mismatch and assist circuitry set the remaining margin. FinFET high-current cells commonly use a PU:PG:PD fin-count pattern of 1:2:2, a device-sizing ratio rather than a current ratio.

Ribbon width lets Intel balance SRAM strengths without adding whole fins. The L2 cell uses its narrowest ribbons for PU and widest for PD, improving writability and read stability respectively. Intel’s disclosed HCC operates without assist; its denser HDC uses negative-bitline write assist. Pulling the selected bitline briefly below ground increases pass-gate overdrive so it can overpower the pull-up at lower supply voltage. That buys density and low voltage writability at the cost of boosting circuitry, switching energy, and additional voltage stress that must be controlled. [31], [32]

Four rectangular ribbons give the perimeter = 8 × (width + thickness), before corner rounding. PG/PU is 1.49 and PD/PG is 1.16.

The L3 structures closely resemble L2 in layout and cell height. Fewer L3 nanosheet widths are tabulated because fewer high-magnification images were available.

DDR PHY

The DDR-PHY trades density for controlled analog behavior and reliable off-chip signaling. It contains drivers, receivers, delay circuits, and calibration logic that set drive strength, sampling time, and voltage margin. Repeated four-sheet devices with similar widths fit the use of regular transistor units for matching and programmable drive. Its wider local wiring provides room for current delivery and separation of sensitive signals, while consuming more area than a dense core-logic grid. The layout serves the memory channel’s electrical requirements as well as digital logic density. [33]

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page.

Intel 3 GPU devices

Vector engine logic

Intel 3’s XVE logic uses two-fin PMOS and NMOS devices with power rails in M0. Its cell height and M0 pitch give a seven-track geometry, two tracks more than the 18A logic. One-fin groups also appear among the two-fin devices.

Intel 3 L2 SRAM

The Intel 3 L2 SRAM uses the familiar HCC sizing pattern: one PU fin, two PG fins, and two PD fins.

N3E GPU devices

Vector engine logic

The N3E XVE field contains repeated two-fin devices with seven-track cell geometry. N3E remains a FinFET process, giving Panther Lake a direct FinFET-to-RibbonFET comparison.

N3E L2 SRAM

The N3E L2 SRAM uses the same PU:PG:PD fin-count pattern of 1:2:2.

Floorplan analysis

Panther Lake-U follows Lunar Lake’s floorplan quite closely. Both pair 4 P-cores with 4 LP E-cores and NPU, media and display engines in similar locations. Lunar Lake also uses Xe2, the direct predecessor to Panther Lake’s Xe3 GPU. This makes Lunar Lake the most direct basis for our comparisons. Arrow Lake differs in core count and uses the older Xe-LPG GPU architecture, so we only use it where it offers a more direct component-level comparison.

Compute tile

Panther Lake compute-tile floorplans remain sparse even months after launch. Intel 18A’s backside metal and dielectric stack must be removed without damaging the underlying structures before a clean transistor-level floorplan can be imaged. Most published die shots hide or heavily process the background, but we are quite proud of the die shot we achieved and are excited to show the work we have done.

We measured the areas of the key components on the compute tile and compared them with their Lunar Lake predecessors on TSMC N3B. These help us to capture changes in block area and compare the two chips across process nodes and designs.

Our total tile areas exclude the scribe-line area. The compute-plus-GPU subtotal below uses the PTL-U compute tile and GT1 GPU; it excludes the I/O tile and passive base. Individual block areas use the boundaries marked on the floorplans

The compute-plus-GPU row is recomputed from the displayed PTL-U and GT1 areas. Component rows use their stated per-region counts and are not an additive partition of the whole tile.

The P-core area remains almost unchanged between Lunar Lake and Panther Lake, despite L2 capacity increasing from 2.5 MiB to 3 MiB. Arrow Lake uses the same Lion Cove core as Lunar Lake but also has a 3 MiB L2. Cougar Cove fits 20% more L2 into the same P-core area. The larger private cache keeps more of each core’s working set close to its execution units, reducing access to shared L3 and DRAM. Extra capacity adds storage leakage and lookup energy, so designers balance it against avoided lower-level accesses. The shared P-core L3 cache also shrank by 14.8%. [2]

Cougar Cove combines a similar footprint with Intel’s reported power-efficiency improvements. RibbonFET’s tighter channel control reduces leakage, while PowerVia reduces supply droop and allows tighter voltage guardbands. [1]

Darkmont’s four-core LP E-core cluster is 5.0% smaller than Skymont’s on Lunar Lake, with most of the reduction in its L2 regions. The 1 MiB region shrank by 8.4% and the 1.5 MiB region by 14.9%. The tag arrays also use one fewer visible row. Tags identify which memory addresses the data array holds, so rearranging them changes the cache’s layout and wiring without requiring less data capacity. [2]

The LP E-cores share one L2. This pools capacity and avoids duplicating all the cache machinery, but the four cores contend for its banks and bandwidth. Their separate cluster also keeps light work away from the performance cluster and its L3, allowing that larger domain to sleep. [1], [2]

Cache area includes more than the storage cells. Tags identify each line, decoders select rows, sense amplifiers read the small bitline signal, and wires connect to the banks. Splitting an array into smaller sections shortens wordlines and bitlines, improving access speed, but duplicates peripheral circuits. Panther Lake’s smaller cache regions therefore reflect the complete memory implementation, including how much of each region is devoted to storage. [34]

Unlike Meteor Lake and Arrow Lake, Panther Lake has no separate SoC tile. The NPU, LP E-cores, memory controllers, PHYs, media and display engines now share the compute tile. This removes an active die and keeps CPU memory traffic on one die. The cost is moving PHY and I/O-related circuitry onto 18A: drivers, receivers and analog circuits must still meet external voltage, loading and signal-integrity requirements, so their area does not shrink like dense digital logic. [1], [2]

The biggest shrink comes from the NPU, which occupies 36.9% less area. NPU 5 consolidates the same total INT8 MAC count into half as many neural compute engines. Each of the three NCEs has a larger MAC array to make the complete NCE envelope 22.6% larger than an NPU 4 engine. Consolidation also halves the number of scratchpads and SHAVE DSPs, from 12 to 6. The MAC array handles matrix multiplication and convolution, while SHAVE executes vector and custom operations that fit the array poorly. [1], [2], [35]

The paired floorplans identify each NCE envelope and its scratchpad, MAC, and SHAVE regions. Each measured MAC polygon is counted once per NCE in the area accounting below.

The scratchpads store weights, activations, and intermediate results near the MAC arrays, allowing repeated use without fetching them again from DRAM. Halving their number delivers the largest measured area saving but leaves less local storage for the same total MAC count. Layers that no longer fit locally require smaller working tiles or more transfers of intermediate data. The benefit depends on keeping the enlarged arrays busy while managing that tighter storage budget. [36]

NPU 5 also adds native FP8. Using half the operand width of FP16 reduces storage and transfer demand, helping workloads fit the smaller local memory budget. Lower precision and format-dependent range make scaling and model validation part of deployment. Hardware activation functions further reduce work that would otherwise occupy the programmable DSPs. [1], [2]

Microsoft requires an NPU to deliver at least 40 TOPS for Copilot+ PCs. Both Lunar Lake and Panther Lake meet this threshold, but Panther Lake uses significantly less silicon.

GPU tiles

Panther Lake is Intel’s first product with Xe3, its latest GPU architecture. It offers two different GPU tiles: a smaller GT1 tile with 4 Xe3 cores on Intel 3 and a larger GT2 tile with 12 Xe3 cores on TSMC N3E.

Panther Lake allows us to compare the same GPU architecture across both Intel 3 and TSMC N3E. Wildcat Lake adds a third Xe3 implementation on Intel 18A. A future newsletter will detail Xe3 and its implementation differences across all three process nodes.

GT2 scales Xe3 to a different physical layout, with render slices arranged vertically instead of GT1’s horizontal arrangement. Slice placement sets the distances to shared cache banks and the D2D interface. Those wires consume area and add delay, so scaling the number of Xe cores also requires a new balance of cache placement, routing and timing. [1]

What’s immediately obvious is that the GT2 tile on TSMC N3E has much smaller Xe cores than GT1. These block areas include logic, caches, and routing.

An Xe core on the GT1 tile is ~69% larger than one on Lunar Lake, and ~55% larger than one on GT2. Intel 3 therefore uses substantially more area per Xe core. The block-area gap exceeds the measured logic and SRAM density gaps, bringing routing, timing targets, cell mix, and floorplan allocation into the comparison.

The measured vector/matrix engine region is almost unchanged between Lunar Lake and Panther Lake’s GT2 tile. Xe3 retains eight 512-bit vector engines and eight 2048-bit XMX engines per core. Its gains also come from feeding those engines more effectively: more resident threads hide stalls, and variable register allocation lets shaders trade registers per thread against the number of threads kept active. [1]

The shared L1/SLM capacity increased by 33% from 192 KiB to 256 KiB, while its area increased only 5%, raising effective density by 27%. L1 retains reused cache lines, while software-managed SLM lets a thread group share data locally. Both reduce traffic to more distant memory. Allocating more SLM per group can also limit how many groups reside on a core at once. [1], [37]

The GT1 tile carries 4 MiB of L2 against 16 MiB on the GT2 tile. GT1 divides its L2 cache into four 1 MiB banks, while GT2 uses eight 2 MiB banks. Each bank contains 128 macros, but each N3E macro stores 16 KiB, twice the Intel 3 macro’s 8 KiB capacity. The N3E macro is only 54% larger while holding twice as many bits, giving it 30% higher density: ~23.7 Mbit/mm² versus 18.3 Mbit/mm². Including bank-level circuitry, the gap widens to ~16.9 Mbit/mm² on GT2 versus ~10.4 Mbit/mm² on GT1.

GT2 gains density with its macros storing more bits per unit area, and those macros occupy more of each cache bank. Larger macros spread decoder and sense-amplifier overhead across more storage, while a more compact bank layout reduces the share spent on control and routing. The compromise is longer wordlines and bitlines that carry more capacitance. [34]

I/O tile

Panther Lake uses two I/O tile variants, both fabricated on TSMC N6. The smaller one provides 4 PCIe 5.0 and 8 PCIe 4.0 lanes and serves lower-tier systems as well as those without a discrete GPU, while the larger one adds 8 PCIe 5.0 lanes, bringing the total to 20 lanes, for discrete-GPU connectivity. Panther Lake SKUs with the larger 10- or 12-Xe GPUs use the smaller I/O tile. [38]

The smaller I/O tile adds a PCIe 4.0 block and a Thunderbolt block to Lunar Lake’s I/O layout, providing four additional PCIe 4.0 lanes and another Thunderbolt 4 port. Its repeated N6 blocks retain nearly identical areas and layouts. Reusing these proven PHYs and controllers avoids porting and requalifying external interfaces on 18A, where faster digital logic offers less benefit to circuits constrained by the off-chip link. [38]

SemiAnalysis’s teardown lab (STEEL) dives deep into the world’s advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

We’re hiring technical experts from system to transistor and everywhere in between. Check out our Careers page.

Foveros-S

Panther Lake offers scalability and modularity through its disaggregated packaging that partition compute, GPU, and I/O silicon into separate tiles allowing for a suite of tile configurations. This partitioning makes the package part of Intel’s node economics as it determines how much leading-edge wafer area each product consumes, which functions can remain on other processes, and how much configuration freedom Intel can offer from a shared set of tiles.

Furthermore, fabricating the compute and GPU tiles separately confines the new 18A process to the compute tile and allows graphics and I/O to use other, more established, and more cost-effective processes. For Panther Lake, the GPU and I/O tiles are assembled alongside the compute tile on a passive silicon base using Foveros-S.

Intel’s current technology brief lists a nominal 36 µm pitch for Foveros-S. Through-silicon vias (TSVs) in the base connect the fine wiring above to the larger package connections below. The functional tiles sit side by side on that passive base in a 2.5D configuration. [39]

Our cross-section through the compute and GPU tiles shows the package’s wiring hierarchy. Microbumps connect each active tile to the passive silicon base; its fine redistribution layer (RDL) carries the short, dense tile-to-tile links. TSVs carry connections through the base to the package substrate, which fans them out to the much coarser motherboard solder joints. The base supplies interconnect, while computation remains in the active tiles above it. [39]

At the compute-tile edge, the higher-magnification inset shows a local microbump spacing of approximately 25.24 µm and a feature width of 12.33 µm.

These local spacings are finer than Intel’s nominal Foveros-S value. The X-ray fields further confirm tighter neighboring bumps, consistent across every die-to-die area found on each tile. Additional X-ray analysis is offered after the paywall.

Putting the memory controller beside the CPU removes the D2D transfer that CPU memory requests required in Meteor Lake and Arrow Lake. This avoids the extra transmitter, receiver, and link traversal, saving interface energy and latency. Panther Lake’s separate GPU still crosses a D2D link to reach DRAM, so its larger local caches also help contain package traffic. [1], [40]

Smaller dies are less likely to contain a random fatal defect, and screening them before assembly prevents one bad tile from consuming a complete package of good silicon. Reuse also spreads design and qualification work across more products. Against those gains, Intel pays for the passive base, D2D circuits, extra bonding and test steps, and losses during assembly. Cost per working product across the portfolio captures the combined effect of wafer yield, reuse, test, and assembly. [29]

Wildcat Lake packaging

Intel launched Core Series 3, formerly Wildcat Lake, on 16 April 2026 for value mobile and edge systems. Wildcat Lake keeps 18A but removes the passive base and combines more functions on one die to simplify the package. The two products therefore reveal two distinct ways to commercialize the same leading-edge process. [41]

Wildcat Lake’s 18A die combines up to two Cougar Cove P-cores, four Darkmont LP E-cores, two Xe3 cores and a smaller NPU. A separate platform-controller die supplies I/O, connected through UCIe, Intel’s first processor implementation of the standard. Consolidating graphics remove a tile boundary and the passive base, reducing assembly complexity for a modest-bandwidth value product. It also ties CPU and graphics scaling to the same die, giving up Panther Lake’s ability to swap in a much larger GPU. [42], [43]

Intel’s manufacturing recovery

In July 2021, Intel CEO Pat Gelsinger set out an ambitious process roadmap aimed at regaining performance leadership by 2025, later described as five nodes in four years. Five years and one CEO later, Intel’s comeback story is not as unambiguously positive as Pat may have hoped. [44], [45]

Intel once set the pace for process technology, bringing high-k metal gate technology and FinFETs into volume production years ahead of the rest of the industry. Its 22 nm FinFET process reached consumers with Ivy Bridge in 2012. [46] Intel’s integrated device manufacturing (IDM) model allowed its architects and process engineers to co-optimize products and processes. Starting with Sandy Bridge, Intel dominated x86, while AMD struggled with Bulldozer.

That lead faltered at 14 nm and broke at 10 nm. Intel targeted a massive 2.7× density increase, but the node arrived years late and required several revisions before it could support Intel’s full lineup. This delay forced Intel to stretch 14 nm across six generations, while TSMC moved ahead in process technology and AMD recovered in x86.

By 2019, Intel was still shipping 14 nm across most of its product stack, with its 10 nm client ramp focused on Ice Lake mobile processors. Meanwhile, TSMC was shipping N7 and N7+, and AMD’s Zen 2 compute chiplets used N7 to raise core counts and improve efficiency.

Intel’s process failures were central to its decline, but unsound business decisions furthered their downward slide. Product delays compounded product mistakes, pushing client, server, and FPGA roadmaps off schedule. Several attempts to enter AI (Nervana and Gaudi) and networking (Tofino) also failed to establish lasting businesses.

Intel’s recovery has focused on consumer CPUs and advanced packaging. Tiger Lake, Alder Lake, Lunar Lake and now Panther Lake have restored Intel’s consumer roadmap. On the process side, Intel 4 shipped with Meteor Lake, Intel 3 with Granite Rapids and Sierra Forest, and Intel 18A with Panther Lake. Intel has also made advanced packaging part of its foundry offering. However, Intel is still playing catch-up in servers. Several Xeon generations arrived years late and trailed contemporary AMD and Arm server CPUs in performance, efficiency, and core count. The process roadmap is back, but Intel does not hold the same process-technology leadership position it held prior to 10 nm.

The introduction of gate-all-around nanosheets and backside power delivery are two of the biggest changes to transistor integration in a decade. Intel took on both changes at once: 18A paired its first RibbonFET with PowerVia in Panther Lake.

Panther Lake is a substantial manufacturing milestone. Our cross-sections show how RibbonFET and PowerVia reshape local contacts and wiring, while the floorplans show where architectural consolidation and process choices save area. A sustained competitive lead depends on product performance, cost, yield, and the next implementation.

The SemiAnalysis STEEL teardown lab breaks down advanced datacenter and AI hardware. To learn more about our pipeline or to commission a teardown, contact sales@semianalysis.com.

WE’RE HIRING: Architecture, floorplan, packaging, manufacturing, and labs experts. Opportunities from system to transistor and everywhere in between. Check out our Careers page..

References

[1] Intel, “Intel Tech Tour 2025 Panther Lake recap,” Doc. 866361, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/866361/ITT_2025_Panther_Lake_Recap1.pdf

[2] Intel, “Core Ultra processors Series 3 datasheet, volume 1,” Doc. 872188-001, Jan. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/872188/872188-001.pdf

[3] N. Horiguchi and E. Beyne, “Backside power delivery: How to power chips from the backside,” imec Reading Room, Nov. 25, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/how-power-chips-backside

[4] W. Hafez et al., “Intel PowerVia technology: Backside power delivery for high density and high-performance computing,” in Proc. IEEE Symp. VLSI Technol. Circuits, Kyoto, Japan, Jun. 2023, pp. 1–2, doi: 10.23919/vlsitechnologyandcir57934.2023.10185208.

[5] K. Fischer et al., “Intel 18A platform technology featuring RibbonFET (GAA) and PowerVia for advanced high-performance computing,” in Proc. Symp. VLSI Technol. Circuits, Kyoto, Japan, Jun. 2025, pp. 1–3, doi: 10.23919/vlsitechnologyandcir65189.2025.11075006.

[6] K.-F. Cheng, C.-L. Teng, H.-Y. Huang, and H.-C. Chen, “Metal oxide composite as etch stop layer,” U.S. Patent 12 176 247 B2, Dec. 24, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US12176247B2/en

[7] S. W. King, “Dielectric barrier, etch stop, and metal capping materials for state of the art and beyond metal interconnects,” ECS J. Solid State Sci. Technol., vol. 4, no. 1, pp. N3029–N3047, 2015, doi: 10.1149/2.0051501jss.

[8] A. V. Mule’, D. J. Towner, D. Seghete, C. R. Ryder, and A. A. Gonzalez, “Multi-layer etch stop layers for advanced integrated circuit structure fabrication,” U.S. Patent Appl. 20220102343 A1, Mar. 31, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20220102343A1/en

[9] C.-C. Wang and J. H. Wang, “Interconnect structure with dielectric cap layer and etch stop layer stack,” U.S. Patent Appl. 20230253247 A1, Aug. 10, 2023. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20230253247A1/en

[10] Y. J. Wu, K.-F. Cheng, C.-C. Lee, H.-K. Chang, and H.-Y. Huang, “Etch stop layer for interconnect structures,” U.S. Patent Appl. 20240332070 A1, Oct. 3, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20240332070A1/en

[11] M. G. Rainville, N. Shankar, K. S. Reddy, and D. M. Hausmann, “Deposition of aluminum oxide etch stop layers,” U.S. Patent Appl. 20180197770 A1, Jul. 12, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.justia.com/patent/20180197770

[12] J. S. Leib et al., “Conductive lines having molybdenum liner and tungsten fill for advanced integrated circuit structure fabrication,” U.S. Patent Appl. 20240429126 A1, Dec. 26, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.justia.com/patent/20240429126

[13] M. Naik, “Cobalt enables power and performance scaling at single-digit logic nodes,” Applied Materials, Dec. 17, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://www.appliedmaterials.com/us/en/blog/blog-posts/cobalt-enables-power-and-performance-scaling-at-single-digit-logic-nodes.html

[14] Lam Research, “Breaking through AI’s invisible barrier with molybdenum,” Lam Research Newsroom, May 6, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://newsroom.lamresearch.com/molybdenum-metallization-ai-revolution

[15] P. Yashar, G. Malyavanatham, and H. Vijwani, “Integrated circuit interconnect structures with niobium barrier materials,” U.S. Patent Appl. 20240112951 A1, Apr. 4, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20240112951A1/en

[16] Applied Materials, “Applied Materials unveils chip wiring innovations for more energy-efficient computing,” Applied Materials Investor Relations, Jul. 8, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://ir.appliedmaterials.com/news-releases/news-release-details/applied-materials-unveils-chip-wiring-innovations-more-energy

[17] Samsung, “Samsung showcases AI-era vision and latest foundry technologies at SFF 2024,” Samsung Semiconductor, Jun. 13, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://semiconductor.samsung.com/news-events/news/samsung-showcases-ai-era-vision-and-latest-foundry-technologies-at-sff-2024/

[18] C. W. Yeung et al., “Channel geometry impact and narrow sheet effect of stacked nanosheet,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Dec. 2018, pp. 28.6.1–28.6.4, doi: 10.1109/iedm.2018.8614608.

[19] S. Mochizuki et al., “Evaluation of (110) versus (001) channel orientation for improved nFET/pFET device performance trade-off in gate-all-around nanosheet technology,” in Proc. Int. Electron Devices Meeting (IEDM), Dec. 2023, pp. 1–4, doi: 10.1109/iedm45741.2023.10413854.

[20] D. G. Ouellette et al., “Fabrication of gate-all-around integrated circuit structures having molybdenum nitride metal gates and gate dielectrics with a dipole layer,” U.S. Patent 12 051 698 B2, Jul. 30, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US12051698B2/en

[21] imec, “Advancing the CFET-based device roadmap with novel integration modules and standard-cell architectures,” imec, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/advancing-cfet-based-device-roadmap-novel-integration-modules-and-standard-cell

[22] A. Reznicek, X. Miao, C. Lee, and J. Zhang, “Wrap around contact for nanosheet source drain epitaxy,” U.S. Patent 11 302 813 B2, Apr. 12, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US11302813B2/en

[23] J. A. Kittl, J. G. Hong, D. R. Palle, and M. S. Rodder, “Methods for varied strain on nano-scale field effect transistor devices,” U.S. Patent 9 978 833 B2, May 22, 2018. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US9978833B2/en

[24] S. Gandikota et al., “Method of reducing metal gate resistance for next generation NMOS device application,” Int. Patent Appl. WO 2024/137272 A1, Jun. 27, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/WO2024137272A1/en

[25] J. Zhang et al., “Full bottom dielectric isolation to enable stacked nanosheet transistor for low power and high performance applications,” in Proc. IEEE Int. Electron Devices Meeting (IEDM), Dec. 2019, pp. 11.6.1–11.6.4, doi: 10.1109/iedm19573.2019.8993490.

[26] C. Yoo, J. Chang, Y. Seon, H. Kim, and J. Jeon, “Analysis of self-heating effects in multi-nanosheet FET considering bottom isolation and package options,” IEEE Trans. Electron Devices, vol. 69, no. 3, pp. 1524–1531, Mar. 2022, doi: 10.1109/ted.2022.3141327.

[27] S. S. Pradhan, D. B. Bergstrom, J.-S. Chun, and J. Chiu, “Tungsten gates for non-planar transistors,” European Patent Appl. EP 3 506 367 A1, Jul. 3, 2019. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/EP3506367A1/en

[28] L. Schloss and X. Ba, “Tungsten films having low fluorine content,” U.S. Patent 9 754 824 B2, Sep. 5, 2017. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US9754824B2/en

[29] Intel Foundry, “Accelerating AI and HPC with advanced process and packaging technologies,” Doc. 367370-001US, Jun. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-11/intel-foundry-hpc-ai-brief.pdf

[30] M. Lofrano, X. Chang, H. Oprins, and Z. Tokei, “Mitigating the thermal bottleneck in advanced interconnects,” imec, Sep. 28, 2023. Accessed: Sep. 14, 2026. [Online]. Available: https://www.imec-int.com/en/articles/mitigating-thermal-bottleneck-advanced-interconnects

[31] S. Bair, “Intel at ISSCC 2025: Navid Shahriari invited talk, eight papers, forums, panelist & product details,” Intel Community, Feb. 19, 2025. Accessed: Sep. 14, 2026. [Online]. Available: https://community.intel.com/t5/Blogs/Tech-Innovation/Edge-5G/Intel-at-ISSCC-2025-Navid-Shahriari-Invited-Talk-Eight-Papers/post/1667592

[32] D. Chandra, E. Potladhurthi, D. R. S. Reddy, and K. S. Rengarajan, “Tunable negative bitline write assist and boost attenuation circuit,” U.S. Patent Appl. 20160203857 A1, Jul. 14, 2016. Accessed: Sep. 14, 2026. [Online]. Available: https://patents.google.com/patent/US20160203857A1/en

[33] Synopsys, “Advantages of firmware-based training in high-speed DDR IP,” Synopsys IP Technical Bulletin. Accessed: Sep. 14, 2026. [Online]. Available: https://www.synopsys.com/articles/firmware-based-training-ddr-ip.html

[34] N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “CACTI 6.0: A tool to understand large caches,” Hewlett-Packard Laboratories, 2009. Accessed: Sep. 14, 2026. [Online]. Available: https://users.cs.utah.edu/~rajeev/cacti6/cacti6-tr.pdf

[35] Intel, “Intel Tech Tour 2024 Lunar Lake AI hardware accelerators,” Doc. 824436, 2024. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/824436/2024_Intel_Tech%20Tour%20TW_Lunar%20Lake%20AI%20Hardware%20Accelerators.pdf

[36] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in Proc. 43rd ACM/IEEE Annu. Int. Symp. Comput. Archit. (ISCA), Jun. 2016, pp. 367–379, doi: 10.1109/isca.2016.40.

[37] Intel, “Introduction to the Xe-HPG architecture,” Doc. 758306, Nov. 4, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-the-xe-hpg-architecture.html

[38] Intel, “Core Ultra processors Series 3 for the edge,” Doc. 855291. Accessed: Sep. 14, 2026. [Online]. Available: https://cdrdv2-public.intel.com/855291/Intel%C2%AE%20Core%E2%84%A2%20Ultra%20Processors%20Series%203%20for%20Edge%20Overview_2.pdf

[39] Intel Foundry, “Foveros technology brief,” Doc. 366411-001US, Feb. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-07/foveros-25d-product-brief.pdf

[40] D. Das Sharma, “UCIe: Building an open chiplet ecosystem,” UCIe Consortium, 2022. Accessed: Sep. 14, 2026. [Online]. Available: https://www.uciexpress.org/_files/ugd/0c1418_c5970a68ab214ffc97fab16d11581449.pdf

[41] Intel, “Intel launches Intel Core Series 3 processors,” Intel Newsroom, Apr. 16, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/newsroom/news/client-computing/intel-launches-intel-core-series-3-processors-changing-the-game-for-everyday-computing.html

[42] Intel, “Core Series 3 launch press deck,” Apr. 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://download.intel.com/newsroom/2026/Intel-Core-Series-3/Intel-Core-Series-3-Launch-Press-Deck.pdf

[43] Intel, “Intel outlines architectures for agentic AI at Hot Chips 2026,” Intel Newsroom, Aug. 24, 2026. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intel.com/content/www/us/en/newsroom/news/client-computing/intel-outlines-architectures-for-agentic-ai-at-hot-chips-2026.html

[44] Intel, “Intel accelerates process and packaging innovations,” Intel Investor Relations, Jul. 26, 2021. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/filings-reports/all-sec-filings/content/0001193125-21-224438/d199788dex991.htm

[45] Intel, “Intel reports third-quarter 2021 financial results,” Intel Investor Relations, Oct. 21, 2021. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/news-events/press-releases/detail/1505/intel-reports-third-quarter-2021-financial-results

[46] Intel, “3rd generation Intel Core processors bring exciting new experiences and fun to the PC,” Intel Investor Relations, Apr. 23, 2012. Accessed: Sep. 14, 2026. [Online]. Available: https://www.intc.com/news-events/press-releases/detail/577/3rd-generation-intel-core-processors-bring-exciting

Extended carrier and bond stack detail

Read more

Every one of those models runs on a datacenter, and China has been building them at a pace that has gone largely unmeasured outside the country. The biggest tenant files no 10-K. Several of the largest landlords have never listed. Most of the primary sources are in Chinese. So the market settled on two lazy assumptions: China is big, and China is empty. Published estimates of China’s datacenter capacity differ by 15x, and reports keep citing high vacancy rates.

As with our flagship SemiAnalysis Datacenter Model, we use building-level data to show which of the most widely cited narratives hold, starting with how large the market is.

The global model tracks over 5,000 facilities across four regions. The US leads the world with 56GW of capacity as of 2026YE, followed by ~15GW for APAC ex-China, ~14GW for EMEA, and ~2GW for Latin America. Until now, it stopped at the border of China. Today we cross it.

Our tracking of 1,000+ datacenter facilities across over 60 players shows that China alone boasts a fleet of over 24GW. Bigger than EMEA. Bigger than the rest of Asia. This excludes ~20GW of dated pipeline and another ~30GW of announced projects.

Growth is at an inflection point. In 2Q26, the combined capex of Alibaba, Tencent, and Baidu (“BAT”) reached $20B, more than doubling YoY, and for the first time on record, all three posted negative free cash flow. This is the largest capex step-up in the sector’s history.

That total leaves out the largest spender of all, ByteDance, which remains private. According to our China Datacenter Model, ByteDance alone occupies roughly a fifth of delivered datacenter capacity in China, and it rents nearly all of it, making it the single most important customer for all wholesale colocation players in the country.

The listed players get the attention, but they are only the visible tip of the iceberg. This is also true for datacenter landlords. GDS and VNET, the only two Chinese datacenter landlords listed in the US, signed 1.3GW of wholesale orders in 1H26. But according to our China Datacenter Model, they captured barely a third of ByteDance and Alibaba orders in 2024–2026YTD.

The buildout is a nationwide effort, and the state-owned carriers are also part of it. China’s datacenter market was historically telecom-dominated. The state carriers held the majority of the market share in the 2010s, and still own a third of national capacity today. The national power grid companies invest aggressively too. Their combined capex accelerated from 2024, ending the 14th Five-Year Plan 24% over the original blueprint. The 15th plan (2026–2030) layers another 40% on top, to over $746B (¥5T).

Largely free of power constraints, labor shortages, and public protests, China routinely delivers 100MW datacenter facilities in under 12 months. Modular DCs are the standard playbook, which Tencent deployed its third-gen modular design in 2014. What is old news in China is only now being adopted at scale in America, as we detailed in The Wild Wild West of LEGO Datacenters.

The growth does not stop at China’s border. Overseas leasing by Chinese hyperscalers is set to double from 2026 to 2029 and approach ~4GW of leased capacity in our SemiAnalysis Datacenter Model. That number still understates their offshore compute, because it excludes the hundreds of thousands of GPUs they rent from Western clouds, which we covered at length in How Oracle Is Winning the AI Compute Market.

However, the market also faces some dark clouds. Vacancy rates are high, and datacenter developers compete heavily on price. Chip supply is also constrained, due to export restrictions. But as explained below, neither has stopped AI datacenters from being built and filled at a remarkable speed.

In this report we cover, in order:

• Introduction to China’s datacenter industry: the world’s second-largest market, built retail-first, and why that history produced overbuild and high vacancy alongside an AI capacity shortage.

• The westward buildout: how the “Eastern Data, Western Compute” policy and hyperscale AI demand are moving the industry inland, at construction speeds and unit costs the West cannot match.

• Behind the paywall: we go tenant by tenant through ByteDance, Alibaba, Tencent, Baidu, and Huawei, from their AI and datacenter strategy to leasing and self-build footprints.

Introducing the SemiAnalysis China Datacenter Model

This report is a preview of the SemiAnalysis China Datacenter Model. It is the China counterpart to our global SemiAnalysis Datacenter Model, built to the same standard, bottom-up and building by building.

Contact our sales team at sales@semianalysis.com for more information.

What you’ll find inside:

• Tracking over 1,000 facilities across 60+ operators

• Quarterly delivery curves through 2032 for every facility, with construction status and timelines

• Tenant identification per building, supplemented by tenant allocations per customer-mix for leading operators

• Hyperscaler self-build vs leased splits and per-tenant capacity paths

• A lease signing tracker and a lease delivery tracker, on one reconciled basis

• Eastern Data, Western Compute (“EDWC”) hub and cluster analytics, load-growth and power-draw forecasts

• An interactive full map view showing where the capacity is concentrated

Consider it the opening chapter. As with our global Datacenter Model, we will keep adding layers (AI capacity, power, suppliers, supply/demand analysis) with every update.

Let’s dive in.

China’s datacenter industry: built retail-first, flipped by AI

One of America’s key advantages in the AI infrastructure race is that its datacenter industry grew up hyperscale. Multi-hundred-MW campuses have existed for years, purpose-built for a handful of enormous tenants. China’s industry was shaped by telecom carriers and thousands of small retail deployments. Understanding that history explains what’s strange about the market today, from overall high vacancy coexisting with an AI capacity shortage, to why the carriers still own a third of everything.

The following chart previews the whole story. Between 2010 and 2022, a leading colocation player in China watched its utilization rate shrink from the mid-70% range to the mid-50% range. Then, from 2023 onwards, its segment disclosure reveals a two-speed market. Wholesale buildings are filled fast on AI demand, back above 70% utilization, while the legacy retail racks sit near 60%.

Era 1 — Carrier hosting (pre-2015): datacenters as telecom real estate

Unlike the rest of the world, where hyperscalers and colos dominate, China’s datacenter market was historically telecom-dominated. The three state carriers (China Mobile, China Telecom, China Unicom) held a combined 60–70% market share, with GDS and VNET, the largest third parties, below 5% each. The carriers enjoyed structural advantages in licensing and networking and benefited from captive demand. A large share of Chinese datacenter demand comes from government, SOEs and regulated industries, which default to state-owned suppliers.

For its first two decades, a Chinese datacenter was essentially telecom real estate, featuring low-density retail racks (~90% of facilities ran below 2kW) rented to websites, online gaming and CDN customers. This was often offered as a bundled product with bandwidth, served by small urban switching centers around Beijing, Shanghai and the Pearl River Delta. Third-party operators existed largely as resellers of carrier space and bandwidth. The hyperscalers were barely a factor, as Alibaba Cloud was launched in 2009 and Tencent Cloud only opened to the public in 2013.

This era matters today because its stock is precisely the capacity now sitting vacant. The legacy retail racks no longer match what an H20 or Ascend server demands, and it’s impractical to retrofit given partial occupancy and high cost. The headline utilization number of ~50% is largely a monument to Era 1 (and Era 3, as we’ll see).

Era 2 — The cloud land grab (2015–2021): growth at any price

The cloud era began around 2015. The two leading players, Alibaba and Tencent, concluded that it was a winner-take-all market and opened a capex race. Alibaba Cloud revenue compounded at a ~110% CAGR across 2015–2019, at the cost of a brutal price war and EBITA losses. Its core products were marked down more than 50% after 17 consecutive price cuts during a 12-month window. Tencent responded in kind, driving combined hyperscaler capex to a 2021 peak.

Third-party colocation platforms (GDS, VNET, Chindata) scaled alongside them. Retail developers, meanwhile, kept building small urban sites on the assumption that cloud demand would spill over indefinitely.

Era 3 — Digestion (2022–2023): the music stops

The wild expansion stopped abruptly in 2022. Policy headwinds, macro slowdown, and share gains by state-owned clouds pushed Alibaba Cloud growth from +50% in 2021 to +23% in 2022, then to effectively zero. Tencent management framed it as a repositioning “from pursuing revenue growth to healthy growth” and saw a similar slowdown. Tencent cut capex budgets by nearly half in the middle of 2022. Alibaba did so in 2023.

Almost simultaneously, in February 2022, the National Development and Reform Commission (“NDRC”) launched Eastern Data, Western Compute guidelines and named ten computing clusters within eight national hubs. These regions, especially the western ones, enjoy fast track permitting and cheap green power.

The result was an oversupply despite a slowdown in delivery. Pricing was hit most. Power-exclusive market rates were cut in half from ~$80/kW/month. On the August 2026 earnings call, GDS management estimated another 18 months to fully digest the price reset, even though it has been going on for a few years.

Era 4 — The AI supercycle (2024–): full steam ahead

DeepSeek and the hyperscalers’ own GenAI ambitions woke the market up over the course of 2024-2025. Combined BATB (ByteDance/Alibaba/Tencent/Baidu) capex went from roughly $35B in 2024 to over $50B in 2025 and is on a trajectory to $100B for 2026.

Wholesale orders returned at a scale never seen before, and the western hubs finally found their tenant. GDS and VNET together signed roughly 1.3GW of wholesale capacity in 1H26, with less than 10MW of retail orders. The AI era is wholesale. The legacy retail stock is not participating.

But watching only these two listed names badly understates total demand, as they only capture ~1/3 of ByteDance and Alibaba orders in 2024–2026YTD per our China Datacenter Model.

To recap, the four-stage history explains China’s datacenter market today: (1) a retail-focused legacy stock around Tier-1 cities, running 30-50% utilization, often physically unsuitable for AI densities; (2) a wholesale AI-driven buildout happening somewhere else entirely; and (3) a carrier share in structural decline, as hyperscaler self-build and third-party wholesale take over the growth.

Eastern Data, Western Compute: the pipeline is heading west

The EDWC policy designates eight national computing hubs, implemented as ten clusters, spanning both the demand centers (Beijing-Tianjin-Hebei, Yangtze River Delta, Greater Bay Area, Chengdu-Chongqing) and the resource-rich west (Inner Mongolia, Ningxia, Gansu, Guizhou).

This is the industry’s top-level design blueprint, and unlike most “guidance” documents, it actually changed how the industry develops.

The policy itself has been covered extensively. What is missing, and what matters, is the transmission mechanism.

In China, the first permit a datacenter developer applies for after securing land is an energy-consumption review filed with the local Development and Reform Commission. That quota is the binding cap on how large a facility can be. Any project consuming over 10,000 tons of coal equivalent (>10MW) cannot legally begin construction without passing this review.

The review regime dates to 2010 and became a rationing tool over time. When EDWC arrived in 2022, the quota system became the most effective mechanism because the same offices that issue quotas are the ones implementing the EDWC policy, so quotas effectively stopped construction flowing to non-hub locations.

Guangdong is the cleanest example. The province stopped issuing datacenter energy quotas in 2021-2022 and reopened applications only for the EDWC hub region in Shaoguan. Similar quota control governs the other eastern metros. Meanwhile, EDWC regions hand out energy quotas, land approvals and tax incentives generously.

Despite the mechanism, for the first two years, migration was slow because the demand of the day (cloud, gaming, e-commerce) required low latency to population, so hyperscalers kept fighting for scarce quota near Tier-1 cities.

A little-noticed episode illustrates how the quota system played out in practice. In mid-2023, Guangdong’s Energy Bureau inspected Tencent’s two self-built campuses in Qingyuan and found each running at roughly 13–14x its approved scale. The projects were labeled “approved small, built big”, and ordered rectification within six months. The local grid company was instructed to hold power supply to the approved quota. Even for Tencent, incremental builds now belong to the province’s designated hub node.

Today, the campuses keep operating, but as our China Datacenter Model shows, Tencent’s subsequent capacity went to Shaoguan. The reason is not simply regulatory pressure.

Two other things changed the picture.

First, telecom-led fiber investments have significantly reduced latency. Shaoguan again is the model case. Since 2022, the local government has invested ~$600M (~¥4B) in network infrastructure, building 13 ultra-high-speed 400G all-optical routes covering the Greater Bay Area, achieving 1.3ms one-way latency to Guangzhou, 1.66ms to Shenzhen, and under 3ms to Hong Kong. A direct Shenzhen-Guangzhou-Shaoguan hollow-core fiber corridor has been planned to cut latency a further ~30% from 2027 onwards. At those numbers, a Shaoguan rack is functionally a Shenzhen rack for all but the most latency-critical workloads.

Second, AI demand did the rest. Training workloads don’t care about latency, but they care about power prices.

Inner Mongolia is the best example here. The region has China’s largest renewable fleet and, critically, western Inner Mongolia is served by the Mengxi grid, China’s only provincial grid outside State Grid and Southern Grid, with its own multilateral trading mechanism and independent pricing.

As a result, Inner Mongolia has emerged as China’s Johor, the default destination for hyperscale AI capacity, with generous quota allocations, land measured in square kilometers, and, most importantly, power prices at roughly half the Tier-1 city level.

The Johor parallel runs deeper than land and power. On economics, Chinese rents are far lower, but so is the capex, while Johor rents are higher, but so is the cost to build. Our data shows that the two markets' yield-on-cost ranges overlap, with the best Inner Mongolia campuses underwriting to the return of Johor's international developers. However, the trends diverge. Inner Mongolia construction costs keep falling as the prefab approach and domestic substitution compound, while Johor developers face capex inflation and overrun risk in a supply chain running hot. Chinese operators building in Johor enjoy geographical arbitrage leveraging the Chinese supply chain and earn higher yields.

Zoom out from Inner Mongolia, though, and the ten clusters tell very different stories. It is easiest to see when the map is weighted by what is actually live and what is committed, rather than by what the policy designated, as the map below shows.

A recurring media take holds that EDWC is a boondoggle of empty western shells. The simplest rebuttal is to look at where the hyperscalers put their own money. As our tracking shows, ByteDance currently operates three self-build clusters: Datong in Shanxi (its earliest), Wuhu in Anhui (an EDWC cluster, serving east-China inference) and a twin-site Inner Mongolia program (Horinger and Ulanqab), with two Ningxia entities registered in Zhongwei (another EDWC region) in July 2026 pointing to a fourth.

Outside of Inner Mongolia, development across the clusters is sharply divergent, but much of that divergence is by design. Gui’an was positioned from the start as a storage hub, as cold data doesn’t mind the mountains. The most famous anchor tenant is Apple’s iCloud China datacenter, operated with Guizhou-Cloud. Less reported is that Huawei and Tencent both have self-build clusters in this region.

Chengdu and Chongqing are different stories. They are pitched as national compute hubs but lost the AI-demand contest to Inner Mongolia on power price and grid flexibility. EDWC succeeded in concentrating the buildout inside the map, and the market then picked the winners within it.

Shanxi is the opposite. It is the largest datacenter region outside the EDWC hub system entirely (the red diamond on the map above), but attracted close to 2GW commitment from ByteDance, Baidu, and JD across self-build sites and colocation leases. The reasons are abundant cheap power (Shanxi is China’s coal province), proximity to Beijing, and first mover advantage. Hyperscalers went there before EDWC existed and kept building out, proof that the quota regime channels the buildout, but electrons and geography still make the site selection.

Speed and cost: a different sport

The US buildout is power-gated with interconnection queues, transformer lead times, behind-the-meter workarounds, which we have covered at length here. China’s is chip-gated: generation is abundant, state-owned utilities treat datacenter load as a development win, permitting is a policy funnel rather than a bottleneck, and construction labor is plentiful. With no physical friction on the facility side, the build cycle is driven purely by internet platform demand.

On speed, the standard delivery time for a 100MW facility has compressed from ~18 months in the cloud era to ~12 months in the AI era, with state-of-the-art projects finishing faster still.

For reference, our US modular deep dive The Wild Wild West of LEGO Datacenters found the best-case full-modular buildout in the US still takes 12–18 months of construction, sitting behind 12–13 months of permitting that cannot overlap. For China, the permitting period before construction is typically 3–6 months, and faster in EDWC regions.

Several structural factors contribute to the overall faster speed.

First, speculative shells ahead of orders. Because hyperscalers now run T+6-month delivery requirements in their tenders, colocation developers pre-build civil works at their own risk, making the tenant competition effectively a race among half-finished shells. When GDS signed a 152MW AI mega-order in early 2025, the customer required delivery within six months and got it.

Second, building greenfield sites in western EDWC hubs changed the construction method. Away from Tier-1 land constraints, developers are moving from conventional multi-story concrete to low-rise, long-span steel halls. The steel is made in a factory while foundations are dug on site, then trucked in and lifted into place by crane. Concrete is used only in the ground, so there is no waiting for it to harden before the next floor goes up. VNET’s Ulanqab footage shows a hall go from bare foundations to a finished building within one summer-to-winter season.

Tencent’s T-Block takes the same logic to its extreme. The entire datacenter arrives as factory-built modules, with the “building” shrinking to a leveled site and a simple steel shell, sometimes no building at all.

Third, prefabrication turns a serial schedule into a parallel one. The bulk of MEP work moves off-site into factories, so electrical and mechanical fit-out no longer waits for the building. The headline example is Alibaba’s CUBE 5.0 “100-day datacenter”, comprising 30 days of factory production and site prep, 50 days of on-site installation, and 20 days of commissioning, measured from a finished shell to the day servers arrive. This roughly halves M&E and commissioning stage compared with market standard 6-month, and brings ground-break to commissioning down to roughly 7–9 months.

Every factor above points in the same direction, which is more factory, less site. Follow that logic to its endpoint, and you would expect Tencent-style full-modular container campuses to be the national standard by now. They aren’t, for two very Chinese reasons.

First, the labor math is different. Modularization’s biggest payoff in the West is dodging scarce, expensive skilled trades. Chinese site labor is affordable and abundant, so a traditional steel-and-prefab build already hits aggressive schedules. The marginal speed gain from going full-container is small.

Second, the financing dynamics play a role. Chinese colocation developers pledge their datacenters to banks as loan collateral. A proper building with a 20-year design life appraises well, but a yard of containers appraises poorly and carries lower residual value. When your business model is “borrow against the asset, lease it to ByteDance,” you build a real building.

Today, full-modular builds remain limited to hyperscaler self-builds and Southeast Asia projects, but overall construction speed is as fast as needed.

On cost, capex keeps falling even as speed rises. Continued supply-chain compression and rising domestic content have pushed shell-and-fit-out costs to a fraction of comparable US wholesale costs, as we show in our model. Alibaba credits its modular approach with a further >10% cost reduction compared to its earlier version on top of the schedule gains.

The demand: five hyperscalers, multiple playbooks

The demand side of China’s AI buildout is concentrated in a handful of names: ByteDance, Alibaba, Tencent, Baidu, and Huawei.

Like their American counterparts, they span social media, e-commerce, search, gaming and short video. However, the consumer internet that built these companies has run out of new users. Today, China’s internet penetration already sits at ~80% and grows only ~1.5% a year.

The growth engines of the past are all in single digits.

• Social media: Tencent’s core asset, WeChat, grew MAU just 2% YoY in 2025.

• Search engine: Baidu’s 2025 revenue fell 3% with online marketing down double digits.

• E-commerce: Alibaba’s customer management revenue fell 7% YoY in the quarter ended June 2026.

• Short-video: Kuaishou’s 2025 DAU grew just 2.7% YoY.

Outside of consumer apps, the same titans lead the cloud industry, with a structural handicap. The market is dominated by Alibaba, Huawei and Tencent alongside the three telecom carriers. But unlike the US hyperscalers, which generate roughly a third or more of their cloud revenue outside the US, Chinese cloud revenue is largely domestic. Alibaba Cloud, the most global of the group, still earns only ~20% of cloud revenue overseas, and the carrier clouds are effectively 100% domestic. In terms of “addressable GDP,” the core market of the US tech titans is simply much larger.

Against that backdrop, AI has become the growth engine, which drives the capex inflection point in 2Q26 we mentioned earlier.

Beyond funding their own model R&D, the big platforms have built an investment web across China’s GenAI labs. Alibaba invested in five LLM unicorns. Tencent backed six and wrote the largest external ticket in DeepSeek’s Series A.

The labs, in turn, train and serve on the hyperscalers’ clouds (in the case of the Alibaba/Moonshot relationship) or have their models served inside the hyperscalers’ products (in the case of the Tencent/DeepSeek relationship). Either way, the hyperscaler datacenter expansion is the compute foundation for the entire domestic AI ecosystem. Hence the deep dives that follow.

We provide a snapshot below showing where the hyperscalers stand:

The leasing: Today, ByteDance and Alibaba each issue annual leasing demand at gigawatt scale and both are actively expanding overseas. ByteDance's is the largest and leases majority of its footprint, so its tender calendar effectively sets the order book for every wholesale operator in the country. Alibaba, as we show below, splits the demand across self-build, BOT and wholesale. Tencent is the most disciplined. It spent the 2022–23 downcycle closing small leased metro sites and consolidating into its own campuses, before leasing picked up again in the AI era. Baidu tenders at a fraction of the others’ scale.

The self-builds: In the 2010s, Chinese hyperscalers began self-building datacenters as their businesses reached sufficient scale, and the strategies have diverged since. Alibaba, Tencent, and Baidu self-build their strategic campuses and lease the rest. Huawei self-builds almost everything because of its business model. ByteDance, the primary tenant, is now ramping its own campuses fast. Our China Datacenter Model tracks all self-builds, and the map below shows the current landscape.

Between pure leasing and pure self-build, a hybrid structure called BOT also emerged. The real anatomy is a two-layer ownership split. The hyperscaler owns the land and the civil shell, while the colocation partner invests in, owns and operates the M&E layer and recovers its capital through 8–10-year service contracts. In this way, the hyperscaler keeps control and speed, the partner’s balance sheet carries the capex, and the IRR is contracted rather than merchant. All three of BAT run some version of this playbook. Alibaba partners with AtHub, Tencent with Kehua Data, and Baidu with Aofei. Alibaba takes the mixing furthest, running self-build, BOT, and wholesale leases side by side within the same cluster, as we discuss below.

ByteDance: TikTok parent, China’s largest tenant, among the world’s largest GPU users

Read more

In 8 months since our last major release of ClusterMAX, slavering investors have just about run out of pockets to stuff checks into. GPU supply has gone to zero. Meanwhile, we have been hard at work putting clusters through the ringer.

Weeks ago, we teased this report with some R-rated anecdotes from our experiences probing the security practices of neoclouds, eliciting a PSA from a neocloud customer that you may have heard of.

Today, we finally go through the full breadth and depth of our testing, more thorough in ClusterMAX 3.0 than ever before. This includes compute, networking, storage, orchestration, UI, monitoring, support, and just about everything that you could think to check on a GPU cloud. We’ll explain who’s been cutting deals, whose engineers have been hard at work, and whose clusters you should negotiate to come with a bottle of Aspirin.

Without further ado, voilà the ClusterMAX 3.0 podium:

Results (Executive Summary)

Podcast summary discussion available on YouTube now!

• ClusterMAX 3.0 debuts with a comprehensive review of the neocloud industry, covering 77 providers.

• We increase our market view to cover 323 providers, up from 209 in ClusterMAX 2.0, 169 in ClusterMAX 1.0, and 124 in the original AI Neocloud Playbook and Anatomy article.

• We have now interviewed well over 200 end users of neoclouds as part of this research.

• We update our itemized list of criteria across 10 categories, and update our direct descriptions of our expectations for Slurm, Kubernetes, Standalone Machines, Monitoring Dashboards, and Health Checks. All of this content is live on our website. We encourage providers to use these lists when developing their offerings. We still consider these lists as an amalgamation of our experience interviewing end users, making them representative of the features that end users expect from their cloud providers.

• Nebius joins CoreWeave in the Platinum tier. While CoreWeave still sets the technical bar for others to follow, Nebius is now established as a provider that consistently commands a premium pricing over others. Strong business decisions by Nebius have put them in a position to serve an entire class of neolabs at seller’s prices.

• Google Cloud joins Oracle in the Gold tier. Azure moves to Silver, Fluidstack moves to Unavailable, and Crusoe drops to Bronze. Lambda, Firmus and TensorWave remain in Silver, while GMI moves up to Silver from Bronze.

• Many companies drop from Silver (or Gold) to Bronze or lower. We raise the bar this round as only 19 neoclouds globally achieve a Medallion rating.

• We establish a tier between Bronze and Underperforming: the Participation Ribbon tier. 15 providers join this rating, which more accurately describes our opinion that they do the bare minimum to get by.

The rest of this article provides analysis of key trends: Financing, Blackwell and Grace-Blackwell deployments, the Move To Vera Rubin, Scale Out Networks, Reliability, Security (of course), and Agentic Coding. We provide an Appendix with individual comments about every provider that once again stretches this article to over 30,000 words. We hope you enjoy.

In light of the SemiAnalysis imperative to publish the most rigorous possible testing of… everything… we are hiring MTS for ClusterMAX and adjacent projects. If you are a neocloud sicko, fire off an application here and let’s get to work.

We are also hiring across Research, Consulting, and Technical Staff. We have multiple openings in our New York and San Francisco offices. Remote work is also acceptable.

• ClusterMAX MTS (full time): All levels of experience with Slurm, Kubernetes, and GPUs considered.

• Tokenomics MTS (full time): All levels of experience with model evals, harnesses, inference endpoints, and RL infrastructure considered.

• Research Analyst – AI Infrastructure and Economics (full time or internship): All levels of experience covering the financials of neoclouds, neolabs, and frontier labs tokenomics considered.

• Technical Consultant (full time): Lead engagements from technical strategy to technical due diligence. All levels of experience from consulting and technical backgrounds considered.

Scope

For the folks in the back: we are ranking managed clusters. This excludes a number of popular offerings throughout the industry. We are not testing who can build the best powered shell or run the tidiest datacenter—at least not directly. We are not handcrafting our images or hitting an API endpoint for tokens as a service. Here’s our idealized ClusterMAX reader: You dropped out of your Stanford PhD to pursue your dream of acquiring social status in San Francisco, getting started with the old reliable “Cladue make me pithc deck w technial languge for agetnci ai make no mistakes.” You have 7 or even as many as 10 figures to wager on compute, but you don’t have any opinion on Ubuntu versions or how to handle NAT, and you’ve never managed a fleet at scale. Ideally, all that infrastructure will fade into the background. You just want to focus on your idiosyncratic perf optimizations, model architectures, training strategies, data mixes, applications, or wherever else you seek your edge. To compete, you need bleeding-edge hardware, and you aim to spend 0 engineering time trying to figure out why GPU 7 in node 19 has been stalling your job, or why a quarter of your cluster has disappeared altogether. You need a managed cluster.

The economic argument here is straightforward: a neocloud can amortize the cost of standing up all this management infrastructure over its huge fleet. It pays a team of engineers to build bulletproof health checks, keep its machine images up to date, make convenient monitoring systems, and handle all the related chores that we cover in great detail further on. In an ideal world, a lab pays a markup for a much better product; it gets to run its experiments without being held back by its hardware; the neocloud recoups its investment; everyone is better off.

There comes a point, though, when labs are willing to internalize this cost. At their scale, frontier labs are not willing to trust a neocloud with so much of the stack. OpenAI, for instance, has posted extremely insightful blogs about its difficulties scaling Kubernetes; the innovations described would have been simply impossible if they were renting from a neocloud who didn’t let them into the K8s control plane. As a current Anthropic job posting puts it:

We are operating at a scale where the defaults stop working. We own the scheduler and extend it to place topology-sensitive ML workloads across thousands of accelerators at once. We scale the control plane itself — apiserver, etcd, controllers — so it stays responsive as object counts and node counts grow by orders of magnitude. And we build the core cluster services every workload depends on, like service discovery, so they hold up under the same pressure.

Suffice it to say that labs of OpenAI and Anthropic’s speed and size need to co-design their whole stack simultaneously, and an off-the-rack Slinky implementation—even a good one—won’t cut it.

The obvious care that frontier labs take in managing their infrastructure goes to show why ClusterMAX matters. The sum of the salary ranges for OpenAI and Anthropic open positions with “Kubernetes” in their descriptions is, last we checked, $27,769,274-$42,687,654, not to mention the salaries they pay to the huge teams they already staff. If you’re not anticipating a trillion-dollar liquidity event and you need your infra to be good enough to give you a fighting chance, you need a neocloud to do the heavy lifting for you.

At least on paper, smaller neocloud customers are David fighting Goliath. OpenAI and Anthropic have remarkable growth rates on their respective logistic exponential curves, and our Datacenter Industry Model currently models them at 56.3% of all lab compute YE2027.

As we will describe in more detail later, there is also a Matthew principle in effect here, due to financing: the profitable frontier labs have an easier time locking down capacity than smaller ones, which get stuck with fat prepays, worse prices, and a much harder time planning years in advance.

Does this mean that managed clusters are dead? cLUsTerMaX iS wAshED? In truth, although Anthropic and OpenAI are taking larger slices by the year, managed clusters as a business are still growing exponentially. We’ve helped countless labs find compute, countless more are on the hunt, willing to pay, and waiting for more capacity to come online. Anthropic and OpenAI are the biggest customers, but there is a long tail of mere multi-hundred-million-dollar deals whose aggregate value is enormous.

One plausible scenario in which managed clusters gain ground on frontier-lab bare metal involves open-source progress. Serving open-source models is already more lucrative than many realize; we recently modeled that “you can make over $100M per MW per year selling open source tokens,” leaving plenty of room for profit even at today’s rental prices. One could imagine open-source models increasing their market share, whether the marginal adopter is more cost-sensitive or algorithmic progress closes the gap to closed-source. Regardless of the dynamics, this would increase the relative standing of managed clusters. Just about any supply-side fracture would make labs marginally less willing to handle the infra, and marginally more willing to rely on neoclouds.

The layers on top of these clusters are rapidly maturing, but they lie outside the scope of ClusterMAX. There’s a handful of niches to consider. There are inference endpoints, which can be either public or private, and bill by the token. There are post-training services, which customize open-source models for specific use-cases. And there are sandbox services, which provide the infrastructure for agents, especially during RL rollouts, and simplify and optimize container and CPU management. These all fit loosely in the category “GPU services” and the border between them can be blurry. For example, a cloud might spin up an inference endpoint with spare capacity after it’s sold managed clusters, or it might provide sandbox infrastructure and also consume it internally as it sells RLaaS. We will be publishing much more about this in the near future, but it is beyond the scope of this report.

Testing Methodology

For ClusterMAX 3.0, we requested the following from every provider:

• 32 GPUs (4 nodes of 8-way HGX, or 8 nodes of 4-way NVL72)

• High bandwidth network (we asked for 800G RoCE or XDR InfiniBand, but many providers did not have it yet)

• 10TB+ of high performance file storage (supporting NFS/POSIX mounts and a RWX StorageClass via csi driver)

• 10TB+ of S3-compliant object storage

• A monitoring dashboard (usually based on Grafana)

• 5 days to test Slurm, and 5 days to test K8s (can be done in parallel, on a SonK cluster, or sequentially, if the provider wanted to take the nodes back and re-provision them)

• Blackwell, not Hopper (B200, B300, GB200, GB300) for NVIDIA, or MI355X for AMD are considered acceptable. H100 is now over 4 years old!

We tested the cluster through 3 phases, described below. Of course, we always complement this testing by contacting customers of all these providers and taking their feedback into account.

Phase 1: Audit

First, the Audit covers yes/no questions. Is the cluster setup properly? Is software installed? Is it up to date? Do utilities work as expected? It takes about 15 minutes to run. It’s available on GitHub or via {uv} pip install clustermax.

The audit checks the configuration before we put it under load. It covers hardware inventory, software and firmware versions, GPU access, containers, scheduler configuration, networking, storage, health monitoring, and security. We check which components apply to the environment we’re testing, and display pass, warn, fail, and skipped for all our checks. We have released this part for free and will maintain it over time. The rest we keep internally for now.

Phase 2: Performance

Performance puts a pass/fail threshold on the performance characteristics of both the individual components of the cluster and the cluster as a whole. These are done through microbenchmarks, and real-world benchmarks.

Specifically, we test:

• GPU Compute

• Networking

• Storage

• Lifecycle

• Training

• Inference

Below we explain this in more detail, but keep in mind that what we test evolves over time.

GPU Compute

We start by checking the real-world GEMM performance on the GPUs. GEMMs are the most critical operation in modern AI workloads, which we have explained many times before, with Grouped GEMMs being particularly important for modern MoE models.

We test Grouped GEMMs on cuBLASLt (or hipBLASLt) and DeepGEMM across many precision types: BF16, FP16, TF32, FP32, FP8 E4M3, MXFP8, and NVFP4, depending on what is supported by the chips in the cluster. We use a common set of shapes for different gate_up and down projections at varying batch sizes in the Kimi K2.5, K3 and DeepSeek V3, V4 Pro models.

We then test GEMMs, GEMV bandwidth, and MAMF (Maximum Achievable Matmul FLOPS, from Stas Bekman), which runs a sweep for a while on each GPU, per precision, using cuBLASLt (or rocBLAS). We report FLOPs for all of our GEMM tests.

We also test memory, through a GEMV test, and `nvbandwidth`. GEMV measures matrix-vector multiplication with a skinny vector, reporting GB/s as a proxy for the streaming behavior of low-batch decoding. NVIDIA’s `nvbandwidth` utility is built into DCGM health checks, and measures h2d (CPU-to-GPU), d2d (GPU-to-GPU), and d2h (GPU-to-CPU) bandwidth. Finally, we also have a custom PyTorch script that measures the same paths through actual tensor copies, recording bandwidth for pinned h2d, d2h, and local HBM copies.

Last, we test HPL-MxP for the boomers. HPL-MxP solves a large dense linear system using low-precision factorization, which means we hit compute, memory, and communication together. The test covers compute, HBM, MPI broadcasts over the scale-out network, NVLink, and numerical correctness in a single workload. We report FLOPs across a range of number formats depending on the chip type.

Although GEMMs are the heart of modern AI workloads, these microbenchmarks very rarely distinguish providers—it would be hard for a provider to mess up this functionality. It’s important to check raw GEMM perf is during an extended burn-in, as described later, when a chip’s clock speeds might drop to avoid overheating if it is not properly cooled. On a GEMM burst, you’re unlikely to find much of note. We run these tests on every provider because it’s quick and because a cluster is immediately unusable if they fail, but we worry more about other tests.

Lifecycle

In order to understand the ease of use of the cluster throughout its lifecycle, we have a set of contrived tests that we use. We start by checking internet speed using multiple approaches for download/upload speed, mostly from Cloudflare. We then test the time it takes to install common packages like PyTorch and vLLM via pip and uv, as well as download containers from Docker Hub, ghcr.io, and nvcr.io, and pull a model from Hugging Face and ModelScope.

We then redo the pip and uv install tests with fresh package caches on each available storage tier: home, shared filesystem, local NVMe scratch, and any additional mounts (referred to as `/home`, `/data`, `/scratch`, and other). We then run `import torch` from each location to see how long that takes. On most providers, this takes 1-2 seconds, but on some filesystems with LOSF performance issues, it can somehow take 10-20 seconds.

Finally, we take the models downloaded earlier, and run a vllm serve command to test the time until we can hit /v1/models. Again, while certain providers can load a small model into GPU memory from shared storage in 10-40 seconds, some providers take over 2 minutes.

Networking

We benchmark a wide suite of communication performance for inter-node and intra-node transfers. This includes MPI collectives, running NCCL or RCCL tests launched through MPI, measuring collective bandwidth and latency across message sizes from 8 B to 16 GB for all-to-all, all-reduce, all-gather, and other collectives. The same workloads are also run through torch.distributed in PyTorch. In addition, we use RDMA perftest, specifically ib_write_bw and ib_read_bw, we measure point-to-point bandwidth on each network rail, along with host-memory fallback (via PCIe), to identify any issues.

We grow the world size in factors of 2 until we’ve used our whole cluster, keeping track of how performance scales. We also run tests with NVLink disabled, particularly on GB200 or GB300 NVL72 clusters, in order to isolate the performance of the scale-out network. It’s important to be careful in your accounting here, since world size, algorithm, and job placement relative to scale-out boundary materially affect results.

Storage

We run fio, ior, Elbencho, a custom checkpoint save/load benchmark written in Torch and Torch DCP, and a custom datagen benchmarked shared with us by Skild AI. Datagen is probably the most interesting, as it approximates their real workload in a robotics data pipeline: aggregate write throughput generating the video (MP4) and parquet sensor dataset.

The test that finds the most issues, though, is good old-fashioned fio. We sweep sequential/random reads/writes with buffered/direct I/O with 1MiB sequential blocks and 4 KiB buffered / 64 KiB direct random blocks from c1 all the way up to c32, or, if we have more than 4 nodes, as many clients as the cluster can tolerate. If storage is misconfigured in any way, there is almost always some setting in this fio sweep that shows poor perf. We track throughput, IOPS, and latency, and of course we also discover inconsistencies in metadata if the test fails with too many clients.

Of course, this significantly depends on how big the volume we’ve been given is and how the SLO is defined. Take these results with a grain of salt—the chart above really only shows that Azure gave us a PB-scale mount. This grain of salt also goes to show why having data on lots of clusters is important. Buffered tests, for example, have higher latency, as you might assume, and can also produce very noisy results if the test is not run for long enough. But how much higher latency, and how noisy? To know the difference between what’s “good enough” and what needs to be flagged to a provider, it’s crucial to compare to a big peer set.

For object storage, we run this suite again against an S3 bucket. We also run a custom benchmark that measures large object throughput on sequential read/write, small object PUT and GET rates, and read latency (p50, p90, p99). In this testing, we keep everything consistent. As always, though, if we notice a cluster struggles with a certain workload, we linger, running more tests to more precisely isolate the issue.

Training

Moving into the real world, we start to see how issues on the network or storage reveal themselves in real testing. We run two jobs via torchtitan: llama 3.1 8B pretraining on the C4 dataset with FSDP, and GPT-OSS MoE training via torchtitan with expert parallelism. The former is FLOPs bound, the latter is collective bound (unless you’re on NVL72). So, the latter makes it plain to see when something is wrong with the network.

Inference

We run the InferenceX AgentX benchmark in single node and multinode scenarios with different models to find compute-bound, memory-bound and comms-bound regimes with profiling traces on, treating the InferenceX results as the benchmark performance result to compare to. As with our training tests, this highlights the importance of the various cluster components covered in the microbenchmarks. If a cluster can’t survive NCCL tests, it certainly won’t be able to produce many tokens, and our InferenceX tests show how goodput is lost.

Phase 3: Reliability

Just because we see solid performance during a benchmark does not mean that the provider can drive solid goodput, and it certainly doesn’t mean they will provide a nice support experience. So, we test reliability, monitoring and autoremediation in multiple ways.

Burn-in

We start with an 8-hour burn-in on the GPUs and network, by running large GEMMs on the tensor cores and all-to-all communications at the same time, with a monitoring script tracking temperature, power, clock frequency, FLOPs, network connectivity, latency, bandwidth, and of course tailing the kernel ring buffer for any errors. You would be shocked how many hardware issues we induce with just this simple test. We cannot emphasize strongly enough how important it is to stress the GPUs and the network at the same time. Burn-in requires simultaneous thermal expansion and contraction of both of these components to approximate the real behaviour of these systems under real stress from real workloads. Lots of providers still use scripts that burn the GPUs and network separately.

Fabric

Next, we test the sustained performance of the scale-out and scale-up network fabric by looping through all-to-all, all-gather, and reduce-scatter of large message sizes. We report average, minimum, and maximum bandwidth, plus collective errors. On the frontend network, we also test management-network packet loss, RTT, and TCP bandwidth between every directed pair across four nodes. It’s not often that this extra test catches errors that the burn-in did not, but we like the data, and it has revealed big performance differences on certain Ethernet networks under load.

Storage

Continuing on, we test filesystem endurance by running 7.5 minutes of sequential mixed reads and writes, followed by 7.5 minutes of random mixed reads and writes, at each client count, using fio (as described previously). We increase clients across the allocation and report bandwidth over time, IOPS, tail latency, and performance degradation. Interestingly, we have seen some providers have bandwidth degradation of almost 40% under load, when compared to the initial benchmark.

When object storage is available, we also run a 15 minute mixed PUT, GET, LIST, and DELETE workload, measuring throughput degradation, p99 GET time-to-first-byte drift, throttling, and errors. We’ve seen about a 10% difference in these tests.

Orchestration

On Kubernetes, we test “churn” by repeatedly creating/deleting GPU pods and checking the API for GPU allocation. We report both scheduling and startup latency, and report failures when kubectl delete po gets stuck. We also increase batches of CPU-only sandbox pods, measuring how many reach Ready, how long they take, and why the rest remain Pending.

Last, we test the PVC lifecycle by repeatedly starting a pod that mounts a volume, writes and fsyncs a file, then gets deleted. We wait for teardown between cycles and use a warmup cycle to remove the initial image pull from the measured runs.

Destructive

Finally, we get to the fun part: breaking things on purpose.

First, we reboot all the nodes in the cluster. You would hope that this is not a destructive test, but it is. On Kubernetes, we cordon and drain the node, issue the reboot, and require a changed boot ID, then wait for it to return to the cluster. The timer stops when we can run nvidia-smi in a fresh pod on the node. With Slurm, we just get an allocation or SSH and run sudo reboot. Slurm-on-Kubernetes is a little different, so we attempt to do things from the Kubernetes layer to test it correctly. Our scripts time everything throughout.

For injecting failures, we start by using the DCGM injection method. If it doesn’t work properly, we follow up by writing a synthetic NVIDIA XID or SXID message into the kernel log, and watch how the provider’s existing health checks and scheduler respond. We leave their health agents and drain automation alone, so it’s up to their tooling to detect the error and act on it. Most health checks are set up to read from the kernel ring buffer, but some are not, so we communicate with the provider to make sure that our trigger will work and that we have access to pull it. As a final, trusty method, we reset the GPU’s upstream PCIe bridge, which produces a genuine XID 79.

In all these cases, we time how long it takes to detect the failure, how long the node stays in `DRAIN` state, and how long it takes to run a basic command once it returns to the cluster. On AMD systems, we inject a UMC RAS error and track everything the same way.

This is our most critical test. Providers where we hear a bunch of customer complaints about reliability generally do not have health checks, monitoring dashboards, and autoremediation in place. Reliability is the #1 most important criteria to many of the biggest customers in the world, as we discussed in great detail in our article “How Much Do GPU Clusters Really Cost?”

It is worth noting that our hands-on testing is only a part of the overall ratings. There are many things that we cannot test hands on, and need to use other research methods to understand in detail. Namely, performance at scale, reliability over time, support experience, pricing, GPU availability and delivery timelines.

Coming Soon: Agentic Workloads and RL

We have two more test suites that we will say much more about in upcoming work.

CPU Compute

The CPU is a critical performance consideration for modern agentic workloads, which we have discussed previously. We have developed a comprehensive benchmark that pushes CPU performance to the limits on exactly these agentic workloads, via common Linux tool use and more microbenchmarks in an upcoming project called SandboX.

SandboX runs a range of CPU-intensive workloads representative of code execution, including ring-shuffle producer-consumer queues, Redpanda (measuring a local Kafka-compatible broker and its throughput), and a dynamic load bench.

We also benchmark the sandbox lifecycle of Kubernetes clusters, for insight into how co-locating sandboxes for RL on the leftover CPU/memory resources in a GPU clusters might work. We report cold-start latency, scheduling throughput, density per node, capacity, and baseline.

Reinforcement Learning

An RL run has many moving parts: generators that roll out trajectories, environments that execute actions in sandbox, and a trainer that receives rollouts, updates the policy and pushes updated weights back out to the inference engine. Scaling RL means balancing maximizing utilization of trainer, inference engine, environment sandbox executions, rollout transfer, weight sync among other things, bottleneck on any one can stall entire RL stack.

We’re also seeing the emergence of Post-Training-as-a-Service, or RLaaS, where customers bring their own environments, or have forward-deployed engineers build them, and hosted training providers set up the infrastructure to post-train open-weight models. The pitch is customization while protecting the customer’s IP, which apparently is Satya’s tagline now.

In our upcoming PostTrainingX benchmark, we sweep trainer-to-inference GPU allocations, batch sizes, rollout concurrency, and policy-staleness limits to study their effects on performance and training stability.

We track generator performance through input/output tokens per second per inference GPU, rollout latency, KV-cache occupancy, and prefix-cache hit rates. Communication efficiency through weight-synchronization latency, transfer bandwidth, and network telemetry; and trainer performance and stability through training throughput, optimizer-step time, MFU, train/rollout KL, observed policy staleness, clipping fractions, gradient norms, and rejected rollouts.

For environments, we measure setup time, tool-execution time, timeouts, and infrastructure failures. GPU utilization, memory usage, and power consumption provide system-level context, while archived trajectories, logs, and exact environment definitions make results reproducible.

The goal of PostTrainingX is to benchmark every provider. That includes hosted training platforms like Applied Compute, Azure Foundry, Fireworks, Thinky, Baseten, Trajectory, Engram, AI21’s platform etc, as well as open-source frameworks like Miles, slime, prime-rl and verl. The end result should give customers the data to decide whether to run their own post-training stack on open-source frameworks or hand it off to a hosted RL provider.

We’ve already started running on open-source frameworks, including Miles, prime-rl, and verl, and will expand to hosted providers next. Stay tuned…

Industry Trends

Financing

The hottest topic in neocloud land over the past few months has been financing. Everyone wants to “follow the money”, which is exactly what we do in our AI Compute, Capital, and Markets Model, which we launched in a recent article breaking down how NVIDIA’s Backstop Universe. Below we will discuss a few more hot topics.

Debt and Accessing Debt

Today, neoclouds have realized that a creditworthy customer contract can help them obtain debt at lower rates than their own credit rating would otherwise support. This means that providers without a committed customer (or “IG offtaker”) may need more expensive debt or equity-like capital in order to buy GPUs and either get their business off the ground or grow it. Meanwhile, lenders also need evidence that a provider can deliver its cluster on time and meet the SLAs in their customer contract. Raising debt depends as much on lease termination rights as it does on the cash available for debt service, as well as the TCV. This has led multiple insurance providers to enter the market with parametric offerings that protect neoclouds and their lenders against contract terminations, typically offering a one-month bridge to find a new offtaker in the event of a termination. We are big fans of this structure, and believe it to be a solid method for lenders to use to de-risk these transactions.

Counterparty Risk & Nvidia’s Balance-Sheet-as-a-Service

In a large neocloud project, the GPU lender and the datacenter lender can face different counterparties within the same project. For example, in the Anthropic TPU neocloud structure, Broadcom supports equipment financing while Google supports datacenter rent. Since several loans in the supply chain can depend on the same AI lab, there can be systemic, correlated risk, even when each loan names a different neocloud or datacenter borrower. Like we just discussed, construction delays can trigger cancellations and impair cashflow.

Enter Nvidia backstops. Since Nvidia supports GPU demand through revenue floors, landlord guarantees, and leases that it plans to transfer to third parties, they are able to help neoclouds and AI labs obtain investment-grade financing without a hyperscaler being involved. Nvidia earns its GPU margin at the initial sale but can now also receive a share of revenue above their “AICP floor”. Their financial support therefore drives demand for their hardware. The Balance Sheet Is The Moat.

Depreciation Schedules

This topic has died down a little since we went after Dr. Burry in our article last November. Maybe it was all those 6 year contracts people are signing, or those pesky 4-year-old H100s that just won’t drop in price! Who’s to know for sure.

Either way, we separate equipment purchase dates from in-service dates when we model depreciation. A 6-year depreciation assumption needs provider-specific support, while in NVIDIA’s AICP program, the 6-year revenue floor does not actually establish the useful life of the GPUs. Our depreciation sensitivity tests change modeled margins quite a bit, but IRR remains the priority for lenders, so we won’t be sharing them here. And of course, financial analysis does not establish a real price that a buyer will pay for GPUs in the future. That is up to supply-and-demand dynamics, and the ultimate question: residual value. With lenders unable to set loan repayment schedules separately from the accounting assumptions that we use to depreciate equipment, we are stuck in a world where no one will take on the risk of assessing a residual value on 4, 5, or 6 year old GB300’s today.

Blackwell and Grace Blackwell

Now let’s get back to the technical stuff for the rest of this article. The good stuff.

As we discussed in great detail in ClusterMAX 2.0, the difference between operating 8-way H100 HGX servers and GB300 NVL72 rack-scale systems is massive. Providers who cut their teeth installing H100s 3-4 years ago, are now contending with the following:

1. Grace CPUs based on ARM (instead of x86 Intel or AMD)

2. Blackwell GPUs with sm100/sm103 (with CUDA 13.0+)

3. Mandatory direct liquid cooling (DLC)

4. High voltage power (130kW+ per rack, via 3phase 408 or 480V power delivery)

5. Scale-out networks with 800G CX-8 NICs and 51.2T or 102.4T (800GbE RoCE Spectrum-X) or 115.2T switches (800Gb XDR InfiniBand)

6. Scale-up networks with NVLink switches and backplanes at rack-scale, 72-way

7. BF-3 or BF-4 DPUs for the frontend (or converged frontend/storage) network

8. Multiplanar networks with shuffle boards/shuffle cables

All of these changes have implications on provisioning software, monitoring systems, training of technicians, facility-level electrical, cooling, and mechanical systems, OEM support contracts and relationships, and more.

Just because a provider was successful in the Hopper generation, does not mean that they will be successful in the Blackwell generation.

In our rankings, we put a premium on any provider that delivered Grace Blackwell to us, and frankly gave them a lot more leeway when it came to difficulties during provisioning, monitoring, and reliability/hot spares.

This means that if a customer rents standard HGX nodes that merely connect by RoCE or InfiniBand, we expect to see autoremediation with hot spares available: replacements in under an hour end-to-end and detection in around 2 minutes. However, with an NVL72 system, it does not make sense to hot-swap a tray on a different scale-up fabric.

So, in our testing, as long as a health check notices the unhealthy node (in 2 minutes or less, still) and keeps workloads from being scheduled on it, it has done its job. As mentioned above, we advise many labs and clouds on GB300 NVL72 SLAs, and the most common pattern we see could be called “NVL64+”. This means that there is an SLA applied to each node, but in the event that 3 or more of the 18 nodes in a rack fail at a given time, the whole rack is considered down. This is due to simple powers of two: lots of jobs are divisible by 64, and run on 64 GPUs, while the max config to fill up 72 GPUs is world_size=8 * num_jobs=9.

Moving to Vera Rubin

Interestingly, moving from Grace Blackwell to Vera Rubin requires a lot less changes than moving from Hopper to Blackwell. Providers still manage an Arm-based CPU, a liquid-cooled rack-scale architecture, a 72-way NVLink scale-up network, a multiplanar scale-out network, and some Bluefield DPUs. Back when GB200 came along, all this stuff was new!

The major changes are in the move from 800G CX-8 NICs to 1.6T CX-9 NICs on the scale-out network, and increased power moving from ~130kW to ~200kW per rack, including 800V DC options.

Early demand is more robust than in the Blackwell transition—having partially to do with the fact that it’s easier to port GB software to VR than it was from Hopper to GB—and we’re hearing from many providers that they are on track to deliver VR earlier than expected. Bring up has been surprisingly smooth, and the physical design seems to be much more mature.

Scale Out Networks

The scale-out network is the last big design decision a provider still owns. Nvidia and the OEM/ODM lock down most of the NVL72 rack. Outside it, providers pick the topology, the cabling, and how Kubernetes or Slurm hands the fabric to a job. We found problems at every layer.

Starting with the physical layer, 800G CX-8 NICs are multiplanar: each NIC splits its bandwidth across independent switch fabrics, called planes. Total bandwidth stays at 800G per GPU. More planes give each switch more effective ports, and each NIC more paths to uplink, which needs a shuffle. Every NIC lane must land on the right plane, so shuffle boxes put the fiber mapping in a patch module while shuffle cables build it into the assembly. Both work, but there are tradeoffs in terms of reliability (how often you need to service the devices) and serviceability (how easy it is to repair or replace a device).

Oracle is a pioneer in this physical design, and it shows when you get a GB300 node. Each 4x GPU tray we got had 16x physical 200G ports, which means 4 planes per GPU. Linux exposed 4 usable 800G RDMA VFs (rdma_vf_rail0 to rdma_vf_rail3), one per GPU, each in its own VRF. The aggregate PFs failed with no usable GID, so we had to rely on their SPCX NCCL plugin, using 16 queue pairs per connection, and adaptive routing across planes.

Why bother doing this? The answer is scale. Oracle builds Acceleron on this physical design. Their MRC topology paper shows how each NIC splits into 8x100G, which turns a 51.2T switch into 512 ports, and supports 131,072 GPUs at full bandwidth, with only 2 tiers, and a max of 3 switch hops. OpenAI released MRC through OCP in May 2026, and Oracle runs it at Stargate Abilene. The software layer allows MRC to spray one connection across paths and planes, place data out of order, recover loss with selective acks, and uses SRv6 source routing to steer around failed or congested links. This logical layer can get complicated any scale. Its where jobs have issues.

To allocate an RDMA device and use it correctly, Kubernetes must hand each pod the right devices and interfaces. In other words, there are many ways to configure and use the NVIDIA NetworkOperator. Get it wrong, and it’s a headache for everyone.

Our scripts recognizes the following deployment patterns with the NetworkOperator (resource names are examples from the clusters we inspected):

• rdma_shared_device: rdma/rdma_shared_device_* with a resource-backed NAD, or a host-network fallback

• sriov_host_device: nvidia.com/hostdev or nvidia.com/rdma_host_dev with a HostDeviceNetwork NAD, for direct device access and GPUDirect RDMA

• sriov_legacy: a VF via NAD-bound nvidia.com/<resource> and a SriovNetwork NAD, plus RDMA CNI where needed

• sriov_ib: an InfiniBand VF such as nvidia.com/mlnxics with a SriovIBNetwork NAD. Partitioned fabrics also need the right PKey and UFM config.

• ovs_offload: nvidia.com/switchdev with an OVSNetwork NAD, for hardware-offloaded Open vSwitch.

• rdma_ib / rdma_roce: rdma/ib, rdma/roce, or nvidia.com/rdma_*, with one NAD per rail or a host-network fallback.

• Provider-specific: AWS EFA (vpc.amazonaws.com/efa and libfabric), DOKS (rdma/fabricN with roce-net-fabricN@fabricN), and GKE DRANET (DRA claim templates and pod resourceClaims).

Device setup and the NCCL plugin on top of it can change job performance. On Google’s GB300, the built-in NCCL library picked an unroutable link-local GID and hung when we tested. Setting NCCL_IB_GID_INDEX=3 fixed it, and Google’s gIB plugin hit 99.5 GB/s on a 16-node, 16 GiB all-to-all vs 95.3 GB/s from other providers, i.e., full performance.

With that said, multinode NVLink is its own problem. The ComputeDomain CRD in NVIDIA’s DRA driver gives each job its own IMEX daemons and channel claims, independent of RDMA. Google, GMI, and Firmus Kubernetes used ComputeDomain, with different RDMA paths: DRANET at Google, shared devices with host networking at GMI, per-rail VFs at Firmus. Meanwhile, Oracle, Azure Slurm, GMI Slurm, Firmus Slurm, and Nebius Soperator supplied IMEX from the host with no tenant claim. Azure AKS ran multi-node NVLink with no tenant-visible ComputeDomain, despite GPU DRA ResourceSlices, and the channel source and tenant isolation boundary were undocumented.

Each ComputeDomain must stay inside one NVLink domain, since crossing domains needs RDMA. In other words, this is topology-aware networking on the hierarchical scale-up NVLink + scale-out RDMA networks. When we launched jobs incorrectly, without ComputeDomain awareness, they just hung in clique setup when placement crossed racks. We will leave our rants about EFA not supporting DeepEP and MoonEP for the AWS provider review later on in the Appendix.

Reliability, Health Checks, Autoremediation, and SLAs

Around the time we started our research for How Much Do GPU Clusters Really Cost?, we started injecting failures into every cluster we tested. The differences between providers were big. Handling a failure takes two things:

1. Identify it happened

2. Fix it properly

To identify failures, we recommend monitoring dashboards. A good dashboard shows the failed component, the affected jobs, the current scheduler state, and when each check last ran. A stale green result is not evidence of health. Of course, a monitoring dashboard relies on some telemetry, so you have to have DCGM setup correctly (and securely).

To fix failures, we expect autoremediation: reboots and replacement with hot spare nodes. Autoremediation will not cover everything. Some failures need human intervention, RMAs, and other work that the provider must communicate directly to the customer. We covered the exact cost, measured in the form of goodput, in our article on this topic.

NVIDIA recently open-sourced NVSentinel, a fault detection and remediation system for Kubernetes. It reads DCGM, syslog, and cloud maintenance events, then can cordon and drain nodes, reset GPUs, or reboot. The default install is monitoring only. The software is the easy part. The important part is the mapping from each XID to the action the provider takes. A contained XID 94 needs an application restart. An uncontained XID 95 needs GPU recovery. Clearly, if you naïvely reboot every node for every XID, you’ll kill healthy work, while if you leave GPUs schedulable after serious XIDs, you risk workloads crashing.

This is also why many customers want bare metal only. They just want the provider to close support tickets. Closing support tickets is harder than it sounds. A given ticket can come from any of 100+ XIDs, each with a different meaning and resolution. SOPs cover everything from swapping a drive or power supply, to cleaning cables, to reseating memory, GPUs, or NICs, all the way to RMAing individual components or whole server trays.

There are a lot of different choices for autoremediation, and the right one depends on the system. Also, remediation is a different question for GB than for HGX. On HGX, you swap in a hot spare node with 8 GPUs. On NVL72, you cannot hot swap 8 more GPUs into the NVLink domain. A failed tray takes 4 GPUs down, and the options are to run the rack degraded or to swap the whole rack. The good news is that GB300 tends to fail less than GB200. Either way, the flow from an error to the action in the datacenter is what matters.

Some providers told us they wait before acting, to leave customers time to get their stuff off the node. That is cope. We are simulating a hard failure. The node is dead, and there is nothing left to get off it. (It’s a general truism that if a provider says they choose not to do something because they want to “give their customers optionality,” it is cope, and it’s only a matter of time before the provider recognizes it as such.)

This is where owning and operating the datacenter differs from renting colo. A provider that runs its own facility controls the technicians, the spares, and the SOPs on site. A provider in colo can depend on the landlord’s remote hands and their schedule. Both show up in how fast capacity comes back, and that is what an SLA has to measure.

When it comes to SLAs, we have developed a standardized set of SLAs for 3 tiers of providers

In these documents we provide the following:

• Technical definitions for “node”, “rack”, “cluster”, and “site” in two scenarios (HGX 8-way systems, i.e. H100 or B300, and MGX 4-way systems, i.e., GB200 or VR NVL72)

• Technical definitions of “downtime” for all systems

• Three tiers of associated SLAs (“Bronze”, “Silver”, and “Gold”, with corresponding thresholds and credit penalties associated with downtime)

• Two-way commitments for measuring downtime and providing credits

• Recommended exclusions from downtime measurement, such as software upgrades, security patches, and physical maintenance

• Recommended monthly review cadence for provider performance against SLAs

• Contract termination rights for the buyer associated with downtime below recommended thresholds

• Definitions of acceptance testing, with descriptions of the types of tests to perform across GPU compute, networking, storage and software to validate a cluster is operational and can be accepted, with recommendations to contact SemiAnalysis for a ClusterMAX technical assessment

• Contract termination rights for the buyer associated with missing an agreed acceptance date

• Definitions for “force majeure” clauses

• … and more

We encourage Compute Buyers (Neolabs), Cloud Providers (Neoclouds), Debt/Financing Partners, and Insurance Providers to use these terms as a basis for their contracts, and to include a SemiAnalysis Technical Assessment during acceptance testing and monthly SLA performance reviews. We perform this Technical Assessment using our cmax utility and a comprehensive, proprietary process.

Please contact us at clustermax@semianalysis.com for more information.

Security

Since we put out “Most Neoclouds Suck at Security,” we’ve been encouraged by the response. Our friends have been running our CLI on their clusters to find which dusty old packages to replace, and there has generally been increased impetus to treat this issue with the seriousness that it merits.

Still, we think it deserves more scrutiny. As discourse reaches a fever pitch over whether AI will kill everyone, there has been almost no analysis of the concrete means by which it could actually elude control and do harm. OpenAI and Anthropic are spending gazillions of dollars doing gain-of-function research, some of their infra has been revealed deeply suspect, and yet it has mostly escaped commentary that GPU clusters—the physical hosts that underlie these models—often have nothing like the immune system that you would hope.

The imperative of good security practices holds regardless of your beliefs about AI capabilities. Even if these clusters were made no more vulnerable by AI tools, the fact would remain that we are spending billions on infrastructure that is not secure as it ought to be. There are plenty of unpleasant scenarios to imagine that involve nothing more than a guy with Vim and a good understanding of operating systems. Read through the Hugging Face reports and tally the number of mundane cybersec errors that let the fire spread.

It’s also worth emphasizing that this is, to some extent, a structural risk. As the industry navigates the hazards of heavy-handed regulation and uncritical social pushback, a breach of an LLM factory is the last thing it should invite into headlines. This is true for cybersecurity reasons—each compromised node increases the number of neighbors for bad actors to explore—as well as ordinary sociopolitical ones—for better or worse, detractors will paint the neocloud industry with a broad brush.

Be a good neighbor and don’t get hacked.

Agentic Coding

Agentic coding is already putting margin pressure on managed GPU clusters because customers that use AI tools are more willing to take bare metal or lightly managed clusters. We saw evidence for this all the time in our testing. If we needed to add Linux users, but there was no script on the cluster, Codex /goal make useradd script and add my whole team saved the legwork. Especially on clusters with poor documentation, agents helped grind through all the possible configurations to get us unblocked. The corresponding Slack conversation often went something like:

“hey guys, how are we supposed to use XYZ?”

“nevermind, Codex told me ABC.”

“ya that’s right. we’ll add that to the docs.”

Agents helped us most in environments with clear success criteria and good context; otherwise, their approaches to GPU systems were naïve. They have the relevant knowledge, but it is not highly weighted relative to plausible but wrong approaches. Claude, for example, has a habit of overwhelming the Slurm controller by using loops instead of job arrays. One agent of ours decided the NVLink fabric was too much of a pain to configure, instead launching jobs over InfiniBand and declaring victory; another made a similar mistake running storage tests on NVMe instead of NFS; another blamed the provider for poor perf after it tried to schedule GPU jobs onto a CPU node. We usually let agents run overnight, either with `/goal fix this part of the cluster so tests can run` or `/goal try to break this part of the cluster with a realistic workload`. Overall, this allowed us to multiply our output, but sometimes validating the results took longer than it would have to do it ourselves. If you have a good idea how to set up a cluster, agents make you more powerful, and you might even entertain working with a worse provider than otherwise. If you don’t know what you’re doing, your AGI advisor will cheerfully tell you to shoot your foot off. In this semi-legible business, agents magnify skill deltas rather than flattening them.

An insightful example of this is in CoreWeave’s bring-up, which was already among the most rigorous in the industry. Now, in addition to a stringent classical process, they have LLMs run statistics over their fleet, hunting for patterns that flag failures earlier and improve their procedures. CoreWeave also has a NodeBot that provides lightweight reasoning on top of their automated alerts, summarizing the situation and suggesting next steps to simplify diagnosis for the humans in the loop. For example, the team described an event where a thermal interface material pump-out alert should have moved the node out of the cluster for investigation, but a simultaneous PMU halt alert competed to reboot the node instead of triaging it. NodeBot identified the correct action despite that conflict. CoreWeave has experience running GPUs at scale, knows countless rules of thumb that are absent from the training data, and has experts constantly closing the feedback loop to improve their use of these models; even if OpenAI built a banger chip with AI-generated kernels, we don’t expect CoreWeave to be threatened by upstart neoclouds with vibecoded datacenter automations any time soon.

Despite the fact that—let’s be honest—everyone in the industry is using AI at least for some workflows, the only AI-first offering we saw was a nice MCP server provided by AWS. Otherwise, all the documentation was nominally written for humans. The best thing that a provider can do to cater to agents is to provide exhaustive documentation—ideally, much more thorough than a human could ever have gone through—and leave golden recipes on the cluster as a starting point for hillclimbing. In addition to the legacy processes that are not yet AI-first, there are many systems that are strictly worse because they are not built for agents. For example, although letting AI manage your jobs could be good for cluster utilization, it doesn’t respect the old-fashioned systems put in place to aid research. CoreWeave and Nebius have many Slurm-layer optimizations—real job accounting systems with RBAC and priorities enforced on users and groups, carefully managed partitions, job arrays, logs and metrics—that are wasted when everything is run agentically. CoreWeave also tells us that they have seen head nodes run out of memory as remote-control Claude sloporchestrates the cluster. It will be interesting to watch providers improve their AI agent affordances.

Going Forward

As described throughout this article, we have been historically testing clusters. But the market has clearly moved in two opposing directions.

First, the biggest labs are buying bare metal at massive scale, measured in the 100’s of MWs, and have the talent in house to manage their own training clusters, orchestration software, monitoring and reliability. In essence, their neocloud providers (when they use them, and don’t do self build) are datacenter technicians who close tickets. This is a complicated business, and not to be taken for granted: there are 172 unique, documented ways an NVIDIA GPU can fail (XIDs). And many of these failure modes can overlap, resulting in an incredible surface area of different failure scenarios that require a combination of simple troubleshooting logic, human experience, and increasingly, agentic research by AI to recover from these failures.

Meanwhile, many startups are raising 10s or 100s of millions of $$ and spending effectively all of that money on compute. But since so much of this research is driven by RL, via continual learning on real production traces for long horizon agents, the needs for both offline, throughput-optimized inference, and online, latency-sensitive inference is very strong. At the same time, more and more customers are looking for the simplest way to purchase vanilla open source tokens for their workflows.

This leads to our next article, coming soon, where we will describe the Anatomy of an Inference Endpoint, moving towards the inclusion of inference endpoint testing in future versions of ClusterMAX. We believe that all neoclouds need to have an endpoints business going forward.

We will also be evaluating RL infrastructure beyond inference: sandboxes, hosted training, evals, and more. We continue to see lots of new providers enter (and exit) the market every day. It is an exciting time to be in AI, and neoclouds are at the centre of it.

If you got to this point in the article and are still eager to learn more about our individual experience on every provider, you should consider coming to work at SemiAnalysis!

We are actively hiring across Research, Consulting, and Technical Staff. We have multiple openings in our New York and San Francisco offices. Remote work is also acceptable.

ClusterMAX Engineer: all levels of experience with Slurm, Kubernetes, and GPUs considered (full time)

Tokenomics Engineer: all levels of experience with model evals, harnesses, inference endpoints, and RL infrastructure considered (full time)

Research Analyst - AI Infrastructure and Economics: all levels of experience covering the financials of neoclouds, neolabs, and frontier labs tokenomics considered (full time, or internship)

Technical Consultant: lead engagements from technical strategy to technical due diligence. all levels of experience from consulting and technical backgrounds considered (full time)

Thanks for reading, and we look forward to seeing you for future versions of ClusterMAX.

Appendix: Provider Reviews

Platinum

CoreWeave

CoreWeave remains the industry’s sharpest, sturdiest, most proactive neocloud.

Our CoreWeave clusters are exemplary in all crucial categories. The health checks work as intended, the reliability is excellent, and nearly all tests reach expected values out of the box. Along with Nebius, CoreWeave serves as a measuring stick for others in the industry—not only in the abstract sense of “excellent performer,” but also in the literal sense that we compare their clusters to lower-rated ones as a means of precisely describing others’ shortcomings. Whereas other neoclouds have to be nagged to refresh their moldy GPU drivers, CoreWeave has not only taken care of the basics, but stood up a significant independent stack for marginal performance and reliability improvements.

The ClusterMAX 2.0 report went into great detail on CoreWeave’s design decisions, including its bare metal provisioning, Slurm fork, use of DPUs, and health checks. Since then, one smart feature that the nerds at CoreWeave have added is called “GPU straggler detection,” which is integrated into their best-in-class dashboard. Rather than leaving their customers to dig through logs or do binary search over their fleet to isolate a bad rank, CoreWeave applies handrolled algorithms to NCCL telemetry and recommends a remediation strategy based on its findings. The CoreWeave team tells us that they use this on calls with their customers just about every day. There are all kinds of soft failures that don’t write an XID or leave any other obvious signature, but waste GPU hours as the job waits on a straggler.

Better than spotting bad GPUs is not scheduling them in the first place; CoreWeave also has systems to make sure every node clears a high bar before it’s trusted with a customer’s workloads. When a node is idle in CKS, CoreWeave runs burn-ins for 20-30 minutes of every hour, validating that scale-up and scale-out fabrics are healthy, GEMM numbers are up to spec, D2H and H2D bandwidth are high, and so on. These tests are pre-empted by customer workloads; invisible to the operator, they help make sure that it’s not production workloads that catch faults. CoreWeave is certainly not the only provider with active health checks, but these are unusually robust.

This diligence extends to CoreWeave’s bring-up process. Provisioning around 10,000 GPUs per week, CoreWeave believes that they have more data than God about what proves a Blackwell rack healthy. In CoreWeave’s view, Nvidia diags are good at catching many failures, but their uniquely deep dataset has allowed them to roll out a custom suite of supplementary tests, including for NVLink. The heart of this process is the Fleet LifeCycle Controller (FLCC), which “automates node provisioning, testing, and monitoring” and owns the path from power-on to handover. Importantly, as others operating fleets at scale can corroborate, there’s a long tail of obscure failure modes not well covered by Nvidia documentation. CoreWeave keeps statistics on these failures to better predict future faults and bakes its response patterns into FLCC. This is one reason that experience with operations at scale is a crucial criterion in our ratings. There is a huge difference between a newbie and a provider like CoreWeave that has rigorous runbooks, automation, and burn-ins. As described in the “Agentic Coding” section above, CoreWeave uses LLMs on top of this system to glean insights that would not be spotted by a human engineer. CoreWeave also works with manufacturers before delivery. Involvement varies by OEM or ODM and by production stage, but in general, the goal is to work backward through the supply chain to catch issues as early as possible: never catch in prod what you could have spotted in bring-up, and never catch in L11 diags what you could have spotted on the factory floor.

Their experience with Blackwell NVL72 bring-up and validation, along with their close relationships with Nvidia and Dell, have allowed CoreWeave to be the first provider to announce a VR200 NVL72 system passing L11 diags. CoreWeave has described VR as “the generation where we bring our own IP.” As the physical demands of compute continue increasing, in CoreWeave’s view, “how the building operates is now part of how the computer operates—they are not two individual things anymore.” Thus, CoreWeave introduced its custom Racky, Valvey, and RLCC in a recent blog post. Particularly interesting is Valvey, a programmable per-rack liquid cooling valve assembly. This gives CoreWeave fine-grained control over each rack’s cooling loop and also, in case of a leak or other emergency, the ability to trigger a shutdown. Despite beginning to adopt a hyperscaler mentality in building its own hardware, CoreWeave frames its approach as anti-hyper: whereas traditional infra emphasizes redundancy and is built to prevent failures, CoreWeave builds to fail gracefully and recover fast, limiting failure domains and allowing nimbler response to the inevitable failures.

Amidst the current trillion-dollar buildout, however, CoreWeave is responding to the massive increase in customer demand and dealing with increasing cost of capital. Of course, there is plenty of good news to mention. Its relationships with Meta, Microsoft, OpenAI, and NVIDIA—its largest customers—appear strong, and it recently struck a multi-year deal with Anthropic. It has announced splashy deals with HRT and Jane Street this summer. It is also aggressively expanding its capacity in APAC, indicating that it “expect[s] international markets to become a major driver of growth.” Yet it sits with $35B in debt and as the cost of capital increases, it takes longer to get projects off the ground. As a result, CoreWeave has become focused on long-term bare metal contracts, which are easy to fund with an investment-grade counterparty, rather than managed clusters, which have higher margins but which the market is less keen to fund. Moreover, our last writeup mentioned an impressive lineup of recent acquisitions, but since then its Core Scientific merger fell through and nothing else has been announced.

In terms of feedback, some of which is a bit nitpicky:

1. We wish that enabling local GPUDirect Storage could be an 1 click button in the UX console

2. Onboarding new users and where they put the the ssh pub key input in the UX console is extremely confusing to many users that tried

CoreWeave has begun to diversify from bare metal, first by offering self-service SUNK clusters, which gives them the option to sell into the on-demand market when they have capacity for it. It is also worth noting that not all “bare metal” contracts are the same. In all of CoreWeave’s bare metal contracts, they still manage the scale-out and frontend network, do burn-in, handle repair/replace, monitoring, and other general day-two operations. By comparison, other providers claim to be a neocloud, but are locked out of their own data hall. We will dig into more of these details in an upcoming article where we compare the different bare metal offerings from different providers.

CoreWeave has also launched a managed inference platform, which recently disclosed $100M ARR. We are actively testing this for an upcoming article on serverless inference endpoints, though were disappointed to find no support for Kimi K3 and GLM 5.3, and the only way to consume the public endpoints by the token is via the Weights & Biases portal, or via OpenRouter. For now, let’s just say that CoreWeave has some work left to do on their inference endpoints.

Nebius

Nebius was rated the 2nd-best neocloud in the world in ClusterMAX 2.0, yet it remained in Gold. This time, Nebius is unquestionably an industry leader with strong offerings in every category and the ability to command a significant price premium. Nebius moves into Platinum.

Nebius’s relationship with Nvidia remains strong, including a $2B investment in March, as Nvidia disclosed a 9.3% overall stake in July. Nebius was the first provider to deploy HGX B300s back in December of 2025, and has been among the first neoclouds in VR NVL72 bring-up, expecting to begin deployment late this year or early next.

Nebius’s buildout continues across a number of sites in the US and Europe as they recently raised their 2027 year-end contracted power target to 5 GW. In terms of offtakers, Nebius has signed large deals with hyperscalers Meta and Microsoft and smaller ones with Reflection AI and Palantir, making it one of a few neoclouds with multi-hundred-MW hyperscaler agreements that still regularly competes for much smaller startup contracts. They are the most active of all neoclouds in the short-term cluster market, serving many happy customers. The days of Nebius spooking prospects with their Russian accents are behind them. This is mirrored by its capacity procurement: unlike many of its peers, Nebius has opportunistically snatched up sites in the 5-20 MW range through a broad network of datacenter and software partners.

In terms of the clusters themselves, well, a good cluster is generally boring. On Nebius, burn-ins run without errors; Slurm is topology-aware; the packages we need are on the cluster and generally recent; WAN is good; the orchestration layers are free of footguns. One point of distinction technically between Nebius and everyone else is that Nebius builds and open-sources its preferred SonK flavor, Soperator. (CoreWeave, of course, builds SUNK in house, but it is not open-source.) This round, a Gcore cluster we tested used Soperator, and as did Voltage Park and FPT clusters from previous rounds. Nebius writes that “the real magic of Soperator lies in how we use ‘jail’ Persistent Volume”: Nebius provides a VirtioFS volume at `jail`, then bind-mounts it at `/`, so you don’t need to worry about pointing your cache to the right directory, or think much about taking your files with you when you `salloc` a node. You’ll see in our writeups of other providers a handful of gnarly issues with managing the storage system on SonK: Slurm and Kubernetes have different philosophies of persistence, and it’s not always straightforward to provide the illusion of bare metal when in fact you’re several layers up, standing on an ephemeral Kubernetes pod. Nebius’s approach makes this stress-free for the operator.

In particular, Nebius provided us a handful of storage tiers on every worker: `/jail` VirtioFS, `/home` NFS, `/mnt/data/` VirtioFS, `/mnt/local-nvme` ext4, and `/mnt/memory` tmpfs, as well as first-class S3. For I/O-heavy jobs, `/mnt/data` is recommended, whereas `/home` suffices for shared code and other data with lower-concurrency access. When we first tested `/mnt/data`, we saw very slow 4 KiB sequential allocating writes, while concurrent directory creation from 128+ ranks sometimes returned `EEXIST` due to inconsistent metadata visibility, killing our jobs. With our feedback on the pathological workloads, Nebius retuned this filesystem, and it proved sturdy and fast. We pushed it around and verified that the errors were resolved; per-client, it outperformed the median, and we could further verify that it scaled to handle demanding workloads across two racks.

Nebius has put a great deal of work into its health checks, and in our testing, they performed well. Nebius automatically returned a node to service after a synthetic error injection on our GB300 rack; the process is not yet optimized for speed, taking 8h40m from start to finish, but detection is immediate, and the process ultimately works. Moreover, the other GB300 racks we tested did not automatically reboot and return failed nodes, so the fact that this process is self-driving counts in Nebius’s favor. (As we describe in the “Blackwell and Grace Blackwell” section, it’s common not to autoremediate GB300 nodes at all, since there is no real way to hot-swap a node into an NVL72 rack, and troubleshooting is generally way more complicated than with HGX servers.) Nebius also provided us with an excellent suite of dashboards shedding light on, among other things, cluster health.

This all leads to Nebius watching their revenue per MW climb compared to CoreWeave’s (and their market cap along with it). Since they have so much more capacity to sell at the current prices, we expect this trend to continue. Anecdotally, we have seen Nebius get very aggressive during recent negotiations, including a case where they offered a tranche of capacity as a straight-up auction. In general, demanding high prices and sizable prepayments that reach 100% on 1 year commits seems like quite a nice business, especially when those prepayments can cover the entire capex of the servers at the current TCV. Infinite Project IRR anyone?

At their largest bare metal sites, Nebius continues to face delays in construction and permitting, including both Béthune and Vineland, which we have covered extensively for clients of our industry-leading Datacenter Model. But as more chips of all types come online, Nebius’s track record puts them in a strong position to continue to grow.

Overall, we are consistently impressed with Nebius. While we do believe they still trail CoreWeave on many technical aspects and relationships with both NVIDIA and the frontier labs, solid business decisions have established them as the default neocloud to serve the neolabs.

Gold

Google Cloud

Amidst the strange retreat of Google Brain and DeepMind from the frontier, GCP has been dealmaking like few others. Notable partners include Palo Alto Networks for $10B and Thinking Machines Lab for a “multibillion-dollar deal,” along with expansions of its lucrative relationship with AI safety org Anthropic. With acquisitions of cybersecurity startup Wiz for $32B and energy startup Intersect for $4.75B in cash, Google has been aggressive in expanding its technical capabilities. It also exposes itself to new revenue streams by ramping TPU sales instead of only renting them through GCP, a development we are very keen to track. Its own TPUaaS is still growing, though, thanks to a $5B deal with Blackstone that is targeting 500 MW in capacity. Google has gigawatts in the pipeline, and we look forward to continuing to track its continued competition with the other hyperscalers and labs for access to power and permitting.

With a unique position cutting across so many layers of the stack, Google may have more total engineering talent than any company in the world, yet it has a notably imperfect history in supporting innovation. In ClusterMAX 1.0, we noted a myriad of issues with its platform, yet we predicted GCP would reach Gold or Platinum promptly. It has taken until ClusterMAX 3.0 for this to come true, but Google is finally and comfortably among the best managed cluster providers.

The GCP GPU experience is less refined than that of our Platinum providers, CoreWeave and Nebius. The console feels like the DMV compared to the streamlined UIs of the younger, more focused neoclouds. Access required a Google Cloud CLI that was occasionally uncooperative, forcing us sometimes to switch to the browser-based connection in Google’s console, which was also unreliable. There was a small misconfiguration on our GB200 GKE cluster, where default StorageClass refused to attach to the `a4x-highgpu-4g` nodes, forcing us to try again with a non-default class to get block storage. Nitpicks aside, GKE is very well set up, and we enjoy using it. Google’s managed Slurm is officially generally available and quite solid, with good defaults and health checks configured. A lot of the testing that we did for ClusterMAX 2.0 on their managed Slurm offering is now valid, since the bureaucrats in GCP Product Management are allowing real customers to use the offering, not sticking it with labels like “Beta” or “Pre Release” or “not yet GA.” (By the way, many of the neoclouds we’ll discuss after Google should consider using some of these labels more often—the balance is somewhere in the middle here.)

The biggest point of distinction between our GCP cluster and the other clusters we tested was networking. Our first NCCL test hung because automatic GID selection chose the wrong address, but pinning `NCCL_IB_GID_INDEX=3` completed the run. Even then, our NCCL tests were unsatisfactory: you would like to see a roughly logistic curve as performance increases monotonically in message size, but instead we had the following jagged shape:

This could conceivably have been an issue with NCCL itself rather than a problem with our GCP configuration. NCCL’s heuristics don’t always pick the right protocol and algorithm for the message size and topology, and some hand-tuning is always expected. However, we have gigabytes of networking data gathered at this point, and we knew to expect better perf. The issue turned out to be simply that we did not have the gIB plugin enabled. This is a set of NCCL plugins Google that ships for improved perf on Google’s RoCE network. As of 26.07, gIB is baked into the NGC PyTorch image, so Google’s customers get the custom stack by default. With gIB installed, we got a better chart with 4.4% higher peak throughput for 16-node all-to-all. (Note that these tests are both with the multi-node NVLink fabric shut off to isolate performance of the scale-out network, which is RoCE in this case.)

While not perfect, this looks much better, and the topline number is solid. At this point, it was clear that the fabric was healthy, the configuration was good enough, and an engineer looking to squeeze more juice out of the system would have a solid baseline to start from.

This is amazing work by Nvidia and Google to set up auto activation for Google’s ConnectX-7/8 NCCL plugin. Previously, users would need to mess around with the correct library load paths and env vars to get it set up correctly and optimize performance on ConnectX NICs on GCP Nvidia GPU machines. We gave feedback a couple of quarters ago, and now, it is fully automated!

GKE’s health checks worked well. As with other providers who have experience with NVL72 systems, Google’s health checks intervened but left it up to the operator to bring the sick node back to the fleet. In particular, after our XID was injected, the health check marked `GPUUnhealthy=True` and `cloud.google.com/health-check-status=warning`, but left the node schedulable. This configuration was easy to work with, it’s what GCP’s customers prefer as a default, and it can be configured to the user’s liking to respond to errors of various levels of severity.

To continue improving, GCP could take a look at all the quality-of-life changes that Nebius and CoreWeave have made in the recent past. It should ship a better console that’s easier to navigate, simplify IAM and RBAC, and ensure cluster access is painless, either through standard SSH or `kubectl` or by making its existing CLI bulletproof. Especially now that gIB is available in the default Nvidia images, we hope that GCP’s operators have no trouble getting good performance out of their GCP scale-out. They could also continue to pursue neolabs in the mid-market, improve the hands-on support experience, and improve their relationship with Nvidia. GCP consistently loses so much business to smaller, less capable neoclouds, because they either lack the GPU capacity or refuse to serve the market at a certain price point.

We look forward to testing GCP’s VR NVL72 systems when they are available and seeing GCP’s continued improvements on usability and performance.

Oracle

As we mentioned in ClusterMAX 2.0, Oracle is in a unique position among hyperscalers in that its growth must come from large contracts. Unlike AWS, Google, and Azure, Oracle does not have any agreements to sell frontier tokens, and unlike Meta and Google, it does not have a non-frontier lab it can hang its hat on. Instead, it continues to cut huge deals involving OpenAI, Meta, and Nvidia, reporting “the delivery of 850MW additional datacenter capacity” from June through August, an eye-watering number. We project its relative growth in 2027 to be the largest of the hyperscalers, though from the smallest base.

We tested two Oracle clusters, which we will describe in order. The first was a GB300 NVL72 cluster on Slurm, set on a multi-planar network. This was vanilla Slurm, but due to customer demand, Oracle has Slurm-on-OKE on its roadmap. Once our allocation began, we tried to find our way in through the console, but it proved, like all hyperscaler consoles, a forbidding place.

Importantly, Oracle’s console did not support RBAC—nothing on the console is connected to SSH users—so all our SSH keys had to be managed on the machine. At one point, attempting to append to `authorized_keys`, our intern forgot a `-a` flag and removed all our previous keys. Classic user error, obviously, but this goes to show why we prefer to have a hardened path on a console. We’d rather not trust our interns—especially that one, no offense—to type the magic words on the command line when a mistake can lock us out of the cluster. Thankfully, Oracle’s engineers caught the mistake immediately and fixed it for us.

We hope they fix this and add enterprise grade RBAC.

As expected, Oracle’s cluster performed well from day 1. They provided us with scripts for InfiniBand and NCCL tests, which was helpful because vanilla scripts do not properly use all 4 rails. Burn-in was flawless and Oracle’s managed Lustre saw strong throughput. Oracle’s health checks worked as intended, draining the node on our synthetic error but leaving up to the user to decide whether or not to terminate and replace the GB300 node. Oracle’s managed Prometheus and Grafana were good, though not perfect. For example, they are not integrated with the end user experience for job accounting. However, on the positive side, health metrics are now integrated into Cluster Manager in the console, making it easier to get a view of the crucial statuses at a glance.

Next, we tested OKE with MI355X. This was the only provider other than TensorWave who let us test-drive AMD GPUs. The ride was rough.

As we complained about last time, at first, we needed to SSH into an operator node to steer Kubernetes. This time, the team quickly solved our Oracle CLI issues and we were promptly able to download our credentials locally. By far the greater problem on this cluster, though, was with the NICs. Peak performance was near line rate, so there was no issue with the hardware that we could detect. Rather, the issue was with the ionic drivers. Our per-rail `ib_write`/`ib_read` sweep across all eight NICs hit 391 Gb/s on every rail, then wedged the node the moment it finished: when the pods were deleted, the perftest processes never completed their `rdma_cm` teardown, the forced `kill` timed out the NIC’s destroy-CQ command, and the RDMA admin queue on `ionic_0` and `ionic_4` went dead. Worse, both ports still showed `ACTIVE` at 400 Gb/s and the node stayed `Ready`. The passive checks did not check out the RDMA control path, and no active check had run in eight days. We found the issue by grepping `dmesg` after RCCL kept failing. Oracle diagnosed it quickly on a weekend and rebooted: their own `ibwrite` health check had failed on this cluster in the same way days before. After a call where we explained the pathological workloads, they spoke with AMD and pushed different drivers and firmware to the cluster, which solved the problem.

Separately, we had a GPU intermittently dropping off the PCIe bus, neither caught by the health check nor properly surfaced in the dashboard. Oracle promptly swapped out the node and responded to our feedback on the dashboard.

This was a mixed episode, but Oracle’s responsiveness was good. It’s especially undesirable that no health checks alerted, even though these drivers are known to be flaky. Still, Oracle’s close relationship with AMD is clearly an asset. They were able to speak to the relevant teams, diagnose the issue, and put a patch in place quickly. This type of support is not guaranteed at organizations of Oracle’s scale. Interestingly, Oracle is also the only provider ranked Silver or above, and one of two ranked Bronze or above, with zero installed libraries with applicable CVEs. This is more of an expectation than a benefit for us, but it was a clear demonstration that their automation works for deploying and upgrading clusters.

If we were a neolab, we would come away from this testing experience with confidence in working with Oracle. Yet this is a moot point given Oracle’s increasing commitment to bare metal deployments for frontier labs such as OpenAI in Project Stargate. Success in that business depends more on getting government approval for gas pipeline construction than it does on RCCL tests.

Still, Oracle has made progress since last we tested them, and the experience is very good overall. We look forward to continuing to keep tabs on them going forward.

Silver

Lambda

Lambda was most recently in the news for participating in a $35B Anthropic deal for 350 MW in Nueces County, alongside Hut8 and Nvidia. Before that, they were rumored to be preparing for a 2027 IPO, and they separately raised $1B, $926M, and $1B in debt for their buildouts. Amidst all their growth, after judging them Silver in ClusterMAX 2.0, we were interested in evaluating Lambda’s clusters this time around.

Onboarding began with a nice deck and PDF explaining what to expect on the cluster, and access was sorted out promptly.

We criticized Lambda in ClusterMAX 2.0 for their reliability; this round of testing, it was one of Lambda’s points of emphasis. Their passive health checks cover the relevant conditions and ultimately autoremediated three errors we simulated during testing. Two synthetic XIDs were autoremediated in less than 15 minutes, and there were connected dashboards to monitor node health. Our test that triggered a genuine XID 79 by resetting the PCIe secondary bus reset brought the node into `NotReady` with no `GpuXid`, no cordon, and no monitoring visibility on the Kubernetes layer, but it ultimately rejoined the fleet after 2 hours. Our node-reboot tests on the Kubernetes layer also completed without incident.

The orchestration layer has improved significantly, though it still has a handful of issues. Slurm configuration left CPUs per task unset, so Slurm’s default allocation gave one logical core unless the launch requested more. Passwordless sudo is not enabled on Slurm, as we are using managed slurm rather than unmanaged Slurm. Lambda is the only provider we’ve encountered that makes this distinction, but clearly some of their customers want root access to their own cluster, while others do not (?). We enjoy having root on our own cluster.

Also missing was SSH to the compute nodes without an allocation. This creates a headache during debugging if someone else is running a job and you can’t attach to their allocation, or a job is stuck. Another reason why we enjoy “yolo mode” on our clusters.

More importantly, Slurm had software out of date, including CUDA Toolkit 12.0 at the default location /usr/bin/nvcc (which is so old it’s incompatible with the B300 SM100 architecture), and unacceptably old drivers. On the bright side, Slurm did have topology settings configured, and the NCCL recipes that they gave us worked well. We had no authentic errors during testing, and all our tests, including burn-ins and storage, showed good performance.

Our preoccupation with Lambda’s 1-Click Clusters seems quaint in today’s world of capacity reservations backlogged for months. Indeed, as mentioned in the opening, Lambda is increasingly driving its bottom line via bare-metal builds, apart from managed clusters altogether. Regardless, Lambda’s orchestration continues to improve, and customers are happy, which lands them in the upper ranks of ClusterMAX 3.0.

Microsoft

Using Azure clusters can be very difficult due to the Microsoft bureaucracy. Their engineers do the best they can, but this is not an organization set up to be flexible in accommodating its customers. Microsoft customers do not come first in matters of taste.

During testing, we were not allowed to have our own Azure accounts, so we had to use the cluster under the user of one of Azure’s engineers, which prevented us from poking around the console and experiencing the authentic onboarding experience. Our GPUs were set to be in Virginia, but thanks to some obscure internal decree, we were not allowed to have a control plane there. Instead, our control plane ran in San Antonio, Texas. Access required setting up a VPN, which worked after a bit (see: literally 3 weeks) of back and forth. Pasting was disabled on the bastion, so to run our tests, we had to manually type in our GitHub and HuggingFace tokens. Like, literally type them in one character at a time. We got them right first try though, nbd 😮‍💨.

Once everything finally ran, we found a handful of packages out of date, and bad default NCCL configs kept our first run from going through. On AKS, the `managed-csi-premium-v2` `StorageClass` could not attach, and we got `EAGAIN` trying to run `fio` at anything more than 4 clients on Azure Files Premium. No parallel system attached to our Slurm cluster meant that one of our tests, which loaded DeepSeek-V4 from NFS, timed out.

When we went to run burn-in, we had a genuine failure on 10 nodes of our Slurm rack, killing our job. Microsoft caught the errors, traced them back to an unhealthy host, and opened up a Guest Health Report for repair. An `scontrol` resume command brought them back, and after that, both our Kubernetes and Slurm racks completed their burn-in without error.

CycleCloud drained the node immediately after we tested Azure’s health checks by injecting a synthetic XID, but it did not resume or replace it. Because we had the whole GB300 rack, this was satisfactory, as described in the “Blackwell and Grace Blackwell” section above.

Due to the account issue, we could not see our Kubernetes dash by default. Instead, Azure’s engineering team cooked up a custom dashboard setup for us, and it was extremely detailed—a solid triumph over the bureaucratic straitjacket. We want to shoutout our friend Xu Xue for this one, as we are quite certain that he went way above and beyond the call of duty to build this out for us, but wouldn’t admit it. His curated dashboard is open sourced here, a great resource for anyone building Grafana dashboards to monitor Nvidia GPU clusters from scratch.

Satya once said “we want to build out Azure as being fantastic for the long tail of the workloads,” and that Microsoft is “not in the business of just doing five contracts with five customers being their bare-metal service.” Yet it is exceedingly rare to come across a VC-backed lab that uses Azure for training and inference. “Five customers for bare-metal service” is overstating it. Really, it’s two: OpenAI and MAI. Soon to be three, with Anthropic diversifying.

Azure’s most interesting asset is that it can serve OpenAI’s models and keep 100% of the revenue. They still have full access to all of OpenAI’s IP. Yet Azure can separately boast of large bare-metal deals with OpenAI, as well as an increasingly lucrative relationship with Anthropic. This is an incredibly strong position to be in. If only for the pause.

As Azure mostly competes on a different plane with enormous deployments and hundreds of billions of capex, its ability to manage clusters only makes it more flexible. Like Google and Oracle, Azure gets its cashflow from offtakers who need no help setting up a `ComputeDomain`. But based on the quality of testing, these three could still cut smaller, higher-margin deals with labs who want a higher-touch experience.

This does not seem likely to happen any time soon, though. Support and the console is a headache, even if Azure is the only hyperscaler where you can possibly get the Nvidia Reference Architecture on your scale-out networking. As much as anything, we like to verify that these hyperscalers are light on their feet.

Firmus

One of Nvidia’s favorite up-and-comers, and our favorite in Asia-Pacific, Firmus announced in August a $2B funding round at $10.5B post-money for expansion throughout Australia, Malaysia, and Indonesia, following a $505M round at $5.5B just 4 months before. Firmus also raised a $10B debt facility in March for its Project Southgate buildout in Melbourne and Tasmania. Just last week, it disclosed 900 MW in total contracted capacity and announced OpenAI as an anchor tenant in its Malaysian expansions. All in all, Firmus has disclosed 18,400 GB300 GPUs in the deployment process today and 36,800 later this year in Tasmania, with multiple GWs and tens of thousands of VR in the pipeline.

We had a rough takeoff but a fairly soft landing while testing one of Firmus’s dev clusters. Accessing our chips required using a Microsoft Entra account, Microsoft SSO, then downloading and configuring a custom pre-release version of the vCluster CLI. We’re suckers for new tech, but where cluster access is concerned, we’d much rather just SSH. After a few rounds of back and forth, with the help of Firmus’s team, we were on.

…on the cluster, but not quite off and running. The default setups on both Kubernetes and Slinky left a lot to be desired. Slurm login had no `sudo`, no `vim` or `nano`, HPC-X installed but not on `PATH`, and no NVCC. More importantly, instead of receiving 4 Slurm nodes on one rack and 4 Kubernetes on another, we had 2 of each on each. This was not necessarily a problem, but Kubernetes had `nvidia.com/gpu.clique` configured with nothing consuming it, while Slurm used `topology/flat`. Thus, the nodes visible to each orchestrator spanned multiple racks, and there was nothing to avoid a multinode job needlessly crossing the rack boundary. We had to infer node nomenclature to make sure our tests were well placed.

On top of this, the Kubernetes control plane’s `etcd` visibility flickered several times during testing. which killed our jobs and caused both the Slurm and Kubernetes layers to restart. This was ultimately attributed to an issue upgrading switch firmware and did not recur for the last 10 days. We also lost a NIC to a wedge during reboot, which was not caught by Firmus’s health checks as they were not active on the dev cluster we tested. Firmus was the only GB300 cluster we had that exposed GPU HBM as NUMA nodes, an interesting configuration that we mark as a failure because it lets you accidentally swamp device HBM if you overflow on the host. WAN was inconsistent but often quite poor, maintaining 0.25 Gb/s to NGC. The last of our problems was that one of the racks saw significantly degraded performance on its NVLink fabric, which was explained by a power setting that Firmus had been experimenting with on their dev cluster; fixing this brought our numbers into the healthy range.

Thus, by the time testing ended, we had satisfactory though underfurnished Slinky and Kubernetes layers, we knew how to access them, and we could get strong performance across our test suite. We understand that managed clusters are not Firmus’s priority right now, but we believe there is plenty of low-hanging fruit here for their technical team. As Firmus works to bring enormous capacity online, we look forward to seeing them continue to make life easier for users of their GPU software stack.

TensorWave

TensorWave is an AMD-only neocloud whose MI355X offering we tested via K8s and Slinky. In the last writeup, we emphasized the difficulty of TensorWave’s onboarding process, which required back and forth just to get on the cluster and had a large number speedbumps even still. The improvement this time round is significant. Onboarding was smooth, the `kubeconfig` sent directly to us worked perfectly, the console allowed us easily to add team members with SSH access, and the TensorWave support team was attentive throughout our testing.

Being an AMD-only cloud, TensorWave is fighting an uphill battle dealing with AMD’s software stack, which, despite significant recent progress, still lies well behind Nvidia’s. We saw this on our TensorWave cluster on RCCL microbenchmarks, with all-gather and all-to-all hanging at 32 KiB even using TensorWave’s binaries and recipes. Other collectives showed uneven scaling and poor throughput at certain message sizes. [Editorial comment: RCCL? More like Rick L!] In general, it can be said that the AMD networking stack lacks the community support that Nvidia tooling offers. The same goes, of course, on the hardware side. More important than microbenchmarks, we see in practice that the GB300 NVL72 systems have the most demand across the industry from labs: their performance is so superior that, even accounting for their higher prices, they often offer the best perf per dollar as well. AMD’s rackscale answer, the MI455X Helios, is still ramping up production, and is certainly not yet on offer by TensorWave. Thus, TensorWave’s place in the rankings depends not only on its internal improvements, but on AMD’s ongoing work to close the gap relative to Nvidia.

For what it advertised, our TensorWave cluster was good across the board. North-south bandwidth was strong. Storage performance was exceptional and burn-ins completed without a problem. When we pushed around the Kubernetes layer, its reliability was good, and the Slinky layer was easy to use.

The health check immediately recognized the error that we simulated, bringing the node into `drng`.

The dashboard oddly marked the node as “Allocated,” rather than being broken out into its own category. However, shortly afterward, we were presented with a nice juicy button to click to approve replacing our “sick” node with a fresh one.

The replacement completed in approximately 43m. We would prefer that the autoremediation occur by default—we’d rather not even have to click a nice juicy button—but otherwise consider TensorWave’s monitoring and autoremediation exemplary.

Overall, TensorWave is the leading AMD-exclusive neocloud, providing a comparable (and sometimes better) experience on AMD GPUs than hyperscalers like Oracle and Microsoft. And they are not stopping. TensorWave has entered into an agreement with Fermi for 222 MW in the Texas panhandle with expansion rights up to 650 MW, contingent upon Fermi securing project financing. It has a tailwind with a $350M June Series B, leaving it at $1.55B post-money, and is currently working on large deals with hyperscalers. With both OpenAI and Anthropic announcing strategic partnerships with AMD for the MI450 generation, we expect that TensorWave is set to benefit, and AMD-exclusive neoclouds may be able to break into the Gold tier. We look forward to testing TensorWave’s MI455X Helios once it is available.

GMI

GMI is a neocloud headquartered in Mountain View with roots in Taiwan. With a $500M Nvidia deal announced last November and $12B more this March, hundreds of GB300 racks in the pipeline, and plans for VR down the line, they are keeping on the frontier in terms of hardware. Recently, they have also been growing their fleet by making deals with smaller neoclouds and reselling the compute to their established customer base, which offers an attractive cashflow profile because it does not require up-front capital outlay. After judging them the “top Bronze neocloud” in ClusterMAX 2.0, we were interested to test the progress of their managed clusters.

Onboarding was a bit bumpy, including an empty Kubeconfig downloaded from the console and an SSH key that we uploaded but was never synced to the cluster. After some back and forth with the team, we got in. The problem, it turned out, was that the console was view-only for our testing purposes, so we could not verify its functioning. Last round, we did not get access to a self-service console at all, so we count this as progress.

As last time, the compute was up to spec, burn-in was clean, and the InfiniBand fabric was sturdy. One notable improvement since ClusterMAX 2.0 was in GMI’s storage. GMI provided a performant RWX NFS default StorageClass, which was also available on the Slurm layer at `/home`. We had difficulty trying GMI’s S3, which turned out to be due to a typo in their doc. But once the Brothers Karamazov Karasev hopped on a call and fixed it, we verified that the object storage perf was also good. The biggest blocker in terms of cluster configuration was that GMI expected the IMEX domain to be configured on the Kubernetes cluster by a ComputeDomain CRD. This is perfectly viable—many other highly rated providers do the same thing—but GMI’s Kueue would not tolerate the claim, so our jobs sat in `Suspended`. To work around, we had to submit our jobs in a namespace with no `LocalQueue` named `default`. We recommend that GMI document that MNNVL jobs need to skip the queue, or, much better, that GMI set up its Kueue to tolerate DRA resource claims.

GMI’s dashboards, which did not exist last time we tested, were useful, including hardware information from DCGM and NVLink and InfiniBand monitoring.

We could not get the full GMI health-check experience due to a lack of spare capacity, but what we could test performed well. Nodes were quickly cordoned at the Kubernetes layer, then rebooted and auto-uncordoned. The Slurm layer also brought a node into `DRAIN` soon after a synthetic error was injected. GMI had alerts set up in Slack, so we could see their system detecting the error and thread to discuss next steps with support, which was convenient.

Our GMI experience was solid in every crucial way, and we look forward to continuing to test them as their capacity ramps quickly.

Bronze

Amazon Web Services (AWS)

Our last report summarized the AWS experience as a “headache.” This time around, we had to break out some Amazon Basic Care Ibuprofen Tablets, Fever Reducer and Pain Relief from Body Aches, Headache, Arthritis Pain and More, Brown 200 Count to get through our testing.

The difficulty with AWS begins before you get on the cluster—indeed, before you even try to provision the cluster: it begins when you contemplate AWS’s zoo of managed offerings. To get this all straight, we had to make a Venn diagram:

(Caveat lector: anyone not interested in the minutiae of AWS’s GPU offerings should skip this ¶.) SageMaker is AWS’s fully managed platform meant for minimal maintenance. If you don’t want to handle any infrastructure, it offers serverless model customization and data science environments that abstract away the underlying compute. At the next level up is the SageMaker HyperPod line, which supports both Slurm and EKS. The HyperPod line offers many conveniences like health checks, autoremediation, dashboards, and workload-level features like checkpointless training out of the box for users who want to preserve their `ssh` or `kubectl` access. If you want infrastructure abstracted away but want to handle the ML stack yourself, you can leave the SageMaker umbrella and choose EKS Auto Mode, which takes care of node provisioning, autoscaling, OS patching, and node repair. EKS Auto Mode is a wrapper on top of “managed EC2 instances,” but these are not actual EC2 instances in the sense that they do not show up in the EC2 console and do not provide the same level of customizability as ordinary EC2 instances. In EKS Auto Mode, users do not have access to the control plane, and they must allow AWS to recycle their workers every 21 days for security. Finally, for even greater customizability, there is EKS + Karpenter, which allows the user to manage autoscaling themselves with the open-source tool Karpenter.

Despite Amazon’s reputation as “customer-obessed,” it is clear that this novel-length GPU menu is an example of the adage “you ship your org chart” rather than a reflection of genuine demand segmentation. And the bureaucratic frustrations hardly end once you’ve ordered your cluster. This is a criticism of the organization rather than of any team in particular—AWS has shades of the 3rd-Century Roman Empire. To improve its standing in the next round of ClusterMAX testing, we recommend AWS choose another empire and/or time period for reference.

Nevertheless, there is a unifying trait of all these offerings: they fail to provision the first time you try. Thus, we made another clarifying Venn diagram:

This finally brings us to the AWS operator experience. One SageMaker cluster we provisioned in the console failed due to insufficient IAM permissions. Some of the EKS clusters used Terraform to simplify provisioning, but that path is not battle-tested and requires being in the right subdirectory and on the right commit of the right branch of the right (in-flight) repo and knowing exactly which knobs to turn without making an invalid request. One cluster came up without Lustre and would not let us attach any, despite extended back-and-forth with AWS engineers. For the purposes of testing, we provisioned a new cluster in a new region to gather data on filesystem perf, but if we were a real-life lab, this would have been crippling, at least temporarily. The AWS team we worked with was consistently helpful, but it is clear that they don’t have the liberty to go solve problems or even insight into how the system fits together, especially compared to the flat, nimble neoclouds we’re judging them against. It should not require six engineers on multiple calls to provision a Slurm cluster in the first place. (Edit: AWS would like to clarify that it was five of their engineers and one of our engineers to make six.)

Once the goods were delivered, there were no blockers that kept us from getting reasonable perf out of the gate. The drivers, firmware, and most utilities were up to date. This is the bare minimum from a reputable provider like AWS, however, and there was no shortage of rough edges elsewhere.

A basic point of distinction between the AWS clusters we tested and their peer B200 clusters is that AWS uses Elastic Fabric Adapter (EFA), its proprietary scale-out fabric. In our testing, peak EFA throughput was strong. On a four-node 32-GPU all-to-all NCCL test over EFA on B200 we measured 59.44 GB/s busbw at 16 GiB message size, which works out to about 92% of scale-out line rate given the proportion of traffic that crosses the node boundary. On other networking tests with other traffic patterns and launching mechanisms, as long as message size was large enough, throughput was among the best B200 clusters. However, EFA consistently demonstrated worse latency than well tuned InfiniBand comps. For example, when the message size was reduced to 64 KiB, the same four-node 32-GPU all-to-all NCCL test took 85.26µs over EFA and under 50µs on InfiniBand peers; a delta of 30-45% between EFA and InfiniBand was consistent across small-message tests, which are latency-dominated. Although this may seem like a nitpick, scale-out latency is a critical metric for modern expert-parallel workloads, which frequently send small token vectors across the node boundary. This 64 KiB NCCL test, for example, is similar to the expert routing step of models like Kimi K3 and DeepSeek V4. Our findings are corroborated by an excellent Perplexity technical blog post about optimizing EFA, which reported similar results: EFA demonstrates satisfactory peak throughput but pays a latency penalty of 20µs or so on “the message sizes exchanged during MoE dispatch and combine.” (Note that this Perplexity post compares EFA to ConnectX-7 NICs whereas we tested ConnectX-8, but the nameplate throughput of the two generations is identical.) It’s not a good sign when Perplexity’s cracked engineers are writing painstakingly detailed blog posts about how to “enable” basic functionality on your custom network stack.

EFA still requires extra setup for expert-parallel inference in upstream vLLM and SGLang. AWS reports successful vLLM deployments, but users must install EFA userspace libraries and build the relevant communication components with EFA support. Depending on the workload, these include DeepEP, NIXL, or Mooncake Store. AWS’s DeepEP fork supports EFA through NCCL GIN. DeepEP V2 uses NCCL GIN, while its NVSHMEM-based V1 path is now documented as legacy. Upstream integration and default container packaging still require work. NCCL EP is another route under development; the linked vLLM integration remains a draft PR. In general, EFA is not a priority for frontier communications libraries, and users must wait weeks or months for what support they do get.

AWS is the only cloud distinguished in this way. Every other provider we tested uses InfiniBand or RoCE. Of course, AWS’s customers with GW deals have the resources to make sure their EFA is well tuned, but we still think it a shame that the largest provider’s scale-out stack is uniquely troublesome.

In a different way, storage was another basic cluster service that often proved uncooperative. The B200 cluster we provisioned in `ap-south-1a` failed every request for storage with an ambiguous `Insufficient capacity` error, and the `Failed` filesystems took 5h46m55s to delete.

To test Lustre performance required another cluster in another region, which meant another protracted bringup—another evening of terminal-watching trying to discern whether Terraform was making progress or not.

Claiming storage on AWS via Terraform is like raising a toddler: you can only watch and hope that once it’s done destroying it will start creating. And once the storage is created, it is not necessarily straightforward to access. On the Kubernetes layer, one cluster provided no StorageClass by default, and its only provided StorageClass left our PVC indefinitely in `Pending` because it had the wrong driver. This required us to define our own StorageClass with the correct provisioner.

Powered on and wired up, storage perf was mixed. The provided FSx stack struggled significantly with ordinary buffered I/O, reaching 9.2s in p99 sequential read latency on fio testing, compared to 312ms on the same metric with buffering disabled. Greater latency on the buffered setting is expected, but this is a pathological delta.

AWS’s health checks’ performance was, again, mixed. Our EKS Auto Mode cluster caught the XID we injected, set `AcceleratedHardwareReady=False`, and gave us a new and healthy node only 24m after the original injection. This is good. Meanwhile, our SageMaker HyperPod Slurm cluster immediately detected the XID, brought the node into `DRAIN`, and completed most of the reprovisioning process before tripping over its own feet when it required a deep health check before it could rejoin the fleet. This deep health check was submitted as a Slurm job and couldn’t be scheduled until, well, the node was added back to the fleet. Thus, an operator had to manually push through a `scontrol ... State=RESUME` command, at which point the fleet returned to full health. With our feedback, AWS promptly fixed this circular dependency, and a follow-up test on a single H100 node succeeded without intervention. Better.

Our EKS Karpenter cluster’s health checks were even more self-defeating. The `dcgm-server` DaemonSet did not tolerate the `nvidia.com/gpu:NoSchedule` taint, so it did not attach to the worker node, causing the monitoring agent to report `error connecting to nv-hostengine` and flip `AcceleratedHardwareReady=False` with `Reason=DCGMError`. Karpenter then marked the `NodeClaim` for termination, the replacement node was marked unhealthy for the same reason, and the system spun in circles until we manually modified the DaemonSet to tolerate the GPU taint. The health check had another allergic reaction when it noticed the IMEX daemon did not schedule. IMEX configures the NVLink fabric across nodes, but we had ordinary B200 nodes speaking via EFA, so there was nothing for IMEX to do: this daemon’s failure to land should have been harmless. Instead, the monitoring log emitted the schizophrenic message

ignoring IMEX health code on non-NVLink multi-node system, code=122

sending condition to exporter,

Reason=NvidiaFabricError,

Severity=Fatal

This would again have recycled a healthy node if we did not manually pin NodeRepair=false to keep overzealous garbage collection from blocking our work. (Edit: AWS informs us that this was not overzealous garbage collection; rather, it was a genuine hardware error with an unclear error message. This is worse. In any case, an `nvidia-smi -q` of ours reported “Fabric Health Summary: Healthy” after this supposed failure.) Finally, as our testing came to a close, Karpenter tore down our B200 cluster 39 minutes before the EndDate, force-evicting a 4-node MPIJob in progress. We have gone into such detail here because these bugs are almost unbelievably elementary for the world’s largest cloud. Perhaps the single painfree part of the AWS lifecycle was Karpenter scale-up and scale-down, which completed in minutes with no errors. However, GPU capacity must be secured in large blocks and far in advance, so snappy autoscaling with Karpenter is a meaningless feature.

Despite the many problems with their product, AWS’s specialists and engineers were intelligent and responsive in our interactions with them. They were eager to show what they had improved, sought out feedback on documentation, circled back to rectify errors in testing, and even built a nice, forward-looking MCP server whose skills we plundered for our own codebase. AWS’s issues lie in its organizational structure. The number of basic issues that have slipped through the cracks is astonishing in an organization with AWS’s resources.

In February, AWS announced the expansion of its deal with OpenAI by $100B; in 2Q26, it posted 37% growth in net sales YoY; its Trainium business has hit $25B in ARR with triple-digit YoY growth; its Anthropic investment is appreciating to the tune of tens of billions per year; Bedrock continues to make bed-rocking margins on bed-rocking income.

Managed clusters are not AWS’s priority.

Gcore

Gcore is a Luxembourg-ish neocloud building out in ambiguous region called « Europe, » where it relies on other providers for capacity. They have made no flashy announcements recently, though they are adding Blackwell to their Hopper fleet. In ClusterMAX 2.0, we gave Gcore a Silver rating after testing their Kubernetes and SonK offering via Soperator. This time round, we only tried the SonK again.

Our insight into Gcore’s capabilities was limited by their lack of capacity: most providers granted us at least 4 B300 nodes for at least a week, but Gcore was so constrained that they could only part ways with 3 H200s. Nevertheless, there was plenty to keep us on our toes on this cluster.

Our testing on the Soperator layer required several rounds of back-and-forth with the Gcore team to fix some rough edges, but the team consistently proved helpful. When the majority of our first sweep failed due to a misconfiguration that prevented non-root users from running their workloads, the team responded promptly, confirming `kernel.apparmor_restrict_unprivileged_userns=0` would get us unblocked. Gcore’s team was again responsive in correcting another poor default when each Slurm worker reported `RealMemory=2048 MiB`, apparently having erroneously inherited CPU configurations. This kept us from even running `nvidia-smi`, and we were glad to see it quickly repaired.

Debugging was made slightly more challenging on the Soperator layer because inter-node SSH was disabled by default, but the team pointed us to a `soperator-createuser` convenience script to make life easier. The team also troubleshot an error with their dashboard as we tested.

On the Kubernetes layer, there was no default StorageClass, so users had to select NFS explicitly. The storage on the Slurm layer was properly configured, with the easy-to-use Soperator default of NFS at `/`, providing the illusion of a single shared volume wired to all the nodes. This storage performed well across workloads. The N/S network was fine, and the E/W network ultimately performed up to spec, but the RDMA devices had non-standard names and no `topology.conf`, so configuration took extra steps.

The XID that we injected was quickly caught and displayed in Grafana; interestingly, the node went `Down` on the Slurm layer, but on the Kubernetes layer nothing changed, even though it was injected via K8s. We could not test autoremediation because Gcore had no spare capacity.

We were impressed by Gcore’s attentiveness and their Soperator offering that, after some massaging, did the basics right. We look forward to testing Gcore’s B300/GB300 management, including its handling of XDR ConnectX-8 NICs, and seeing the improvements of their Soperator and Kubernetes products.

GMO

GMO’s managed Slurm cluster delivered strong networking and storage performance on two B300 nodes. Its main gaps were restricted profiling, limited enterprise controls, and a lack of autoremediation.

Slurm arrived with a configured head node, partitions, and topology.conf. Drivers and CUDA/NCCL/HPC-X modules were current and consistent across nodes, and Pyxis was available. Onboarding was straightforward, though the Japanese-only console required some manual ctrl+c, ctrl+v translation for us to understand.

Grafana access took us five days of troubleshooting. We initially missed the macOS login-keychain instruction, then entered the certificate password where the system password was required. GMO investigated and repeatedly followed up. The final mistake was ours, but clearer certificate instructions would have saved time. Or, no unnecessary custom certificates to access the website. Japanese security theater strikes again.

The sixteen 400 Gb/s rails per node delivered strong NCCL results, though two nodes provide less scaling evidence than our usual four-node minimum. MPI launches needed explicit NCCL_IB_HCA settings, which GMO said it already supplied in nccl.conf, but which didn’t work for us initially. Burn-in showed no network errors, retries, or ECC errors, with normal temperatures and power.

Shared home/data/work directories worked out of the box. FIO sequential reads and writes, random reads, and torch.save were solid, built on a well-tuned DDN Lustre filesystem. However, our contrived tests such as `import torch` and vLLM Serve from shared storage performed poorly; GMO offered to investigate but we chalked it up to classic Lustre metadata performance tuning issues and moved on. Like many other Lustre implementations, the GMO storage offering lacked snapshots, automated backups, and disaster recovery options.

The cluster also lacked `sudo`, Docker on the workers, and required the use of snodes, a convenience script that they built to show metadata about the cluster, since they prevent users from running any scontrol commands on their own cluster. Security theater once again.

As a final step, SSH to worker nodes on the cluster was also blocked. And inside Slurm jobs, `RmProfilingAdminOnly=1` blocked Nsight Compute’s GPU counters, and perf was not usable for our profiling needs. We believe that managed clusters should allow users to profile their own workloads if they choose to go for the yolo-mode option. We wanted it, but GMO could not provide.

Finally, Grafana exposed useful DCGM metrics but lacked XID detection, estimated TFLOPS, SM Active, and SM Occupied views that we enjoy for performance and reliability monitoring. The Slurm job summary view was also pretty basic. Our synthetic XID 79 log injection produced no observed alert or node drain. GMO said it runs dcgmi health in the Slurm prolog, confirmed XID 79 was not whitelisted, and agreed to investigate for future. We finished our testing by determining that GMO lacked any autoremediation system in place, or at least we couldn’t verify it.

Overall, GMO’s lacking Kubernetes and enterprise features such as RBAC, SSO, and customer-accessible audit logs put it in the camp of niche, local player serving the Japanese market effectively. Our support experience with their team was helpful, but followed standard Japanese business hours. Going forward, we expect GMO to continue to be a leader in Japan, but lack the ambition required to expand their business beyond their local geography.

Verda

Helsinki-based Verda (formerly DataCrunch) now has $450M in total funding as of September 2026. They maintain a close relationship with Nvidia and continue to aggressively pursue financing. On the technical side, they have implemented a number of features since we tested them in November, and we are optimistic about their engineering team and roadmap.

Last round of testing, Verda’s Slurm was officially in beta, and they did not offer Kubernetes. This time, they offer a managed Kubernetes and have moved their Slurm offering to Slurm-on-Kubernetes to simplify deployment. We tested their cluster on both the Slurm and K8s layers; it had its fair share of rough edges, but represents significant progress and a strong starting point for continued development. We also found that security was solid.

The purchase and onboarding process was smooth, and the cluster provisioned without hiccups in 30m. The Grafana dashboard worked out of the box, and we SSHed onto the cluster without incident.

Once we got on the cluster, though, there were some bumps in the road. Our Slurm testing began with an interesting footgun: when we SSHed in, we assumed we were on the Slinky layer, but we actually had landed on the outer VM with wrappers that ran `salloc`, `sinfo`, and similar commands via `kubectl exec` into the `LoginSet`. The behavior was unusual—all Slurm commands escalated us to `root`, for example, and our Python environment disappeared—and it took an evening of debugging before we puzzled out that our Slurm access was illusory and a different SSH command would be required to access the Slinky layer properly.

Verda’s engineers were responsive, quickly hopping on a support call in which they reasonably explained the rationale behind these convenience functions and pointed us to the supported path. Although this particular feature caused us a headache, it’s bullish when engineers have gone out of their way to try to make their cluster easier to use.

The Slinky layer proved serviceable but imperfect. Verda ran a NCCL-test prolog that exceeded Slurm’s 10-second `MessageTimeout`, causing a worrisome though harmless `Prolog hung` alert every allocation. More significantly, Verda did not support Pyxis or Enroot, so we had to stand up our environment via a massive `.sif` file. Once our workaround was finished, we completed an 8-hour burn-in without incident and measured satisfactory numbers on our microbenchmarks.

Meanwhile, the Kubernetes layer could only be steered through the bastion, offering no way to download a `kubeconfig` through the console. This is a relatively simple fix to align Verda’s offering with users’ standard workflows.

The biggest shortcomings that surfaced during our performance sweep were of the shared filesystem. Gross throughput numbers were excellent, but cross-node file locks seemed simply not to be enforced. We found 100/100 exclusive-lock violations for both `flock()` and `fcntl()` across nodes, while same-node controls refused every attempt. This strongly suggests the issue is with `virtiofs` failing to publicize locking behavior. We also saw corrupted data when downloading with multiple clients from Hugging Face and `ENOENT` failures on Elbencho with 32 clients. This is more low-hanging fruit, and crucial given the fundamental importance of a reliable shared filesystem when running Slinky.

Lastly, Verda’s health checks programs are not yet fully configured. They are literal “health checks” in the sense that they evaluate whether a node is healthy, but there is nothing to consume their output—they exit `1`—“yup, this one’s dead”—and move on. A user must therefore bring their own reliability suite if they don’t want bad nodes silently killing their workflows. There is of course also no autoremediation provided by Verda.

Verda consistently communicated promptly and clearly and seemed to have engineers with the liberty and motivation to improve their product. On September 10, after our testing completed, they pushed a number of changes to Instant Clusters that appear meaningful. We look forward to seeing them continue to sand the edges of their Slinky and Kubernetes offerings, tune their storage system, and implement robust health checks and autoremediation.

Moonlite

Moonlite, a neocloud headquartered in Chicago, is a new inclusion as of ClusterMAX 3.0. With an ex-Crusoe founding team, Moonlite currently operates a Hopper fleet and has recently handed over its first B300s and GB300s, which it will be continuing to ramp up in the coming months. According to LinkedIn, Moonlite has the fewest employees of any company we worked with this round and oriented around engineering.

We tested Moonlite’s Kubernetes and Slinky. The handover process was smooth, with a nice onboarding document defining validated recipes and a Kubeconfig that gave us everything we needed. By default, everyone shared the same root user on Slinky; when we asked for non-root users, Moonlite made a `moonlite-adduser` convenience script in half an hour and baked it into the login-pod image same-day. This shows Moonlite’s responsiveness to feedback, but also their greenness. We loved their response. We would rather have a battle-tested system, ideally through the console, to handle basics like this.

On the hardware side, everything we tested worked well out of the box. NCCL tests hit ConnectX-7 specs, NFS was good, compute benchmarks were solid, and burn-in completed without a problem. However, it bears repeating that we were testing H100s, which by now are quite familiar to the industry and are therefore easier to get right than the GB300 NVL72s we had with other providers. Nonetheless, we were happy with our Hopper perf.

One component of the system that we could not test was the NVMe, which existed but could not be accessed. Nothing exposes it: no local-path `StorageClass`, `hostPath` unavailable, and Slurm workers are Slinky pods with only VAST `/home`. On the Kubernetes layer, there was no default `StorageClass` at all, and unqualified PVCs hung `Pending`.

We tested Moonlite’s health and monitoring system via a PCIe secondary bus reset, and Moonlite alerted us in 11 minutes. With our permission, they cordoned the node, ran diags, and returned it to our fleet after 57 minutes total, including a few minutes where they waited for our instruction on how to proceed. In practice, with actual customers, they define policies ahead of time so that their team can follow a runbook depending on the particular error. Moonlite’s typical SLA is to respond within 15 minutes as their team receives alerts in Slack and PagerDuty. We hear from their customers that so far that this is exactly what they do.

In our view, any autoremediation process that brings us back to a full fleet within an hour is good enough. But while manual processes are fine, at scale, we want to see this automated. We would also prefer greater visibility on the cluster in the event of an error. The only change visible to the user was that the node, while remaining `Ready` on the Kubernetes layer, advertised 7 GPUs instead of 8 in the time between the reset and Moonlite’s manual intervention.

The Slinky layer lacked any container runtime on the workers but otherwise got the job done, and had most of what we ask for on the login pod. Moonlite’s Grafana was fine, but it lacked detail at the scheduling layer, and does not display the most crucial information, which is error status.

What Moonlite attempted to do, it did well. We look forward to testing them again as their products mature and they get some GB300 calluses on their hands.

Together

In ClusterMAX 2.0, we wrote the following about Together:

Together is a strong provider with a robust cluster offering for both Slurm and kubernetes, but it is held back from the gold category due to reliability issues.

This round, Together’s issues—generally but not exclusively having to do with reliability—are cause for downgrading it to Bronze.

Two nodes of ours failed during a compute test; one came back, but the other was sent to RMA without notice, so we were left wondering where it went. Later in testing, Together’s Slurm `HealthCheckProgram` tried to run `dcgmi diag -r 1` under a 45s timeout on nodes that were already scheduled; the diag did not land, the timeout was misread as a failure, and the nodes were drained mid-run. Together quickly fixed this with our feedback. Separately, one of our nodes could not access VAST because its storage NICs were in `operstate=down`, which Together explained was due to a race condition in NIC initialization in provisioning their new lightweight VM stack. There was nothing to catch this error, and manual intervention was required to reprovision a healthy node and bring our test fleet back up to 4. While we like the sound of “reduced cluster provisioning time,” we obviously don’t like to see it affecting reliability.

Together’s health checks did work correctly in some instances, showing progress since ClusterMAX 2.0. When we triggered a PCIe secondary-bus reset on the Slurm cluster, the node was drained in 67s, and an automatic node replacement completed in 35m7s without manual intervention. A synthetic error also cordoned a node on the Kubernetes layer and correctly did not trigger autoremediation because we had `Auto Repair` turned off.

The Together Kubernetes and Slinky layers were both functional, though with a handful of challenges. Interestingly, Together advises to avoid using the Kubernetes layer of the Slinky cluster: to convert Slinky nodes to K8s, we were instructed to tear them down and reprovision as vanilla Kubernetes. The Slinky offering should therefore be evaluated as plain Slurm. Summarizing its pain points as briefly as possible: OCI image extraction failed because of default settings on `/tmp`, so we had to redirect extraction to `/scratch`; when this happened, Slurm exited `0` anyway; NFS rejected any filename containing `:`; and non-login worker shells have neither `nvcc` nor `mpirun` on `PATH`, so ordinary batch scripts behave differently from interactive setups and lack the tooling that they need. Together also offered S3, but it was slow and unreliable. None of these is fatal, but adding them up, along with other more minor errors, meant a few dev days before we were able to run our stuff unimpeded.

A public IP check confirmed that this flaky B300 cluster had org `AS53735 IREN`. This N+0 Prince George datacenter is one of two datacenters in the industry notorious for unreliability, along with Crusoe’s cursed Reykjavik site. We strongly suspect that one or both of these datacenters was not inaugurated with a Land Acknowledgement. We know of multiple exceptionally unhappy customers of Together struggling with link flaps, power issues, cluster access being shutoff, random upgrades that aren’t approved by the users going wrong and not being rolled back, and weekend-long outages that eventually get blamed on a single ISP (yes, a single ISP at this site—no redundancy in Northern Canada). Maybe their other sites are better but the TogetherAI site at IREN Prince George is completely bad.

After playing the blame game for some time, it is clear that reliability issues cannot be blamed entirely on underlying providers. Unforced errors from customer support engineers have created a lack of trust between customer and provider.

Of course, managed clusters are not Together’s bread and butter, nor are they its primary source of revenue. When, in July, they announced an $800M Series C, the post made passing reference (at best) to their managed clusters business, emphasizing, on the contrary, that “Together AI is a research-driven company.” The offtakers of their 250MW in Saudi are as yet undisclosed, but we are aware that Together would prefer to get paid per token, not per GPU-hour. Despite this, we hope that Together manages to Get It Together, and address its cluster reliability issues to make its Slurm and Kubernetes products something to be proud of.

Crusoe

The Crusoe section of ClusterMAX 2.0 ended on a cautious note:

Crusoe is at risk of being downgraded to ClusterMAX Silver due to many of their top individual contributor engineers quitting, leaving the culture in their cloud division beginning to resemble big tech. There are too many middle managers across the organization, especially in engineering. This has caused incredibly slow moving releases, such as their AutoClusters feature, leaving us with concerns about the future of Crusoe’s public cloud offerings. Chase needs to do a rapid course correction if he doesn’t want to lose all of his 10x engineers and eventually lose their Neocloud business with it.

In the eight months and change since we published that paragraph, Crusoe has had its fair share of wins. Its valuation has grown from around $10B to $30.9B, it has announced 900MW and 1.0GW campuses in Texas and many more elsewhere, and it remains an innovator in the wild wild west of datacenter construction. They have the largest real pipeline of any datacenter builder.

Its inference endpoints business is also extremely highly regarded, following the acquisition of some cracked Israeli engineers from Atero. They have also landed awesome deals with firms like Jane Street.

Plenty to keep their hands full and business churning, some clusters are amazing, but others from them are terrible.

Yet this round of ClusterMAX testing finds that Crusoe’s neocloud business has not course-corrected rapidly enough. The problems go back to the fundamentals: reliability and hygiene.

We emphasize health checks in our expectations and our write-ups, but genuine hardware failures are rare in testing because our ClusterMAX allocations are too small and short-lived. Crusoe bucks this trend. On an H100 cluster we had in May to test AutoClusters, we saw an impressive barrage of XIDs—more real errors on one cluster over five days than we saw in the rest of testing combined.

In a pattern that would repeat throughout testing, exceptionally poor performance in some areas was paired with mature services in others: this storm of XIDs was accompanied by convenient emails, attractively styled, politely notifying us of the errors.

In addition to being responsible for the majority of this round’s hardware failures, Crusoe bears the distinction of having had the oldest Linux kernel and the oldest GPU drivers we tested. Crusoe has since rectified these shortcomings, but at the time of testing there were several software versions that were well below our minimum expectations: the aforementioned NVIDIA drivers failed our `cmax audit security`, along with Docker, ConnectX firmware, and `runc`. These issues speak to a lax security posture and poor engineering hygiene.

Nor are these software-version criticisms merely academic. The Linux kernel that our B300 cluster ran, `5.15.0-185-generic`, was earlier than a Linux patch series called “writeback: Avoid lockups when switching inodes.” The particular problem that our kernel experienced is well explained by the description of commit `e1b849c`, which fixed it for upstream kernels:
```
There can be multiple inode switch works that are trying to switch inodes to / from the same wb. This can happen in particular if some cgroup exits which owns many (thousands) inodes and we need to switch them all. In this case several inode_switch_wbs_work_fn() instances will be just spinning on the same wb->list_lock while only one of them makes forward progress. This wastes CPU cycles and quickly leads to softlockup reports and unusable system.

```

When might a `cgroup` that owns thousands of `inode`s exit? One answer: when a Slurm job is torn down. In our case, as an ordinary workload wrapped up, every CPU on the NUMA node ended up stuck contending for the same `list_lock`, making the processor unusable. The local NFS client, which happened to be on the frozen NUMA node, couldn’t get CPU time, and a liveness probe for that client timed out and reported the node unhealthy. Kubelet tried to control the damage with a `SIGTERM` and a `StopContainer` operation, but the kernel was taking its sweet time doing the laundry—reassigning dirty `inodes`—and took over two hours before it could honor the termination requests. The final result was another `NotReady` email from Crusoe—this one, uniquely, downstream of an old CPU kernel.

By the end of our testing, this kernel was not merely old and unperformant: it was officially marked a security vulnerability by CVE-2026-64378, which exploited the buggy writeback behavior, published on July 25 with severity 7.8; a separate 7.8 for this kernel was posted on July 27. Thus, we added the Linux kernel to our collection of Crusoe software exposed to CVEs. Crusoe has informed us that they will be upgrading to a 6-major Linux kernel, a necessary patch.

When we ran our XID injection on Crusoe’s Kubernetes layer, the error was quickly detected and a `Node Replacement` workflow was triggered, but the node could not be put in `drain` due to a mismatch in Slurm naming convention that slipped through Crusoe’s CI. The health monitoring system spun its wheels, repeatedly catching the error and trying but failing to mark the node unhealthy. Crusoe’s engineering team diagnosed the error impressively quickly, rolling out a replacement within hours. However, our replacement node had faulty storage NICs, immediately bringing it into `drain`. This issue persisted after the node’s VM was manually reset. Crusoe’s engineering team has informed us that the failure of AutoClusters to remediate in this latter event was “a known gap,” and that they will be “rolling out auto-remediation actions for this in a few weeks.” This is a surprisingly lethargic response to a basic failure of reliability.

Last and briefly, it bears mentioning that this cluster’s WAN was among the slowest we tested, clocking in between 0.47 Gbps and 0.78 Gbps depending on the context. Iceland has some skinny cables getting off the island.

All these problems add up to a cluster notorious for its flakiness, as confirmed by our friends at frontier labs who complain about both the number of failures and the apparent lack of interest from Crusoe engineers to fix the issues promptly. To be clear, we at one point viewed Crusoe as the #2 neocloud in the world, and as a result the #2 place in the world to rent GPUs. We made recommendations to this effect. Oh, how the mighty have fallen.

But while Crusoe’s machine images are badly due for a refresh, and their engineers can’t get Claude Tags or Codex approved, they do have automations like this set up in their internal Slack:

Does a CVE from November of 2025 count as “historical baggage”?

In all seriousness, this is a problem of priorities, a problem of bureaucracy run amok. We are not sure what to make of job postings like the above one for a senior Linux kernel engineer. Literally, we’re just asking everyone to keep things up to date. You don’t need to hire Linus to understand why this makes sense. So while it’s cool to see Crusoe flexing its muscles and paying top dollar for a good person, it’s not a lack of people or funding or technical skill that has brought Crusoe here in the first place. Our complaints focus on the most basic issues possible, not the Slinky configuration or console or CLI that no doubt take up much more engineering time and that we enjoy using.

We enjoy working with Crusoe’s engineering team and believe they have plenty of talent to improve their standing. Frankly, in the last few months, things seem to be getting back on track. Engineers are locked in, pushing code, and excited about upcoming releases. The Atero acquisition has been worth its weight in gold, skyrocketing Crusoe to the top of many buyer’s lists when it comes to inference endpoint quality. We love to see it.

Still, in the managed clusters business, TCO comes down to goodput, and goodput comes down to reliability, and Crusoe’s clusters are exceptionally unreliable.

Crusoe comes in towards the bottom of our Bronze category for this round.

Prime Intellect

Prime Intellect closed a $130M Series A this July at a $1B valuation, disclosing $100M of “annualized revenue run rate across compute, RL and post-training, sandboxes, inference, environments, and evaluations.”

We tested Slurm and Kubernetes B300 clusters and mostly enjoyed both of Prime Intellect’s orchestration layers, although each was missing a handful of packages we would’ve liked to have. The huge Slurm control plane was nice and snappy, having machines with 2TB of RAM for some reason. Plenty of swap space to run salloc.

After a quick fix to standardize the NCCL bootstrap interface, our networking tests ran up to 13 nodes on the ConnectX-8 NICs with excellent throughput. Storage was no issue, with Weka /data volume performing very well at 4 nodes and scaling steadily beyond. PrimeIntellect’s managed Grafana was one of the nicer ones we encountered. However, we were on a cluster that was still going through burn-in, so we saw a steady stream of errors. An easy way to test the monitoring and health checks on the system!

First, a node on our Kubernetes cluster was marked NotReady, apparently due to on-site techs monkeying with adjacent servers; it returned within 5 minutes. Then, during a PyTorch networking test, a different Kubernetes node stopped posting kubelet status and the distributed job could not complete; the node came back after a reboot and the job completed. The next day, 2 Slurm nodes went down, one with NVLink issues and the other stuck after a reboot. Both returned promptly after intervention, but the latter was soon back to its shenanigans with another surprise reboot, causing a storage test to fail. An MPI test aborted in its last phase, and the corresponding Slurm job remained in COMPLETING as all workers remained unreachable in the brief period before we handed the cluster back.

All of these reliability issues can be blamed on the fact that we got on this cluster in a small window while they were still doing burn-in for a paying customer. Fair enough. But the lack of compute at Prime’s disposal that would allow them hold back hot spares points to a bigger problem of growing capacity constraints. Prime is a reseller of other’s compute, and depends on their underlying providers to guarantee customers with a high quality experience. This comes with challenges that high quality training/RL frameworks, dashboards, and maniacal Slack engagement from a cracked team of engineers can’t always cover up.

With that said, Prime’s health checks performed impressively throughout, quickly intervening and limiting the damage of each error. We thought the system’s default responses were sensible and even liked the Slack alerts, which the team promptly followed up on. Given that Prime gets its capacity from a number of providers, the quality of their software layer is important for labs to feel comfortable with what they’re getting. Of the marketplaces we’ve tested, Prime seems to be the best.

This leaves Prime Intellect with a straightforward assignment, and one that we cannot have direct insight into: improve reliability, build a base-load of compute capacity, and rise in the rankings as a true neocloud. We have no doubts about the team’s technical ability; they have done a lot of hard things correctly on this cluster, and are the only provider on this list that injects real experience training models into their approach to software and support. For better or worse, their work elsewhere is much more impressive than the bread-and-butter managed compute offering evaluated here.

We got a preview of their hosted training product during this period and are very impressed. RL infrastructure is decidedly more complex than a simple managed Slurm or Kubernetes cluster, since its built on top of the same foundation but with three components that need to stay in sync:

1. Training

2. Inference

3. Sandboxes/environments

Coming soon, we will be evaluating hosted training infrastructure providers and RLaaS companies for hire through a project provisionally called PostTrainingX. If we had to put out an initial cut at those rankings, Prime would be in the hunt for Platinum! Stay tuned.

DigitalOcean

This marks the first round of testing that we have been able to get the full DigitalOcean developer experience. Marketing itself as “the first cloud built end-to-end for the inference and agentic era,” DigitalOcean distinguishes itself from “bare-metal focused neo-clouds and inference wrappers that lack cloud platforms.” In fewer words: DigitalOcean sells, among other things, managed clusters.

DigitalOcean’s managed clusters are orchestrated by Kubernetes, which proved solid but slightly immature. Their onboarding doc asked us to install Multus ourselves, then create network attachments, install MPI Operator, and create the MPIJob manifest. There’s no reason not to set this up for the customer in advance. Once configured, our cluster performed well, hitting all the standard benchmarks for compute and XDR networking, and completing burn-in without incident. The cluster also lacked a RWX StorageClass out of the box, adding another basic setup step that the better K8s providers handle for you. Once the storage was configured, its performance was interesting: sequential writes were approximately 6x faster than sequential reads at several different client counts, the inverse of what we see on most clusters. We expect that DigitalOcean’s sequential read performance could be significantly improved with retuning.

When we went to test the health checks, our first XID injection went unnoticed: the monitoring system did not detect anything, and the node remained available for scheduling. When we alerted the DigitalOcean team, they responded promptly, found the bug in their system, and asked us to reperform the test later in the week. The second try went smoothly, and we had a fresh node 50 minutes after injection.

When a neocloud is “built end-to-end for [...] inference,” that means it doesn’t offer Slurm. The DigitalOcean team helpfully offered us a runbook to stand up Slinky, and it got the basics right, but would require significant work before it could be considered production-ready. No SSH access, nor, as mentioned previously, any RWX volume by default, which means that there is nowhere to keep a `/shared`. Anyone who plans to run Slurm on DigitalOcean should expect to stand up their orchestration themselves.

DigitalOcean will be adding 60 MW of capacity throughout 2027, and, aside from its managed K8s, it has an interesting selection of GPU services like Droplets and bare-metal Hopper and MI300X that fall outside the scope of ClusterMAX testing. But no plans for GB’s or VR’s from what we can see at this point. We look forward to the team continuing to improve its managed Kubernetes service the next time we test.

Hyperstack

Hyperstack, under parent NexGen Cloud, is a UK-based neocloud that advertises Kubernetes clusters in the US, Canada, and Norway. It recently raised $45M at a $354M valuation with plans for many AI new products for “full-cycle development.” As it announced a $34M debt facility to build out a B200 fleet, it also disclosed plans to deploy 4,500 B300s later this year and for an additional 56 MW online in 2027.

The managed Kubernetes cluster that we tested marks significant progress since ClusterMAX 2.0, and a solid foundation for its continued growth. Compute and networking tests were up to spec, and burn-in sustained good numbers with no hard errors. Our Kubernetes node-reboot recovery was quick, and we had no trouble mounting and unmounting PersistentVolumes. The N/S network was fine, and storage was good enough, although we were not able to get Weka and VAST, which Hyperstack usually offers, on our timeline.

Our synthetic XID was detected in under a second with a `NvidiaFatalXid=True` flag. The test was awkward as Hyperstack had no extra capacity, so we had to take a node out of our fleet to simulate a spare. The support team notified us that they had detected the error 12m after we injected it, and our spare rejoined the fleet after a total of 52m. Interestingly, at this point, the “sick” node was still not cordoned, which took until 1h30m to register on the Kubernetes layer. This was all driven by manual intervention by Hyperstack’s SRE team after our synthetic error triggered Grafana alerts.

Overall, this test was a success, and we appreciate Hyperstack’s commitment to helping it go smoothly. However, as the team is aware, the amount of human support required makes us worry about its replicability. Hyperstack offers 24/7 support and targets intervention in under 60m in the event of single-node failure, but it’s better not to have to count on another engineering team pushing the right buttons before your cluster gets back to full health.

The only outright error during our testing was that two pods, `cilium-envoy` and `csi-hyperstack-node`, crash-looped with `Too many open files`. This was a minor annoyance and did not affect our workflows.

We look forward to continuing to work with Hyperstack as they grow their team, gain experience managing larger clusters, and continue to automate and harden their infrastructure.

Participation Ribbon

Vultr

Vultr advertises itself as “the world’s largest privately-held cloud infrastructure company,” which they are not, and “the world’s largest privately held hyperscaler,” which they wouldn’t be even if they were. Still, they are a relevant player in neocloud land, with modern GB300s and MI355X including a claimed 50 MW AMD site in Ohio. In ClusterMAX 2.0, Vultr’s cluster was delivered with a handful of basic errors. Unfortunately, the pattern is the same almost a year later.

The images that Vultr built for our testing had many libraries with applicable CVEs, including CUDA, DCGM, Docker, runc, and ConnectX firmware. Our Slinky login pod did not mount `/shared` and lacked the Pyxis plugstack config, so we had to orchestrate from a worker pod until the Vultr support team reprovisioned the login pod mid-campaign, fixing both issues. There was another basic problem with the storage visible from the K8s layer: the default storage configuration was incompatible with our bare-metal cluster. The default StorageClass, `vultr-block-storage`, provisioned and bound without complaint, then the pod sat in `ContainerCreating` forever. However, the Lustre tier, which came attached to the worker nodes, was excellent, achieving 91.7 GB/s aggregate read across 32 clients. Grafana was configured but only partially correct, as NVLink read 0 due to unconfigured `DCGM_FI_PROF_NVLINK_*` fields.

Vultr is known to have been experiencing issues with the reliability in its scale-out networking, which has discouraged partners from beginning or expanding existing deals. We recommend that Vultr continue to improve the sturdiness of its clusters and work to create golden images with all relevant utilities and orchestration reasonably configured and up to date.

Vessl

Vessl is a new Korean neocloud who fancies itself a “multi cloud orchestrator” rather than a broker, buying bare metal long-term and providing a management layer with SLAs on top. They claim 5,000 GPUs currently in operation and have an ambitious target of 100 MW by the end of 2027. An old Vessl BizOps job posting says to expect “Speed. We MUST move fast, learn fast, and iterate constantly,” demanding “a minimum of 60 hours per week.”

We love the ambition; however, the immaturity of Vessl’s cluster, which we sampled in its Kubernetes and Slinky flavors, gives them plenty to work on. Our onboarding began with a notice to pin three settings to render the RDMA fabric usable. A heads-up is better than nothing, but we much prefer to be given a cluster that performs well out of the box.

The third bullet was in response to a particularly cumbersome configuration: by default, due to Vessl’s Kubernetes `LimitRange`, each container would hard-fail if it used more than 2GB of host memory. Although a defensive default might make sense in certain cases, this is a poor setting. Nodes rarely are called on to run more than one job at a time, so containers can expect to have all of the host DRAM to play with—hundreds of GB—and OOM-killing them for going above 2GB is stifling. We recommend that Vessl relax this limit, bringing it in line with the physical capacity of the host while also making failure more graceful.

In terms of management, Vessl provided no way to handle users on the Slinky layer except via traditional Linux tools on the machine. The SSH key we added from the console did not show up on the cluster, nor did the one we added via the Vessl CLI; we later learned that these utilities are independent of Vessl’s Blackwell clusters. We could therefore use only the private key and `kubeconfig` DMed to us on Slack.

The cluster provided a reasonable (though strange) `add-user` convenience script to make new Linux users.

It did not provide, however, basic packages like `pip`, `git`, or even `sudo`.

The cluster therefore took some work to get to the starting line. But our intern was still not done putting in reps as Linux sysadmin. Vessl ran the Slurm login node with an ephemeral OverlayFS root, so runtime changes to `/etc/passwd` and `/usr/bin` existed only in the container’s writable layer and were not preserved when Kubernetes recycled the pod. This caused our users to disappear midway through testing.

Another Slinky misconfiguration brought down the Slurm layer for part of testing. Our workloads wrote to an overlay that they did not clean up, accumulating 863 GiB in the root volume, causing Kubernetes to mark the node `DiskPressure` and restart it. This not only brought down the local worker, but the Slurm controller daemon as well, rendering the whole Slurm layer intermittently inaccessible. We infer that `slurmctld` was scheduled on to a node also running `slurmd`. Co-scheduling a `slurmd-pyxis` pod with a `slurmctld` is generally a bad practice for reasons well illustrated by this anecdote: the data plane should not be bogged down by the administrative duties of the control plane, and the control plane should not be within the blast radius of the data plane. Vessl quickly stopped the bleeding by having Enroot write to a larger volume, but to our knowledge the root cause—the co-scheduled worker and controller—remains unaddressed. It is clear that Vessl does not have experience running Slinky at scale in production.

A separate error made Kubernetes access inconsistent. The URL in our `kubeconfig`’s `server` field did not always resolve to the same IP address: Vessl had erroneously published the private `10.x` IP to the record set in addition to the public one. Sometimes our `kubectl` command resolved to the right (public) IP, and we could access, but other times it would resolve to the wrong one and our request would hang until a retry went through. We recommend that Vessl remove the `10.x` IP from the public record set.

The `NetworkPolicy` we set on the Kubernetes layer of our Vessl cluster was accepted but not enforced, as HTTP requests between all tenant pods were accepted without regard to defined rules. Vessl did not have a health check program configured and should prioritize this along with all the mundane “quality of life” features indicated above.

The hardware itself that Vessl offered performed well throughout testing, including on NCCL tests over InfiniBand. Given that it has such a large fleet advertised as coming online soon, Vessl has every motivation to harden its offering, providing superior services to its users and enabling it to command superior prices. We will be interested to observe Vessl’s progress when we next test.

Runpod

Runpod is a neocloud headquartered in Moorestown, New Jersey with a fleet including RTX PRO 6000, H100, H200, B200, and B300. This June, they raised $100M at a $1B valuation, announcing on Twitter that their revenue doubled to $240M ARR from February to June.

The cluster we were able to test was in Seattle, and was so new that it did not even have storage configured yet. Our “Instant Cluster” was just about instant, spinning up in under 2 minutes.

Runpod’s user model is unusual. In an ideal world, we manage permissions via a web console with RBAC, where we can add our team members by email, increase or decrease privileges, make groups, and drop in our SSH keys to be automatically synced to the cluster. Instead, taking advantage of its snappy provisioning, Runpod recommends minting Instant Clusters as needed. Engineers then operate their cluster with a single tenant and return them to the pool when finished. This means that budgeting, monitoring, and other such policies are handled above the cluster level, rather than via traditional tools like Linux user management or Slurm accounting. Runpod’s design choice is viable, but it is revealing we did not end up using the cluster in the suggested manner, preferring to fall back to the more familiar pattern of making Linux accounts tied to our SSH keys.

Our first burn-in failed as one of our nodes had port 1 links go down on two NICs, suspected to be due to an issue with a leaf switch. There was no health check to catch the error, but there was a Grafana dashboard that highlighted the error if you know where to look. With our feedback, the Runpod team quickly improved the dashboard to make InfiniBand state easier to spot.

On the second burn-in attempt, the networking performed well and the test completed without error.

The cluster was so new that there was no shared filesystem to test. On the software side, though, our initial audit found many things to be improved. The CUDA Toolkit, GPU driver, and NIC firmware were deprecated for security reasons, and the CUDA Toolkit was additionally a liability because 12.8 does not support SM103, the B300 SM architecture. This caused some of our benchmarks to fail on the first try. The cluster was also missing lots of expected tooling, with no container runtime and many basic HPC packages absent. Slurm also occasionally marked waiting jobs as `InvalidAccount`, which prevented jobs from running, even though `AccountingStorageEnforce=none` and `AllowAccounts=ALL`. The workaround was to spam `scontrol update JobId=<jobid> Account=me` until Slurm accepted.

We also saw a node drained when an image extraction filled our 50 GB root overlay, which was shared with `/var/spool/slurmd`, leaving `slurmd` unable to write. Our tenant lacked credentials to `scontrol update`, so our testing continued shorthanded. Separately, Linux rejected `unshare -Ur`, so we had to run with `udocker` instead. Our standard XID injection route was blocked because we lacked permission to write to `/dev/kmsg`; we had no mechanism to issue node reboots; passwordless `sudo` was not enabled by default. The theme here is clear: within the Runpod environment, users lack many permissions they need to get their work done.

We find the Runpod team unusually receptive to feedback and eager to explain their decisions. We like that the majority of their job postings are in serious engineering roles, and we hope they continue to work to strike a better balance between ease of access and developer control.

Bitdeer

Bitdeer was a new inclusion in ClusterMAX testing as of ClusterMAX 2.1. Yet another crypto miner pivoting to AI, Bitdeer already has an impressive 1.75GW in total electrical capacity according to its most recent announcement. However, this mostly feeds cryptocurrency mining rigs: our SemiAnalysis Datacenter Industry Model estimates they only have a handful of AI MW online. Still, Bitdeer is steering hard toward the higher-margin GPU business, with hundreds of MW already in the pipeline over the next couple of years as it works on sites, either converted or now, in Knoxville, Wenatchee, Fox Creek, and Rockdale, as well as in Norway and Malaysia. Hoping to grow its business vertically as well as horizontally, Bitdeer is not content with mere colo deals, with fledgling offerings from token-as-a-service to managed agents to, of course, managed clusters.

We spent a lot of time trying to figure out what exactly Bitdeer expected us to test when they gave us credits on their public console. Health checks were not set up, the monitoring dashboard did not work, many basic libraries like PyTorch and Docker were not installed, networking settings were empty, IMEX was not configured, and the console did not support ordinary conveniences like RBAC. Bitdeer’s monitoring installer downloaded a binary for the wrong CPU architecture. SSH worked eventually, but oddly required an RSA key rather than ed25519. Kubernetes queries repeatedly lost their connection. In short, the management of this cluster was strictly nominal. The most that can be said is that it had GPU drivers, NCCL, CUDA, and an OS and kernel, and that all were sufficiently modern.

Once we had the time to set things up, performance was fine. GPUs hit expected numbers on burn-in and NVLink supported the expected traffic.

However, the cluster’s WAN performance was exceptionally bad, averaging around 0.1 GB/s across several tests. The cause of this poor performance is unknown to us; needless to say, it would be an impediment to real-world usage.

Most of all, getting support was a massive pain. The entire technical team was based in Asia (Malaysia, or Singapore in our case), which meant 8-12h TAT when we were trying to troubleshoot issues. It was impossible to consider Bitdeer for anything other than this tier.

We look forward to seeing Bitdeer’s improvements in future rounds of testing as its managed cloud offering matures and it implements features like health checks and managed Kubernetes. It will be interesting to see where Bitdeer allocates its time and attention, as management has unambiguously announced that its neocloud business is not its main focus: “Executing on colocation lease agreements is THE top priority.”

Shadeform

Shadeform continues to focus on brokering and small scale marketplace reselling of individual GPU VMs with a small team. We enjoy the interface and working with the Shadeform team, but without more ambition they will stay in our lowest tier.

Radiant

Radiant was formed after Brookfield acquired Ori, a UK-based neocloud that we discussed in ClusterMAX 2.1. Unfortunately we have not been able to test a working, secure cluster. Despite almost a year of planning, along with quite a bit of marketing effort, and announcements that mention Billions and Gigawatts, Radiant is yet to deploy a Blackwell GPU.

FPT

FPT suffered from a number of security issues during our testing, which we described in ClusterMAX 2.1. We have yet to re-test and are not aware of any Blackwell capacity.

Core42

Core42 is a dark horse in this category. They have a solid team, allocation of GPUs, political backing, and an infinite money glitch in the form of Mubadala. We expect to see them put the pieces together and rocket up the rankings as they enter the US market.

Latitude

Latitude is making progress to keep their modest cloud business afloat, being recently acquired by Megaport and as a result going public. We had the chance to test some RTX Pro 6000 Blackwell servers, but as long as Latitude is missing the latest and greatest GPUs, we expect them to be stuck in this category.

IBM Cloud

We have not had the opportunity to re-try IBM Cloud since our horrendous experience in ClusterMAX 2.0.

BuzzHPC

Buzz is making waves in Canada, latching onto the Sovereign AI wave associated with the Carney vs Trump showdown, to the tune of 1.2GW and $50B in Saskatchewan, in partnership with Bell and others. This is quite the complement for their 300MW+ elsewhere in the country. BuzzHPC is one of the many public companies on the ClusterMAX list, via parent company HIVE Digital Technologies Ltd. Unfortunately, all the GPUs that we have tested from Buzz were in clusters of questionable quality, relying on questionable software vendor partner choices. We have not had the opportunity to re-test since ClusterMAX 2.0, but we hope this changes in the near future.

Neysa

Indian neocloud Neysa announced a $1.2B in financing this February, including $600M equity and $600M debt, in a round led by Blackstone that left them at $1.4B post-money. They claim “9,216 air-cooled NVIDIA B300 staged between December 2026 and March 2027,” with an equal number of AMD MI350X in the pipeline as well. Impressive growth, and clearly the leader in India in this regard.

This round, our testing unfortunately revealed a cluster that was riddled with errors. Many software packages that came pre-installed during cluster provisioning were months or even years out of date, with several serious CVEs applicable. CUDA Toolkit 12.0, for example, was released in December 2022, making it older than Neysa itself. Somehow this was installed on our cluster at /usr/bin/nvcc.

We enjoyed working with the Neysa team and encourage them to continue to improve their security posture and shore up their fundamentals as their capacity spikes and the bring on large customers from abroad. While current Neysa customers are mostly stuck on the Hopper generation, with a few B300s available, they are no GB’s (and, as a result, no direct liquid cooling) that have made it to production on the Indian subcontinent.

Vast.ai

A reseller/marketplace that keeps coming up in customer conversations for random GPUs on-demand, representing one of the better partnerships teams in the industry, finding unused capacity all over the world. Unfortunately, the UI is just terrible, so hard to use.

Hyperbolic

We haven’t had the chance to re-test Hyperbolic since the last round, but they got a basic security compliance attestation done finally! Hyperbolic moves onto the list.

STN

STN has been very confusing. One of the earliest clouds to deploy B300s, and one of the only providers with an option for “private cloud”, where customers take care of procuring the cluster themselves, and keep the chips on their balance sheet, but STN takes care of cluster operations. We hear from multiple customers how they enjoy this arrangement and prefer it over price gouging neoclouds.

We’ve always enjoyed working with the STN team, but unfortunately each of the last three times we’ve engaged with the team, we have run into some performance, reliability, or configuration issue, asked for a fix, been promised a fix, not gotten the fix, and then time has run out.

We regrettably keep STN in this tier until we can see a complete end to end testing experience as the customer would experience it.

Not Recommended - Underperforming

Based on hands-on testing, these providers can quickly rise to Bronze by fixing one or more critical issues, such as offering only older GPUs, missing basic security attestation (SOC 2, ISO 27001), misconfiguring key server features (leaving PCIe ACS enabled, or failing to enable GPUDirect RDMA), or charging for GPU hours during cluster creation or hardware downtime.

SharonAI

Sharon AI is an Australian neocloud that claims 132MW in its pipeline, with 116MW already contracted out, against a current footprint that we estimate around 4 MW. They do not have SOC 2 or ISO 27001. We were able to test 2 bare metal H200 nodes to verify basic functionality. We look forward to testing Sharon AI down the line as their offerings mature.

IREN

IREN continues to come up in our conversations with neocloud customers, and usually for the wrong reasons. They’ve got rock bottom prices on HGX B200 and B300 GPUs in Prince George and Mackenzie in Northern BC, Canada, and we’ve heard the stories. Multi-day power outages, network upgrades, storage failures, air quality controls leading to link flaps, XID’s everywhere. The #1 worst site in the industry according to users. And we are some of their users, as we’ve gotten to test IREN GPUs from multiple providers that resell their capacity. A lot of our users that aren’t having an good time that we are hearing from aren’t renting from IREN resellers but IREN themselves

With that said, the new builds in Childress and Sweetwater look much better (read: not N+0 power, cooling, and ISPs!). We’ve covered plenty about these sites in our Datacenter Model, suffice it to say that despite all the technical shortcomings in their cloud services, we expect the $2.1B investment from NVIDIA will help them cut less corners this time around.

We recommend that IREN stop pretending to offer managed clusters and inference endpoints in public marketing and investment materials, and focus on their bare metal offerings, where there is plenty of market demand for their services.

Hydra Host

Hydra Host raised a $100M Series A in June 2026, with NVIDIA among its backers. They are focusing on brokering deals, and our earlier testing is all we have to go on for managed cluster experience. We need to test a real managed cluster before considering an upgrade.

FarmGPU

We couldn’t make much progress in our testing of FarmGPU. The Slurm layer did not advertise GPU resources properly, while Kubernetes did not expose any RDMA devices for scale-out networking.

With that said, we very much appreciate FarmGPU’s open development culture, including their solid Grafana monitoring experience and detailed notes on provisioning. We find the technical team to be trustworthy and solid to work with, even if they are stretched too thin just trying to get a few small clusters deployed correctly.

WhiteFiber

WhiteFiber raised $159.4M gross at its initial IPO closing in August 2025. Our prior testing found good network performance but unusable Slurm/Kubernetes integration and weak job monitoring. More recent customer feedback suggested improvement, though the ephemeral Slurm login filesystem remained a concern. We need to verify that these operational problems are fixed.

PaleBlueDot

PaleBlueDot raised a $150M Series B led by B Capital in January 2026. We previously found its marketplace easy to use for individual VMs, but it lacked a complete managed-cluster experience. Unfortunately, they have shut down their on-demand console in favor of serving some big customers in dedicated datacenters.

Akamai

Not much has changed in the last year at Akamai/Linode, with single-node GPUs being the focus and no managed Slurm or Kubernetes available. We’ll keep watching to see if this sleeping giant wakes up anytime soon.

Hetzner

Hetzner’s low-cost hosting model still provides some cheap PCIe GPUs, specifically the RTX 4000 and 6000 Pro Blackwell Edition. We’re waiting to see if they take the plunge with anything bigger, building on all their datacenter operations experience to really serve the AI market.

Mithril

Mithril, formerly Foundry, announced $80M in funding back in 2024 to “restore the promise of public cloud to AI”. Today, it’s a marketplace wrapping 3 Nebius availability zones with H100 or H200, though only the H200’s have an option with a scale-out network.

There were some announcements about TPU support that we were quite excited about, but that partnership seems to have been scrubbed away.

OVHCloud

OVHcloud has substantial infrastructure globally, operating in tons of datacenters globally, some of which are shared with top neoclouds on this list. But our concerns remain that general purpose IaaS is not the growth engine of the AI age with companies doing crazy things to get access to the latest and greatest. Another sleeping giant.

Massed Compute

Massed Compute last announced capital raise involved up to $300M in equity and revenue-share financing from Digital Alpha in August 2025. They have used this to drive reasonable growth for a bare metal business, but that alone does not cut it on the ClusterMAX rating system, especially since their slop cannon chatbot is still getting indexed by web crawlers.

Not Recommended - Unavailable

Our “Not Recommended - Unavailable” tier is the same as ever, but to clarify, for this section we include four types of companies:

1. Those that we have previously tested who now claim to have zero spare capacity and/or refuse to test with us. This includes Fluidstack, Cirrascale, Lightning AI (recently merged with Voltage Park), Scaleway, CUDO Compute, Denvr Dataworks, and Atlas Cloud.

2. Those that are yet to launch, yet to be tested, and are generally interesting to us, though we believe some unnamed of this bunch are misleading investors (i.e., lying) by claiming in public marketing materials that they do managed clusters when they really just do bare metal. This includes SpaceXAI, Mistral, Poolside Infrastructure Company, Nscale, Highrise, Corvex, Andromeda, Volta, Firebird, Tatra, Sesterce, Yotta, Boostrun, GlobalAI, Argentum and Qumulus.

3. Those who are big and important, but who we cannot test properly due to geographic or regulatory concerns. This includes Alibaba Cloud, MegaSpeed, BytePlus, RunSun, SK Telecom, Naver Cloud and Indosat/Zankore/Lintasarta.

4. Those who are too small to be relevant. These are still covered in our market view, but not part of the rating system.

Fluidstack

Fluidstack has (currently) exited the managed clusters market, as far as we are concerned, because their focus has shifted from managed Nvidia clusters to bare-metal TPU deployments at 100K+ chip scale. We look forward to testing with them again in the future, with whatever chips they may offer at that time.

Cirrascale

Unfortunately, after receiving a legal letter to modify our previous article’s description of our experience working with Cirrascale, we have not been able to establish a productive working relationship.

Lightning (merged with Voltage Park)

Following the merger of Lightning and Voltage Park, the combined company has unfortunately run out of GPUs, and we haven’t been able to test the reconciled products. This is despite the company consistently advertising that they are the 3rd biggest neocloud in the world in terms of GPUs deployed. They are not in the top 10.

Scaleway

Recently, we hear that Scaleway has de-prioritized a number of customer engagements and decided to voluntarily break ties with NVIDIA, in favour of AMD, for seemingly no good reason. We think this is a shame.

CUDO

CUDO did not make a representative managed Slurm or Kubernetes environment available for this cycle as it focuses on other business priorities (namely, managing the build out of a bunch of bare metal datacenters). We look forward to reassessing CUDO when an environment becomes available.

Denvr Dataworks

Despite changes in leadership and random bouts of activity in the community, we have yet to make any progress testing Denvr and have not seen any meaningful growth in the business that would allow them to dig themselves out of the hole their founders put them in.

Atlas Cloud

We haven’t been able to get GPUs from Atlas recently, despite big announcements about bare metal datacenters from their crypto parent company and random appearances on endpoint provider lists with their managed inference offering.

SpaceXAI

We love Colossus and can’t wait to test the SpaceXAI neocloud offering!

Mistral

Mistral just raised a €3B Series D, and has yet to confirm the validity of a hacker from TeamPCP claiming to put its entire codebase up for sale on the dark web. Anyway, we are very excited to test their neocloud offering. The team told us the platform was still coming up and onboarding its first customers in August, so we very much look forward to testing with them in the future. It seems obvious to us that companies with experience building training clusters for their own models (see above) will be successful neoclouds if they choose to pivot.

Poolside Infrastructure Company

Following a $6B licensiquihire by Nvidia that took an incredible group and buried them in the Nemotron bureaucracy, Poolside’s legacy lives on through the Poolside Infrastructure Company, which is developing their Project Horizon campus in West Texas. We have not yet tested anything from them, but their experience and ambition makes them an immediate contender if the skeleton crew decides to go for it on the neocloud business.

Nscale

Nscale announced a $2B Series C in March, announced the acquisition of AnyScale in May, and just filed an S1 to go public on NYSE a few days ago. It is unbelievable to us that they can have such a big business, and continue to claim in public marketing materials that they offer managed clusters and inference endpoints, when this is clearly not true. Obviously, though, they’ll continue to be successful. They’ve got access to lots of land and power, some construction and datacenter operations experience, and people really want that bare metal at a nice price.

Highrise

Highrise, Hut 8’s AI Cloud business, is constantly in the news. We hope we can test some managed clusters from them soon.

Corvex

Corvex continues to run scared from ClusterMAX, though they announced a $33M private placement in August to expand their datacenter capacity. Their choice of merger partner was Movano, maker of the Evie smart ring. Quite a change in product roadmap. Privately, they claim some secure government customers in the USG. We look forward to testing.

Andromeda

Andromeda takes capacity from other providers and puts a managed cluster service on top. We have discussed testing multiple times, and enjoyed working with the technical team, but find the business side to be a little shady. Resellers have a perverse incentive to keep the customer in the dark about who the underlying provider is, out of fear that the customer can just go around them and contract with them directly. Unfortunately, this can lead to bad games of broken telephone, and lots of finger pointing in the worst case. Specifically, we have a problem with non-circumvention clauses and hope that Andromeda can build a durable business model that allows them to be honest about their solid technical team and the value it can provide.

Specifically, we are aware of three different options from Andromeda:

1. Rent compute - typical 1:1 neocloud relationship, where Andromeda aggregates capacity from underlying providers and resells it, but depends on them to deliver support

2. Manage compute - some providers have a growing list of 3 or 4 different providers and want one throat to choke, so Andromeda becomes that throat

3. Bring your own cluster - also called private cloud, this is when customers want to own their own hardware (and control their own supply chain) but still need help managing it all, effectively SRE-for-hire

We are seeing big growth in Option #3 on that list. And we look forward to testing some of Andromeda’s Blackwell capacity soon, wherever it may be.

Volta

Volta launched with $300M in venture funding and a separate $5B financing pool for customers. Its first large announced contract uses Bitdeer’s Norway site. We have not tested the cloud software or customer operations.

Firebird

Firebird is building GPU infrastructure in Armenia, with $60M in construction financing announced by Ameriabank, and expansion plans in Kazakhstan. Recently, they became the subject of a story about Nvidia chip access and the Armenia-Azerbaijan peace process; co-founder Razmig Hovaghimian responded that the project had roots going back years before those diplomatic developments. GPU diplomacy. Pressure’s on to deliver for the people of Armenia! We still need to see the managed cloud in action.

Tatra

Tatra is developing B300 and GB300 capacity in Slovakia, using the country’s nuclear and hydro power base. We have some great feedback about the technical chops of the father/son team—hard working, impressive family. For now, though, we need them to get their first tranche fully deployed so we can put them to the test.

Sesterce

We spun up an on-demand H100 through Sesterce in August. The console showed an out-of-date Ubuntu option, provisioning took its time, and eventually we got a working machine. Then we found Shadeform underneath. A broker on top of a broker. We would quite like to know how many companies need a margin before the customer gets to run a workload! With that said, Sesterce is pursuing some big bare metal Blackwell clusters with big offtakers. We want to test those.

Groq

Groq set out to challenge Nvidia, but after a licensiquihire took Jonathan Ross and most of the engineering team, the remaining company has pivoted… to renting out Nvidia GPUs! Becoming a neocloud is the move if you want another $350M to live out your dreams (as announced in August). We look forward to testing more than just endpoints one day. Jensen gets the chip team and another customer. Not bad.

Yotta

The four-node Yotta H100 Kubernetes cluster we tested had many problems. Within a week, there was genuine XID 94 on one node and a bad rail on another. Yotta’s bring-up process appears brittle, as our small cluster had important discrepancies from node to node. It also had many out-of-date packages. One of our nodes failed to come back online after we rebooted it. Yotta’s health checks did not intervene when we manually reset the secondary PCIe bus to trigger an XID 79, leaving us with one GPU unreachable until manual intervention, although the Kubernetes layer did nothing to distinguish the shorthanded node.

Yotta must successfully deploy Blackwell GPUs and improve its reliability to keep pace with the industry.

Darya

Darya is bringing GPU cloud infrastructure to Tajikistan, right on the border of Afghanistan, which certainly expands the map of places we need to test. Its H200 cluster launched in June 2025, and it subsequently signed an agreement with Yotta to develop a hydropower-fed AI datacenter in Darvoz. Interesting stuff, but after our experience navigating Yotta’s portals, we are particularly curious which parts of the operating model make the trip to Tajikistan. We have not yet had access to Darya’s cluster, though we would love to visit the site.

Boostrun

Boostrun has some solid bare metal, but bought its way past some of the Kubernetes engineering: vCluster says the company launched managed Kubernetes in under 45 days with no new platform engineering hires. They went with isolated tenant control planes, dedicated private nodes, and Netris for network provisioning. Sensible. We would rather see a small provider use working software than spend a year rediscovering Kubernetes. In customer discussions, we have still considered adding another operator for a fully managed training experience. We have tested a Boostrun cluster indirectly, through another provider reselling their capacity, but await the opportunity to engage with their support teams directly. We want to see how much of the day two operations Boostrun handles for themselves in a real engagement before moving them onto the list.

Global AI

Global AI takes the sovereign-cloud pitch quite literally: dedicated, single-tenant, air-gapped clusters sold by the data hall (or so they say). Customers decide when Global AI can access their systems, a solid offering for bare metal. However, when advertising an option for managed clusters, we need to see more. Just to assess bare metal we want to see things like how provisioning and security patches get handled, how comprehensive the monitoring stack is, how reliability and health checks work, and generally how competent the onsite team is at closing tickets quickly and correctly. We have not yet gotten to see this level of detail.

Argentum

Argentum’s website says “200,000+ GPUs available now.” Had you heard of Argentum before you got to this section of the article?

QumulusAI

QumulusAI found a different source of GPU debt: a $500M non-recourse facility through USD.AI, using GPU Warehouse Receipt Tokens as collateral for stablecoin borrowing, with financing for up to 70% of approved deployments. GPU Warehouse Receipt Tokens is the actual name. The announcement describes a facility, so we are not counting $500M of installed hardware. In our customer discussions, Qumulus has come up as a capacity supplier that may need another operator on top for managed training. We still need to inspect its own software and support, but initial customer feedback has been surprisingly solid.

Alibaba Cloud

Alibaba has considerably more software to show us than the usual neocloud startup. One example is its ACK Slurm operator: a SlurmCopilot component coordinates resource allocations between Slurm and Kubernetes so idle resources do not remain stranded in one scheduler. This is the kind of orchestration plumbing we like to poke at, and obviously this is a company that is not ready for a poking. We have an account, have been whitelisted for a few model endpoints, and have engaged directly with the sales team, but have yet to get access to any modern GPUs or this ACK software for testing.

Megaspeed

Plenty of hardware to investigate in their Malaysian and Indonesian sites, but no login yet.

BytePlus

BytePlus’s GPU service advertises switch-affinity placement on a multi-rail network, direct RDMA access to its vePFS parallel filesystem, and integration with our favourite model ever: Seedance 2.5. We have not yet completed an evaluation of their clusters, but there is clearly stuff to watch out for. Tik, tok.

Humain

Humain has given itself plenty to do. Alongside the Saudi GPU buildout, Tareq Amin announced a roughly 6 GW datacenter ambition, as well as Humain One, a real-time voice interface for any computer. We would be happy to start with a login and a functioning cluster, but we have yet to get a response from the team, leaving us wondering what secrets they’re hiding next to their Groq systems.

SK Telecom

SK Telecom runs over 1,000 B200s for Korea’s sovereign foundation-model initiative. As we covered in a recent article, the government also rented ~3,000 H100 equivalents from SK Telecom and Naver combined for the competition’s first round. SK Telecom has announced a much larger 2 GW NVIDIA DSX AI factory, expected to deploy Vera Rubin systems with SK Hynix HBM4, within SK Group’s planned 5 GW first-phase buildout. We have tested SK Telecom GPUs through Vessl, and they were solid, but nothing with SK directly. We need to see more from them on the international market to consider them a serious player, but the technical foundation is clearly there.

Naver

Naver, much like SK Telecom, is highly credible. It already operates hyperscaler-class datacenters, and its July plan with NVIDIA and Brookfield calls for expanding the GAK Sejong AI factory from an initial 55 MW buildout to 200 MW by 2028. We are interested in how much of that operational experience reaches external customers, vs the internal research teams at Naver. We have yet to test that experience.

Indosat (Zankore)

Zankore has announced the first phase of roughly 200 MW of GB300 NVL72 capacity is to be delivered in the first half of 2027. Even if they are a little behind schedule on deploying Blackwell, they are still well on their way to a long-term 1 GW ambition, primarily to serve offtakers (rumoured to be) from mainland China. And they’ve landed a $3.1B loan to go do it!

Where ClusterMAX Goes From Here

Read more

The best place to begin is the service as a whole. Inference runs inside a cluster coordinated by an orchestration layer such as NVIDIA Dynamo, Mooncake, or a custom scheduler. These work closely with inference servers like vLLM or SQlang, and those also have their own orchestration features. This article will not go into the details of how you work with the orchestration software or which you should choose. The article aims to provide an overview of the process and reasons for various features you may use.

A user (or their agent) starts a conversation with a request (query), and the conversation may continue after answers with more requests. The request- answer is a “turn”. In modern AI systems the user is often running a in a client application (whether GUI or command-line oriented) and some of the answers coming back from the AI are intercepted by that application as instructions for it to run, things like editing your code or searching your company guidelines on an HR question.

The AI system can also do some of those things itself from its data center, for example it may search the web. The results from these intercept tool actions are also returned to the AI as requests, creating more turns. The server keeps a context (often referred to as KV cache or just cache) which is the distillation of the session, allowing each new request, from user or from tool, to be properly interpreted for overall forward progress. There can be thousands of turns per hour when the user launches an agent on a long-running task. The conversation can also pause and resume hours or even days later.

When a request arrives at the inference servers the orchestration puts it into a queue, while will feed into an input-fill worker, along with any context from system prompts and prior turns in the conversation. The query is then converted into new context appended to the conversation. That new context state is then moved to a decode worker. Decode repeatedly reads the accumulated state, generates answer tokens, and those answer tokens are also appended to the conversation state. The process may loop through tools, agents, users, and further input-fill work. A model worker running on GPUs is only one part of this token factory; storage, networking, and orchestration connect its stages.

Inside the workers, four operating regimes are worth distinguishing from the beginning:

1. Prefill, where the initial block of new tokens is processed together.

2. Midfill, where a continuation request is appended to an existing cached context.

3. Decode attention, where the context is used to generate the basics of newly generated tokens.

4. Decode experts, where each basic new token is refined by a selection of experts a large total set of possible experts.

These regimes place different demands on compute, memory, and networking. Prefill and midfill are closely related, with Prefill being a Midfill with zero prior context. However, Prefill is a common special case, corresponding to one-shot queries like classic “”chatbot” use, and can be optimized a bit differently than Midfill. Prefill reaches high arithmetic intensity (ratio of computation to data movement) and does not need to wait for context to be located and read. Midfill begins from an existing KV cache for prior state and appends a new request input sequence. This generally has a more moderate arithmetic intensity than prefill because there are fewer new tokens and more existing data to move per token.

Decode attention and decode expert work are generally low arithmetic intensity since there are few new tokens compared to the prior context or expert weights, which at decode state are data that needs to be moved. Meanwhile, across all of them, the tensors for attention and for experts are the same regardless of token value, so there is benefit to sharing a single reading of a tensor with as many tokens as can be queued to use it, even tokens coming from unrelated requests by other users which simply happen to be running at the same time and which are networked closely enough to share. Treating all four mode as the same workload gives away much of the structural advantages that MoE models offer in terms of different intensity and different sharing patterns.

In this document we will treat the 4 stages separately, and sometimes assume they are disaggregated. There are tradeoffs between aggregated (where the models stay on one server, which reconfigures as the request progresses between the stages) and disaggregated where the orchestrator may find a free slot on another machine already configured appropriately. Generally we will talk about the stages as disaggregated, with aggregation being a special case of always making the current node available when a previous stage finishes. There will be a section on the trade-offs at the end, after the work to be done has been more completely introduced.

A transformer model is a deep stack of repeating layer groups. A group may contain one kind of layer, or it may contain a full-attention layer together with several linear, local, selective, or otherwise optimized-attention layers (this is a hybrid group of layers). Within each layer, attention, shared transformations, routing, selected experts, and recombination occur in sequence. Those steps are known in advance and repeat continuously, allowing the same hardware resources to serve different parts of the flow at different moments.

The machines can also be described equally simple recurring units. A node is a group of accelerators, typically a tray in a rack, each accelerator pairs compute with local fast memory. A node is coordinated by a CPU and connected to other nodes through NICs (network interface controller). Nodes stack into racks. A rack may act as one scale-up domain when very large tensors are , several smaller scale-up islands, or a set of pipeline stages. Datacenter networking then joins bounded workers to storage and orchestration rather than participating in every inner tensor operation.

This essay follows that progression. It begins with the global service and the operating regimes, then opens the model into wide layer-group slices and maps them onto accelerator nodes and racks. From there it develops pipeline, tensor, and expert parallelism; the limit imposed by KV-cache memory; the different value of batching in prefill, midfill, and decode; and the role of scheduling in a many-rack token factory.

1. Orchestration delivers requests via queues into pools of workers

An inference service is a cluster where prior requests and immutable state are stored when idle and then queued for delivery to specialized workers when a new request pulls in context or a prior request reawakens for continuation. The incoming queue is not yet a batch. Each request carries prompt tokens, references to reusable context, service objectives, and often an existing conversation or agent state. An orchestrator may place it into a prefill, midfill, or decode batch, and the request can join or leave that batch independently as it completes, making that capacity available for new work. Continuous batching works best when the scheduler knows which workers have slots that match the request’s context length and service objective.

Prefill workers create context from a substantial block of new tokens. Their output is new KV state for every model layer, stored independently of the worker that created it. The KV state moves along model layers, the output of layer K from one turn becomes input to layer K in the next turn. In pipelined servers this creates a natural “torrenting” of output and input from separate NICs.

Midfill workers extend previously processed context. The cached prefix may contain system and user prompts, memories, earlier conversation turns, agent steps, or retrieved documents. Large roots may be already cached, while the incremental input can be much smaller than a comparable uncached request: a half-million context may receive only a few hundred or a few thousand new tokens.

Decode workers consume the processed, accumulated context and generate one or a few tokens (multi-token prediction is becoming quite common) per pass. Every generated token adds a small amount of new state. The same context can later return to a midfill worker when a user, tool, or agent contributes another block of input. Agentic work using tools creates a repeated sequence of context extension and generation, not merely one prefill followed by one decode.

A worker may occupy a node, a tray, or a rack. The service becomes large by operating many bounded workers and moving requests and state among them. Depending on precision and working-state requirements, even multi-trillion-parameter models can fit within the aggregate memory of a modern rack-scale system. The datacenter network remains essential because the factory must connect many such workers to shared storage and continuously match worker configurations to demand. KV transfers may be gigabytes in size, but they can be striped or torrented across multiple links and destinations; their movement time can remain well below the lifetime of a long decode job.

This separation is useful for both performance and operations. Prefill and midfill are predictable once the cached-prefix size, new-token count, and worker configuration are known. Decode completion time is stochastic because arrival of the final output token cannot be predicted in advance. Shared memory and storage between the pools let each stage run at its own cadence. Later sections of this article will discuss queues, ready buffers, and worker reconfiguration. For now, the important point is that the service is a loop of stages joined by movable state.

AgentX examples

This is viewed with the SemiAnalysis Conversation Explorer. It shows that context growth during conversations can be quite rapid but also goes through large fluctuations due to compaction and other model behavior which may not be explained in public discussions. Orchestration of the inference context is a competitive advantage for AI companies.

We can take that same dataset and pull out the cache, new input, and result length values seen at every turn. This visual shows how the Token Factory needs to serve many different workloads. In principle there could be requests similar to each dot in those traces (and more dots from other workload datasets) all running at the same time, assigned to some worker in the AI cluster.

This article aims to survey some of the major functions in inference that make it possible,

2. KV state moves as immutable blobs

Reusable context at data-center scale becomes a shared object-store of immutable blobs rather than a file attached to one machine. A system prompt, project context, previous user turns, tool output, and generated tokens can be in separate blobs. A new operation reads the blobs it needs and appends new ones; existing blobs normally remain unchanged. This forward-only structure follows the causal transformer itself. Changes can be handled by backtracking and creating a new branch rather than rewriting the common path.

The durable source of these blobs is a fast, parallel, scale-out memory and storage pool. It needs enough aggregate bandwidth and network reach that prefill, midfill, and decode workers can be selected for suitability and availability rather than because one machine owns the only copy of the context. Newly idle blobs can first move into shared network-attached DRAM, including node-CPU memory and dedicated memory appliances. As that tier fills, a classifier can discard blobs that are not likely for reuse, or promote longer-lived state to SSD. The underlying text and references are often orders of magnitude smaller than the expanded KV representation, so it is possible rebuild a KV blob from that smaller text if the triage decided to reuse space and discarded the expanded embedded state. AI calculations can vary subtly, so while the rebuilt state can be expected to be a valid context, services may take different approaches to how casually they discard and rebuild, and how much they try to keep important immutable state.

For a deeper dive on how the blobs are managed, Unified Radix Cache: One Tree for Hybrid Model Prefix Caching - LMSYS Org is recommended reading. It is not the last word, but it is clearly written and will point you to other sources if you want to look for more.

There are also frequent compactions of state with long-running agents that build up to the maximum context, so in general after a compaction there will be many prior blobs which have been abandoned in favor of a new context. The space from the pre-compaction contexts is probably recycled for new work.

You can see compactions occur in the right half of these conversations as contexts get close to the practical context limit for each model (250k tokens for Opus 4.8, 1MT for Fable). There are also “stalactites” in both lines which seem transient and reflect some proprietary behavior in Anthropic models.

Accelerator HBM is the “hot” working tier. It is too expensive and supply-constrained to be a good default for passive context storage. The active prefix and suffix should enter HBM shortly before use and leave promptly after the worker has finished with them. CPU DRAM is a useful staging and assembly tier, especially for outgoing state: a completed worker can move newly generated blobs into CPU memory while the storage system chooses placement and redundancy, after which the NIC sends them into the shared pool.

For incoming data, RDMA-capable systems may allow the network to place data directly into accelerator memory, avoiding a full copy through CPU DRAM. CPU memory still remains useful for metadata, coordination, partial assembly, fallback paths, replicas, and outgoing staging. When a worker uses multiple GPUs the incoming context can be striped directly to their destination memories in parallel.

Rack-level hot-blob caches can also be useful, whether implemented as memory/storage appliances or making use CPU memory which is assigned to the distributed storage pool. Very common objects such as system prompts are natural candidates. These caches should remain shared scale-out resources rather than private state that binds a request to one decode machine. HBM is several times more expensive and more supply-constrained than DDR. HBM’s best use is for data ina an active batch currently earning revenue.

Data input is generally larger than output. Midfill commonly reads a large prefix and appends a meaningful but smaller suffix. Decode reads the accumulated context and appends only the state for one or a few generated tokens. The reads are repeated, the writes usually occur just once. The system design should show these differences directly with the width of each data path.

3. Prefill, midfill, and decode create distinct operating regimes

The global service contains several computational regimes. Their differences are more useful than the generic labels compute-bound and memory-bound, because each has a different opportunity for reuse and a different natural mapping onto hardware.

3.1 Prefill

Classic prefill begins with little or no reusable KV state and processes a substantial block of new tokens. Those tokens share weight loads, use matrix operations efficiently, and create enough routed activations to form useful shared-expert batches. Attention and dense transforms can reach high arithmetic intensity, so prefill is often constrained primarily by compute. Long initial input contexts can raise memory traffic but these rarely get to 100,000 tokens in length, while midfill is commonly beyond that.

3.2 Midfill

Midfill appends new tokens to a much longer cached prefix. The new block can still provide hundreds or thousands of activations for shared weight and expert reuse, raising arithmetic intensity to thousands of ops per byte of data. Attention must also read a large existing KV state, which can be tens of GB. It therefore sits in a mixed regime: more arithmetic reuse than decode, but much more cached-state traffic than an initial prefill of the same new-token count.

3.3 Decode attention

Decode advances on input of one or a few tokens per query. Each query brings its own KV state which is similar in length to what midfill processes. While prefill differs from midfill on length of cache, decode differs from midfill in input sequence length. These one to few input tokens keep arithmetic intensity low, this work is clearly dominated by the memory movement around the KV cache.

For long contexts, the attention read can exceed the size of tensor operations associated with model weights. Attention is therefore one of the largest operations in modern decode even though most model parameters reside in experts. Delta, top-k, or linear-attention algorithm layers can reduce the burden, but periodic use of a layer with full attention remains a major memory load. Each query brings its own context, so batching queries does not change the arithmetic intensity of attention.

3.4 Decode experts

Experts are not specific to the decode phase. Everywhere the model runs the experts run, and they require no context other than one token’s embedding. The history of the query has already been compressed into the activation presented to the expert. This allows various designs that share the experts across all available GPUs, reducing memory size per GPU and increasing the memory bandwidth / memory size, or memory intensity ratio. You can look at the interval needed for a GPU to read all its experts as the best case interactivity (token rate seen by each user) and so if the memory bandwidth remains constant but the GPU is responsible for fewer experts, interactivity can be higher. The gotcha is that networking needs also rise, and the all-to-all pattern needed for routing to experts is a difficult one.

At the end of attention (in prefill, midfill, or decode) a router calculation (a small tensor operating on the unrefined token) selects a few experts for each new token. Tokens that select the same expert can share a tensor-weight load even when they come from different requests, so long as they can be gathered together in time to take advantage of the same weight loading. In practice their may be 15,000 experts across all the layers, so taking advantage of such coincidences requires deliberate synchronization like awaiting until all queries in a batch have completed attention until switching over to sending them for expert work, and that in turn may be improved by having multiple instances in different nearby machines also synchronized to that schedule. If you have ever wondered why Nvidia makes such an effort to connect 72 GPUs so closely, this is a big part of why. In a large token factory it is feasible to have those 72 machines all running in sync, allowing the experts in one layer to be divided up to 72 ways. If a layer of the model has 256 experts then each GPU is handling only 3 or 4 in each layer, permitting very fast complete cycling through all the experts. And a lot of all-to-all traffic.

Something interesting happens if your experts can all be kept in SRAM. You need a huge number of accelerators and a monster network to collect them all to all back to the GPUs, but now there is little reason to wait for a batch to form. Each expert can run whenever it has input. The energy per byte loaded is as much as 100x better than loading from HBM, so a single token running can be as efficient as perhaps a queue of 100 was before. In return for this DAF (disaggregated attention-FFN, where the experts are the FFN) you can free up the instances from needing to be synchronized. This advantage does not help as much with prefill or midfill since they trivially have a batch of tokens they can sort into shared queues on one layer, but it still may offload all the expert weights from those GPUs allowing them to optimize their local memory for other uses like longer contexts. The networking is still a gotcha, where an SRAM based accelerator could turn around an expert calculation in less than a microsecond but now you may be flooding a switch connected to hundreds of nodes with the all to all traffic for millions of tokens. The network may absorb more money and power than the experts they connect.

Inference workers can improve sharing by coordinating many attention instances on the same model layer. Their routed activations then draw from one layer’s expert bank, reducing the number of different experts each destination must load. In a pipelined server this also means the networking connections used for expert parallel can be local to that stage of the machine, there will never be a routing that needs experts from another layer. Even so, each decode request contributes only one new token with typically 8 to 16 routed experts per pass, and a layer may contain hundreds of experts. At small batches, sharing remains limited and memory bandwidth continues to dominate both the KV scan and the active expert loads, with very little computation per byte of weight loaded. Expert memory needs to be very low energy per bit, no matter what kind of memory is in use.

These regimes explain why prefill, midfill, and decode may prefer different worker configurations; why decode attention and decode experts can prefer different placements within a node; and why batching helps each stage by a different amount. As prefill in modern agent work rapidly expands to the 500k token range there is a lot of memory movement, especially in token decode - the stage where the model is generating useful intermediate and final results. Fast and large 3D RAM will be decisive in efficiency. Fast memory is also useful for midfill. However, capacity needs to reach a minimum level for the large contexts we now see as normal. If memory capacity is too small then the losses in networking and excessive swarms of chips can waste power compared to future memory types which will use 3D capacity expansion without giving up on throughput.

While Vera Rubin 72 can crush at the mid-fill layer, which is arithmetic intense, it is not so effective at the switch to memory intensive decode where the Rubin cannot fully exercise its computational prowess. With the large context there is a premium on keeping the decode work in the same node and the midfill, avoiding sending the context across the network to a different node. This could favor an all-around chip which both has super memory thruput and really efficient computation.

4. Flow through an MoE prefill layer

Classic prefill begins with attention. A block of new query tokens attends to reusable prefixes like the system prompt and to earlier positions in the new block. The attention operation produces one activation for every new token, and shared transformations prepare those activations for the routed-expert stage.

The router assigns several experts to every token. If the model selects eight routed experts, each incoming token produces roughly eight routes across a bank that may contain 256 experts or more. In token order, those records are sparse and interleaved: neighboring tokens can have completely different destinations. Across the full input block, however, every expert can accumulate a short list of assigned tokens.

The worker sorts or queues the route records by expert identifier. All activations assigned to expert 037 become one message stream; those assigned to 142 become another. A single expert-weight load can then serve all activations in that bucket. Several requests can contribute to the same buckets, and separate attention instances running the same layer can merge their routed work at the expert destination. Converting token order into expert order is one of the central efficiencies of MoE prefill. Scale-up networks like NVLink can be organized as memory mapped connections so sending requests to specific experts could be mapped to pushing the message into a specific memory mapped location which is configured as a hardware queue. There is very little start and stop time. The experts can listen with the equivalent of an RDMA CIQ that again leverages hardware acceleration to deliver the message stream without need for complex protocol start and stop. Simple headers inline on the message identify the sender and receiver. A similar connection can be set up for the return flow from expert back to the decode worker.

The expert outputs are returned to token order, weighted, combined, and passed to the next layer. At the same time, attention creates new K and V state for the appended tokens. Across the model, those values form an immutable KV suffix which are used to decode the next token, and which are eventually stored as the next blob in the context for use by the next turn in the conversation.

Prefill therefore combines two forms of weight reuse. Attention and dense transforms reuse weights across many tokens, while routed experts reuse each selected expert across the tokens collected in its bucket. The gains eventually flatten as growth in activation traffic, KV traffic, and output handling take more time compared to the tensor calculations. Batching can delay time to first token, but a single request containing several thousand new tokens already provides substantial arithmetic intensity; prefill does not always need a large number of requests in each batch to run efficiently. Modern single-socket GPUs may be able to do the prefill even on a large model for a small batch in around a second.

5. Flow through an MoE decode layer

Decode begins with a much smaller amount of new work per forward pass. The current token, or a handful of speculated tokens in MTP (multi-token prediction) is the query. Attention reads the prior KV state together with the new token’s KV entry. The new entry appends to the result and is appended to the KV state for the next pass.

For agentic contexts, the prior KV state is often a heavier read burden than the attention and expert weights used by that one token. The new query and its local projection weights are comparatively compact. Attention may be partitioned across KV heads, context ranges, memory channels, or accelerator units, after which partial results are reduced to one token activation. That activation is small enough to move cheaply even though it will trigger much larger local tensor operations in the next phase.

The activation passes through shared transformations and routing. The router selects a few experts from the available bank, and only those expert tensors become active for the current token. Their outputs are weighted, combined, projected, and passed onward.

At each layer, attention creates and retains that layer’s K/V entry for the new token. The activation then advances to attention in the next layer, where the same sequence repeats. By the time the final layer produces the next token, every layer has appended its small K/V update. Those updates can be assembled into a new immutable KV suffix while the token is streamed to the client or agent.

The flow therefore contains both a broad operation and a sparse operation. Attention reads widely across private query state. Experts activate a small subset of total model capacity. The hardware mapping should support both without forcing the entire layer to use one parallelism strategy.

6. Midfill is a distinct operating regime

Midfill combines a long, cached prefix with a block of new tokens to prepare another turn in a conversation or agentic work. Traces show the cached input to be many times larger than the incremental query, although the incremental query typically varies from 50 to 5,000 new input tokens. The presence of cached system prompts, conversation roots, memories, prior agent steps, and retrieved documents can quickly raise conversational and agentic contexts to a million limit. AgentX datasets show frontier experts are repeatedly compressing contexts to stay under a million-token limit.

This makes midfill neither a small prefill nor a large decode. Like decode, it must read a substantial private KV cache. Like prefill, it processes enough new tokens to reuse weights and to sort routed activations into useful expert batches. Its arithmetic intensity can be hundreds of times greater than one-token decode, yet its cached-state traffic can be tens of times larger than that of a similarly sized initial prefill.

Modern accelerators can generally exploit the arithmetic reuse available in a block of hundreds of tokens, so the difficult balance is often between the large KV read and the burst of routed-expert traffic. That burst does not match the smooth rhythm of a decode pipeline: a midfill can occupy a stage much longer than the decode batches in neighboring stages, leaving those stages underused.

The ideal parallelism can also differ. The worked example later in this essay finds midfill configurations that use different pipeline, tensor, and expert-parallel settings from decode. Reconfiguring a GPU in the middle of a live decode pipeline can be expensive or impractical. A service may therefore benefit from a dedicated pool of midfill-optimized nodes. The hardware can be identical to the decode pool while the loaded model stages, parallelism, and batch policies differ.

7. The model is a tall stack of repeating layer groups

A model’s parameter count quantifies the total learned state, but serving depends on how that state is organized and used. The model is a stack of layers, and increasingly it is useful to group those layers into repeating architectural units.

In some MoE models, a layer group is the same as one layer. In others, a group contains one full-attention layer together with several linear, delta, top-k, or otherwise optimized-attention layers. Typically 1 to 3 of the initial layers may be dense or otherwise designed to get a clean start. The rest of the model will normally repeat one recurring group pattern.

A wide, shallow crêpe is a useful representation of a layer group. The broad surface provides room for attention, shared transforms, routing, expert capacity, and residual flow. Its shallow thickness marks the group as one slice in a much taller repeated model. It also echoes the physical form of an accelerator: a very thin active layer spread across a broad package. The crêpe is a useful intuition for mapping a wide algorithmic flow onto a wide hardware surface.

The repeated structure simplifies function planning. A mapping created for one group can be reused for the next. Corresponding memory regions can hold the next group’s weights. The same compute and communication plan can execute repeatedly. Uniformity matters not only for runtime efficiency but also for compiler, kernel, and operations tooling.

During decode, one layer is active at a time for a given token. Within that layer, several operations occur sequentially. Neighboring layers may be prefetched, and different pipeline stages may process different queries simultaneously, but the logical path of one token still climbs through the stack in order.

8. Parallelism should partition the narrow dataflows of the model

Parallelism is not one decision. Pipeline, tensor, and expert parallelism divide different dimensions of the model and should be judged by different communication patterns.

Pipeline parallelism follows the layer stack

Layers immediately suggest pipelining. The activation passed between layers is compact—often one embedded token, a small multi-token-prediction block, or brief summaries of prior hybrid group results —while the weights and KV state used inside a layer are much larger. A pipeline stage can therefore own one or more complete layer groups and pass a relatively small activation to the next stage without overpowered networking.

Pipelining divides model weights cleanly and keeps hot within-layer communication local. Its main drawback is multi-batch occupancy: a fully active pipeline has a different query or batch in every stage. Each stage therefore holds its share of the KV state for all live streams, and the memory benefit eventually stops improving as contexts grow.

Tensor parallelism belongs where one operation is intrinsically broad

Attention may need to divide weights and KV state across several accelerators. KV heads, context ranges, or another linear dimension can be partitioned so that each accelerator processes its share before a compact reduction of the token result. Tensor parallelism can also serve unusually large dense transforms.

TP should not spread automatically into every tensor of the layer. A collective that is sensible for a broad attention operation may cost more than the arithmetic of a small expert tensor.

Expert parallelism follows independent expert tensors

Most MoE parameter memory may reside in experts. Experts retain no query history of their own; context arrives in the activation. The expert bank can therefore be distributed widely across a peer set and shared by several worker instances.

Wide expert parallelism is useful because the experts are independent and numerous. The router sends compact activations to selected destinations, and those destinations return transformed activations. Expert-parallel width can remain large even if pipeline stages become smaller, allowing switch radix and local memory placement to be chosen independently.

The general rule is straightforward: use pipeline parallelism for the vertical layer sequence, tensor parallelism for large indivisible operations, and expert parallelism for the context-free routed-expert bank. A stage fits comfortably inside a scale-up domain when its available fast memory safely exceeds its local weights, live batch state, and working buffers:

M_{free,node}>\frac{W}{P}+M_{batch}+M_{working}

Modern GPU nodes often exceed that threshold by multiples. When they do, the demanding bandwidth, power, and latency requirements can remain inside one node while thinner pipeline activations and KV movement use scale-out links. The rest of the fast memory is probably not earning any revenue. Data sitting idle in memory is an expense, data moving to be processed is revenue.

9. A logical accelerator node

A logical node is easier to understand as a peer set arranged in a ring than as a literal circuit-board layout. The ring contains perhaps sixteen accelerators. Each accelerator pairs a compute region with local fast memory. The units connect through a strong scale-up fabric. A CPU coordinates workload and memory management, while a NIC connects the node to rack and datacenter resources.

The ring is just a visualization which shows the accelerators as peers, with equal standing. I am not implying the node-internal network should be a ring. It could be hub and spoke, rail, all-to-all, torus, hypercube .. whatever floats your boat, so long as it can handle the highest dataflows the rack contains with low latency and low energy per bit. I’ll just draw it as a ring for simplicity.

This representation avoids committing too early to package placement, board routing, or switch implementation. The actual machine may use a central switch, several switches, direct links, or a hierarchical fabric. The logical requirement is a bounded peer set with predictable fast local communication.

The accelerator design may vary. One version may use SRAM-heavy processing-in-memory units. Another may use hybrid-bonded DRAM, or IGZO memory cells placed on top during BEOL processing. Another may use a hybrid-bonded high-bandwidth true 3D memory. Memory capacity and bandwidth can change while this logical model remains useful.

The node is the natural place to keep work that repeats at high frequency. Attention partitions exchange compact partial results. Routed expert activations travel to local destinations. Expert outputs return for recombination. Shared state and scheduling metadata remain near the CPU. The NIC carries pipeline activations, KV blobs, and work assignments beyond the node.

A node may be smaller than a rack like we see in an SGX node with 8 GPUs, or a scale-up system may span the whole rack like we see with NVL72. The logical symbol remains valid in either case. If scale-up already covers the rack, the separate rack-backbone level disappears and the next boundary is directly to scale-out networking.

10. Nodes stack to make racks

The ring becomes more expressive when tilted as broad, shallow hardware crêpe. Several units can be stacked vertically in a rack, like model layer groups stacked in model depth.

This repeated form makes the model-to-hardware relationship easier to see. A layer-group crêpe can be placed onto one hardware crêpe. Several layer groups can occupy one node if memory permits. One layer group can span several tightly connected nodes if attention or expert capacity requires it. The mapping can expand or contract without changing the basic visual vocabulary.

A rack may be organized in several ways:

• one rack-wide scale-up domain;

• several node-scale islands connected by a rack backbone;

• pipeline stages distributed among those islands;

• wide expert placement within each stage;

• or a mixture of stage-local scale-up and shared scale-out links.

The rack backbone is therefore an optional middle level. Some systems extend the scaleup fabric across the rack needing no separate node connector. Others use strong local node fabrics and a distinct top-of-rack or shared-rail network. Above the rack, scale-out networking connects storage, prefill pools, decode pools, and other racks.

The repeated rack units also help separate two kinds of scale. The model can fit within one bounded rack worker, while the factory scales by adding many workers. The first problem is function placement. The second is worker orchestration.

11. One layer group reuses the node over time

A layer group does not require every operation in the algorithm at peak intensity simultaneously. Its operations form a repeating sequence that reuses the same hardware for different work over the decode cycle.

Attention may spread across much of the ring, using many memory channels and several compute units. The query and output transformations use model weights in more compact operations. Shared feed-forward work may use a dense region. Selected experts then activate local memory-and-compute regions around the ring. Recombination returns a compact result.

The same physical accelerators can participate differently at each step. Execution units that calculate attention may later execute one or more experts. Memory channels used to stream KV state can later feed expert weights. Buffers and local links are reused as the token advances.

This time-sharing is central to an efficient design. Static diagrams can make the machine appear underused because not every block is active at once. In reality the hardware is serving a sequence of different operations, and the goal is to keep the sequence moving with little setup or synchronization overhead.

The broad layer-group crêpe and the accelerator peer set are complementary. The crêpe shows the algorithmic flow; the ring shows the reusable physical resources beneath it. Mapping is the act of aligning each stage of the flow with a suitable subset of the ring.

12. Put throughput boundaries where the flow is smallest

Once the model and machine are drawn as connected flows, the placement rule becomes clear: frequent, latency-sensitive communication should remain inside the strongest local fabric, while weaker links should carry compact or amortized transfers.

Attention partitions may exchange partial results every layer. Expert routing sends activations to selected experts and returns their outputs. This traffic is not always the largest by byte count, but it is frequent, bursty, and sensitive to synchronization. It belongs inside the chosen scale-up domain whenever practical.

The activation passed between layers or layer groups is much smaller. A pipeline boundary can therefore cross a weaker link without moving the layer’s full internal working set. KV blobs are larger, but they move at worker boundaries and are amortized across many generated tokens. Their path can use striped or torrented scale-out networking and shared storage rather than consuming the innermost tensor fabric.

The router-to-expert boundary is typically a medium-throughput all-to-all carrying compact activations. Its difficulty often comes from endpoint count, synchronization, and routing efficiency rather than raw byte volume. Keeping it within the scale-up domain of a node or rack limits both latency and operational complexity.

This hierarchy is easier and cheaper to build than a machine in which every link is equally strong. Scale-up paths surround the operations that exchange embeddings and partial results within every layer. Scale-out links carry pipeline activations, KV movement, scheduling traffic, and traffic between bounded workers.

The tiers also have different latency requirements. A pipeline transfer between layer groups may barely notice a few microseconds if throughput is sufficient. By contrast, when an expert load and multiply take only a few microseconds, an additional 100 ns at each routing hop is material. The closest links should therefore be integrated tightly with the accelerator fabric, while less frequent transfers can tolerate the scale-out network.

13. Pipeline by layer group where practical

The layer group is the natural first candidate for pipeline placement. In a classic MoE model the group repeats one kind of layer. In newer structures it may contain one full-attention layer followed by several optimized- or localized-attention layers.

Keeping a group together can preserve internal dependencies and allow one placement and compilation plan to repeat across stages. Uniform plans reduce tooling cost and simplify deployment. Kernels, memory layouts, communication schedules, monitoring, and failure recovery all become easier when each stage runs the same recurring structure.

It is still possible to cut a pipeline in the middle of a group, although the resulting placement is more complex. If the layers have no special cross-layer dependency, the compact activation boundary remains a low-volume link suitable for networking. A stage may contain several whole groups, one group, or part of a large group depending on memory and throughput requirements.

The guiding principle is not an absolute prohibition on splitting groups. It is to prefer regular boundaries and to understand when a nonuniform split is worthwhile. Tooling and operational simplicity can be as important as a small theoretical placement gain.

14. Pipeline depth ends at the KV-cache crossover

Pipeline parallelism divides model weights cleanly. If a model has weight footprint (W) and uses (P) stages, the idealized weight footprint per stage approaches (W/P).

KV-cache demand behaves differently. A full pipeline remains occupied by running a different query or batch in every stage. In an eight-stage pipeline, each stage owns one eighth of the layer-wise KV values for each of eight live streams. The reduction in layers per stage is offset by the number of simultaneous streams, so the stage does not receive the same (1/P) reduction in live KV demand that it receives for weights.

A useful stage-memory estimate is:

M_{stage}\approx \frac{W}{P}+M_{KV,stage}+M_{working}+M_{slack}

Once weights per stage fall to the same order as KV and working-state memory, deeper pipelining produces little additional capacity benefit while adding coordination and latency.

The crossover depends strongly on model design. Classical full-attention approaches can require hundreds of kilobytes of KV state per token across all layers. MLA-like designs are commonly in the tens of kilobytes, while recent hybrid layer groups with mostly linear, recurrent, or local attention can be lower still. A working average near 25 kB per token may be reasonable for some new hybrids; approximately 70 kB per token is a conservative reference point for a proven compressed-attention design.

As a back-of-the-envelope design target, take 70 kB per token and two million tokens of aggregate live context assigned to one stage. That is roughly 140 GB of KV state. Double buffering allows one working set to move while another continues to run, raising the KV allowance toward 280 GB. Add model weights, activations, transient buffers, routing tables, fragmentation, and operating margin, and a stage target of roughly 400–500 GB of local fast memory becomes a useful 2027 benchmark.

Other designs can reduce requirements through streaming, compression, partial residency, or additional tiers. The estimate is not a universal requirement; it is a standard that a practical frontier system should be able to accommodate without heroic assumptions.

The stage can distribute this memory over several accelerator sockets if they share a scale-up fabric. More than 95% of the weight memory in a large MoE is in routed experts, which divide naturally among sockets. KV state can be partitioned by head, context, or attention instance, with the resulting collectives matched to the same local fabric.

15. Prefill and midfill turn sparse routes into dense expert batches

Prefill and midfill have strong batching opportunities because many input-sequence tokens complete attention before expert execution begins. The router can score all tokens, emit several destinations for each one, and then sort the route records by expert.

A practical input block can contain enough assignments that essentially every expert receives work. If 128 new tokens each select eight experts, the router emits 1,024 assignments across a 256-expert bank. Routing remains sparse for each token; sorting changes the execution order so that expert e000 processes all of its assigned tokens, then e001, and so on. Each expert is loaded once and reused for its compact token list instead of being repeatedly loaded in token order.

For an expert with (W) weights, weight traffic (B_W), activation traffic (B_A) per routed token, output traffic (B_O), and (n) assigned tokens, a rough arithmetic-intensity expression is:

AI_{expert}\approx \frac{2nW}{B_{W}+nB_{A}+B_{O}}

When weight traffic dominates, increasing (n) rapidly improves reuse. Eventually activation, output, and routing traffic become important and the gain flattens.

The sort is not free. Route records must be built, counted, permuted, sent to expert destinations, and restored to token order. Expert capacity limits can cause overflow or rerouting. Even so, prefill and midfill begin with enough tokens that the optimization is usually meaningful.

This is the key distinction from decode. The experts are identical; what changes is the number of routed activations available when the worker executes them.

16. Decode gains less from batching

Decode attention receives little direct cross-query reuse. Each additional query contributes another KV cache and another context scan. Shared query-projection weights can be reused, and software overhead can be amortized, but the dominant long-context read remains private to the query.

Decode experts offer potential sharing because the expert weights are context-free. The practical limitation is sharing probability. A model may contain hundreds of routed experts in every layer and thousands of layer-experts across the model. Each query contributes one new token per pass and selects only a few destinations. At small batch sizes, only a few routed tokens are available and most active experts receive work from just one token. Some sharing occurs, but it grows gradually with the number of tokens in the pass.

Shared transformations provide more conventional weight reuse, but they are a smaller fraction of an MoE model. The overall throughput gain from enlarging a decode batch can therefore be modest compared with the increase in latency and KV-memory pressure.

A batch of one is not a trivial workload when context length is 100,000 tokens or more. The attention scan already occupies substantial memory bandwidth. The optimum batch size is not necessarily one—software launch, allocation, routing, and hardware utilization still matter—but it may be much closer to one or five than to the large batches associated with throughput-oriented serving.

This matters economically. Interactive tokens can be more valuable than bulk delayed tokens. Multiplying latency to gain a fractional throughput improvement may be a poor trade. Hardware and software should remove impediments to useful small-batch operation rather than making large batches a prerequisite for efficiency.

17. Scheduling makes disaggregation useful

Disaggregation becomes most effective as a statistical system at cloud scale. A large pool can contain workers configured for long prefill, short prefill, long-context decode, short-context decode, or other useful classes. The orchestrator assigns each job to a suitable available worker and moves the required KV blobs through shared storage.

Prefill and midfill work are predictable once the prefix size, new-token count, and worker configuration are known. Decode is stochastic because output length is uncertain. Storage between the pools decouples predictable production from irregular consumption. A ready buffer of completed prefill and midfill work can keep decode workers fed even as individual queries finish at different times.

The scheduler should observe customer workloads and time-of-day patterns, anticipate known ramps, and reconfigure workers as the mix changes. Reconfiguration may mean loading different model stages, changing pipeline cardinality, reallocating expert capacity, or moving a rack among prefill, midfill, and decode.

Scale matters. In a small fixed benchmark, one input-fill stage may be connected directly to one decode stage. A slow stage can strand capacity elsewhere in the other stage. In a large pool, workers complete and start independently so no worker need be stranded.

High interactivity and high utilization are therefore not inherently opposed. The scheduler can keep batches small while keeping machines busy, provided it has enough workers, enough storage bandwidth, enough visibility into the work, and fast enough response times to adapt to the moment.

The service overview in Figure 1 provides the corresponding system picture: four resource pools with the orchestrator spanning beneath them.

18. Feedback can turn a small imbalance into oscillation

A tightly coupled system can amplify ordinary variation. Suppose admission to input fill is gated directly by the retirement of decode queries. A long-running decode query delays retirement. Input-fill admission falls. Decode later drains its ready queue and becomes underfed. Admission then opens aggressively, producing a burst of input-fill jobs and a new wave of congestion.

This is a delayed feedback loop. The control signal arrives after the system state has changed. Large batch recovery can make the loop worse by increasing KV-memory pressure and response latency. The factory alternates between full and empty rather than operating near a steady point.

No single scheduler policy solves the problem. Several familiar control techniques help:

• use ready-work buffers between stages;

• regulate admission from smoothed backlog and predicted service rate rather than individual retirements;

• separate fast local controls from slower capacity controls;

• apply hysteresis to worker reconfiguration;

• reserve capacity for short interactive work;

• and avoid changing batch size more quickly than the system can observe its effect.

The same principle extends across timescales. Microsecond worker scheduling, millisecond token flow, second-scale buffers, minute-scale worker reconfiguration, and hour-scale demand planning should not all respond to the same noisy signal.

19. Aggregation Compared to Disaggregation

Now that we have exhaustively described disaggregated inference, is there a case to be made for aggregation? To some extent, yes. Clearly, since conversations can pause for minutes and hours it is not reasonable to keep a whole conversation tied to one system. That HBM can be put to better use within seconds of the end of a turn, and modern scaleout networks can move gigabytes of KV cache to a cheaper place in a parallel storage system, which can tier DRAM and SSD, in less than a second.

However, during a turn, do we need to schedule more than one place to get the work done? It is expensive to have too many different designs of server, adding operational costs as well as design work. What if there were only one design, and it centers on the midfill case? A request can be scheduled to that empty worker and stay there for prefill, midfill, and decode. It would need to have enough compute for prefill and midfill, and enough memory bandwidth for decode. There would always be some stranded capacity for either compute or memory transfer, depending on which stage the model is running, but we already tolerate low MFUs on GPUs. We try to overcome that with clever kernels, but in practice clusters run with stranded functions all the time.

Complex orchestration will result in some stranding. Some of that is avoided if a turn runs on a single aggregated worker. No schedule gap when pre/midfill hands over to decode. Natural sharing of one copy of the wide parallel set of experts. No network overhead to move the context from fill to decode. The orchestrator still looks at the shape of the request – cache size, input sequence length, model type, customer interactivity level – and makes one decision which available worker should handle it through all stages of the turn. Then it leaves it alone until the stop token comes out and the turn is delivered to the user and cached to the fast parallel file system. The one worker uses kernels and dataflows adapted to the given request shape as it works through all stages of the turn.

Aggregated workers have their attractions and advocates.

All systems will always strand some capabilities of the accelerators in real world uses. In a disaggregated system there can be two (or more) types of machines built with investment focused on compute, memory, and network actually used, arguably the best use of that investment. Prefill machines might invest in compute, while decode machines might invest in memory. If workers are aggregated, then one machine will be used through all phases so it needs to invest in the strongest compute and the strongest memory thruput. Maybe also strong network. If is not an all-around standout, then it will be held back by its weaker sections and be an overall “meh”. So, aggregation has advantages but it also leaves nowhere to hide in delivering the best of all-around performance. If you can build that magic all around star, then aggregating the whole turn in one worker could be best.

20. Memory bandwidth and memory capacity are different objectives

Memory is often discussed as two independent scalars to maximize. In practice, capacity and bandwidth trade against each other, and their economic value must be considered together.

Data earns revenue when it moves; it incurs cost while sitting idle. An accelerator with very high memory throughput can produce tokens at a rate that lower-bandwidth memory cannot match, even if the lower-bandwidth configuration has more capacity. Capacity is valuable until the active weights, KV state, activations, and operating slack fit cleanly. Beyond that point, additional local capacity may add cost but not revenue.

The capacity curve is not a vertical cliff. Below the useful limit, adding accelerators supplies both more memory and more bandwidth. The advantages end when other limits appear: maximum scale-up width, poor model cuts, the dominance of latencies, extra synchronization, or network contention.

A safe-capacity design point must sit above those limits. It includes margin for realistic context distributions, double buffering, fragmentation, failures, and transient peaks. The revenue-capacity curve rises steeply before that point, rounds smoothly, and then reaches a flatter ceiling determined mainly by bandwidth and compute. Revenue may even decline if excess capacity becomes a sunk cost that operations try to justify with big slow batches of high throughput but inferior value.

Accelerator HBM is too expensive to hold idle data without immediate use. Completed KV blobs should move into CPU DRAM for outgoing staging and then into scale-out storage. Frequently reused common blobs may remain replicated in shared CPU memory or nearby cache appliances. Only the active working set can pay for the most expensive memory.

Tall stacks of high-capacity memory may impose costs in power, signal quality, packaging, and frequency. A design with just enough capacity and exceptional bandwidth may serve interactive inference at higher margins than one that maximizes capacity per accelerator with idle excess.

21. A practical mapping method

The prior sections can be condensed into a repeatable method for mapping a model onto a machine.

1. Separate the operating regimes. Measure prefill, midfill, decode attention, and decode experts independently. Each may split further by context length or workload class—for example agents, coding, media, or content creation. These become prepared configurations that the orchestrator can mix in production.

2. Describe the repeating layer group. Identify full and optimized attention, shared transforms, routed experts, and any cross-layer dependency.

3. Measure active state rather than total parameters alone. Separate stage weights, active experts, live activations, KV state, and transient buffers.

4. Choose the natural parallel dimension for each operation. Pipeline the layer stack, tensor-parallelize broad indivisible work, and distribute independent experts widely.

5. Map the broadest operation first. Long-context attention often determines memory striping and the minimum useful scale-up participation.

6. Place expert weights in regular local destinations. Let routing select among known peer locations rather than creating a global event for every token. A large model may contain on the order of 15,000 layer-experts, and each can be placed with regard to the other experts in its layer and the expected routing distribution. The goal is even use of memory capacity and bandwidth.

7. Keep repeated high-throughput cooperation inside the strongest fabric. Let thinner activations and amortized KV movement cross weaker boundaries. Respect latency and power cliffs as well as throughput cliffs.

8. Stop increasing pipeline depth at the KV-memory crossover. Once the model and target context fit with safe margin, additional stages bring little memory benefit and add stage-transition latency.

9. Use input fill and decode differently. Sort prefill and midfill routes into expert batches; preserve small-batch decode interactivity.

10. Design storage and scheduling with the worker. KV movement, admission control, ready buffers, and worker reconfiguration determine whether the physical mapping produces steady throughput. Short batches and continuous batch updates let the orchestrator replace individual requests without waiting for the whole batch to retire.

The method does not select one universal topology. It makes the trade-offs visible. A peer set may be one package, one board, or a rack-wide scale-up domain. A layer-group crêpe may map to one node or several. The important point is that the communication hierarchy follows the model’s actual flow.

22. The next view is time

The diagrams so far describe structure and flow. The next step is to place time over them.

Interactive decode has a tight end-to-end budget. Execution of one layer commonly needs to fit within roughly 50 microseconds to 1 millisecond, which across a typical 60-layer model corresponds to roughly 15 to 300 generated tokens per second. Each layer contains several sequential phases, so individual arithmetic operations and handoffs live in tens of microseconds, microseconds, and sometimes fractions of a microsecond.

At that scale, tensor arithmetic is only part of the result. Kernel launch, queueing, synchronization, memory setup, routing, reduction, and link latency must all remain small compared with the operation they support. An expert multiply that takes a fraction of a microsecond is of diluted value if dispatch and data movement take several microseconds each.

The timing view also sharpens the network hierarchy. A high-bandwidth path may still be unsuitable if its setup latency is too large. A small local buffer can be more valuable than a large distant memory tier for a sub-microsecond handoff. A scheduler that creates millisecond variation can overwhelm careful microsecond engineering inside the worker.

The factory spans several control timescales:

• local arithmetic and link handoffs in fractions of a microsecond to microseconds;

• layer execution in roughly 50 microseconds to 1 millisecond;

• token generation in milliseconds;

• ready buffers and admission control in roughly 0.1 to 10 seconds;

• worker reconfiguration by the orchestrator over seconds to minutes;

• KV-blob retention and lifecycle in the fast parallel file system over minutes to days;

• and customer demand planning over hours and days.

These loops should be designed separately and joined carefully. Fast controls should not chase slow demand noise, and slow controls should not react to every transient queue fluctuation.

23. Worked example: projected Kimi K3 performance on Blackwell systems

The preceding sections describe a method for reasoning about model placement. A useful check is to apply it to a specific frontier model and explore the resulting trade-offs together. The following charts show modeled Kimi K3 performance on B200, B300, and GB200 systems using configurations from 16 to 64 GPUs in a rack. These are workload-and-hardware projections, not measured benchmark results. Connected lines identify projected Pareto frontiers: configurations for which interactivity cannot improve without giving up throughput or another selected objective.

22.1 Interactivity and throughput

The first chart compares time per output token for decode, or time to first token for prefill and midfill, against tokens per second per GPU. Normalizing throughput per GPU makes configurations of different sizes directly comparable.

The lower-left group contains decode frontiers for 8k and 32k input contexts. Feasible configurations appear for all three GPU types. GB200 is strong across much of the frontier, while B200 and B300 also offer competitive points. The lowest-latency decode configurations generally combine pipeline and tensor parallelism. At the throughput-oriented end, the highest tokens per second per GPU tend to come from one attention instance per GPU, without tensor or pipeline parallelism, while expert parallelism remains as wide as the system permits.

The upper-middle group is midfill: 127k cached input tokens followed by a 1k-token append before decode. GB200 leads the low-TTFT portion without pipeline parallelism, using TP=4 or TP=2. B300 catches up toward the throughput-oriented end. B200 and B300 generally use PP=4, with expert parallelism bounded within each stage so the expert all-to-all remains inside the node.

The two groups at upper right are 8k- and 32k-token prefills. All three GPU types converge on similar configurations: PP=4, with TP=4 or TP=2 favoring lower TTFT at some cost in throughput. Their close projected performance reflects the shared Blackwell compute architecture and the compute-limited nature of these prefill workloads.

22.2 Energy per query

The second chart uses the same simulated configurations but changes the vertical axis to energy per completed query. Decode is the largest energy workload because the modeled output length is 1,000 tokens, requiring roughly 1,000 passes through the model for every query. Midfill uses the least energy: it appends a relatively short sequence after one pass through an existing long context. Full 8k and 32k prefills require successively more energy.

The model includes Kimi K3’s localized-attention layers, so both retained state and computation reflect the expected savings from the attention structure rather than treating every layer as full attention.

22.3 Fast-memory requirements

The third chart shows peak HBM residency per GPU. It excludes the extra buffers used while KV state enters or leaves the worker, which may add several gigabytes per GPU. Between conversational or agentic turns, inactive KV blobs can be transferred in parallel to network-attached DDR, including the DRAM attached to node CPUs. That memory is cheaper, more available, and less constrained by the HBM supply chain.

The projected frontiers cluster near the lower end of the feasible memory range. For prefill and midfill, the downward steps come from adding pipeline parallelism and increasing tensor or data parallelism, including data-parallel ring-batched instances. Decode reduces per-GPU residency first through tensor or data parallelism, using pipeline parallelism later.

Most frontier configurations remain below approximately 80 GB per GPU even after allowing for practical KV-transfer buffers. This supports the earlier conclusion that bandwidth can be more valuable than maximizing HBM capacity on every accelerator, provided the broader storage and orchestration system moves idle state promptly.

22.4 Network peaks and the scale-up boundary

The fourth chart shows projected peak network traffic per GPU. Prefill and midfill make intensive use of the network for expert parallelism because many routed activations are in flight at once. Pipeline parallelism is therefore particularly useful on B200 and B300 systems: placing one stage inside a node keeps its expert all-to-all traffic on the local NVLink fabric.

GB200 NVL72 extends scale-up bandwidth across the rack. More GPUs can therefore participate in one layer in parallel, with all instances advancing through that layer in synchrony, to improve latency and throughput without crossing a weaker network boundary.

Decode shows lower interval-averaged peaks because a small batch consults relatively few experts at once. Individual transfers still burst at the physical NVLink rate; the lower plotted value reflects a measurement interval longer than the transfers themselves. This distinction between instantaneous link rate and sustained traffic matters when sizing the fabric and interpreting utilization.

The four views tell a consistent story. The best configuration changes with the operating regime and the selected point on the latency-throughput frontier. Prefill and midfill reward organized parallelism and stage-local expert traffic. Decode benefits from parallelism at the low-latency end, but the highest throughput per GPU comes from simpler attention instances combined with wide expert placement. Across the projected frontiers, fast-memory capacity appears less restrictive than network placement, memory bandwidth, and orchestration.

Closing perspective

The useful unit of analysis for inference is not total parameter count alone. It is the active flow of the model joined to the memory, communication, and timing hierarchy of the machine.

MoE provides structure. Layer groups repeat. Experts are numerous but individually manageable. Attention is broad and increasingly shaped by KV state. Operations occur in sequence and can reuse hardware. Pipeline, tensor, and expert parallelism can each follow the dimension where it is most natural. Prefill and midfill can sort routes into efficient expert batches, while decode can preserve short batches and high interactivity.

The hardware can reflect that structure. A logical accelerator peer set provides a bounded scale-up domain. Nodes stack into racks. A rack may be one scale-up worker or several pipeline stages. Shared storage makes KV blobs movable. Datacenter networking connects bounded workers rather than carrying every inner tensor operation. Scheduling keeps predictable and stochastic stages in steady flow.

Holding the problem this way does not remove its scale. It makes that scale easier to divide. The model, worker, rack, storage system, and scheduler become parts of one coherent design that can be measured, illustrated, and improved.

Read more

With Engram model architecture optimization, it allows for lower HBM capacity to be needed for models at the same quality. This does not mean there won’t be an insane demand for HBM but it just means that model architecture will continue to innovate around constraints.

This model architecture design is naturally codesigned for parameter offloading: each token accesses a few embedding rows whose addresses depend on token IDs, not hidden states. The runtime can prefetch those rows from host DRAM while earlier layers compute, keeping the table outside HBM without transferring entire weight matrices. Our Memory model contains our latest estimates of quarter by quarter HBM, DRAM, & NAND supply and demand.

Offloading frees HBM for model weights and KV cache, potentially supporting larger batches or more concurrent sessions. When DRAM becomes the next constraint, NVMe offers another tier. Recommendation systems already cache frequently or recently accessed embedding rows in faster memory while backing colder rows with SSDs. After NVIDIA roadmap had to change due to massively despec’ing Rubin Ultra from 1024GB to now ~200GB of HBM per chip, model architecture optimizations like emgram maybe helpful.

Our DeepSeek-V4.1-Flash configuration uses roughly 189 GiB of memory for Engram. We replace it with a memory-mapped (mmap) file and measure serving performance with offload to SSD.

Later on in our report, we will show our Engram offloading experiments along with the official InferenceX agentic inference serving results on Engram models like DeepSeekv4.1 Flash across all 6 NVIDIA GPU SKUs along with MI355X. Unsurprisingly, the CUDA Moat is still mogging MI355X on the ultra popular DeepSeekV4.1 Flash model.

We also show how even on high capacity HBM SKUs, offloading emgrams to DRAM could result in even better performance for most of the pareto than keeping the emgram in HBM.

Our benchmark has been widely reproduced, validated and/or supported by almost every major buyer of compute from Google Cloud to Microsoft Azure to Oracle, to Meta and many more. Furthermore, it has the support of the ML community including from vLLM, LMCache, SGLang, PyTorch, Huggingface and the support of major labs like OpenAI, MiniMax, ZAI, Qwen, Moonshot Kimi, etc.

Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. InferenceX is the only inference benchmark in the world to have TPUv7, Jalapeño, Nvidia Rubin NVL72, AMD, and soon, SambaNova and Trainium. Due to how realistic AgentX scenario is to real world agentic inference workloads, AMD has committed to collaborating on MI455X UALoE72 too.

Engram Benefits

DeepSeek did not release the original paper’s two trained Engram models. We replicated its setup on fineweb-edu using the released code and training hyperparameters, at an estimated 6E18 FLOPs per run. We observed the same U-shape scaling:

Engram improved performance over pure MoE baselines. We also reproduced DeepSeek’s results, where earlier-layer representations with Engram resembled those of later layers.

What is Being Memorized?

Like the original Engram paper, it is possible to probe Engram’s gate scores to see what n-grams DeepSeek-V4.1-Flash makes the most use of. Our gate scan finds names, code fragments, relational phrasing, and boilerplate. These examples prioritize interesting-ness over gate strength.

One unexpected result was Wright : Ace Attorney.

These examples suggest learned memory optimizes the training objective, not a judgment of which facts deserve storage. Licenses, bibliography fragments, API scaffolding, and website furniture can provide prediction shortcuts, so the value of additional Engram capacity may depend on what survives data preparation. This does not show that table capacity is “wasted”: the evaluation-corpus scan establishes neither training exposure nor the capacity occupied by each category.

For offloading, strong gates do not identify cache-hot rows. Low gates do not automatically save reads either: computing the gate requires the retrieved key and undoes the performance gain of a fused kernel. Skipping reads would require a separate usefulness predictor before retrieval.

Removing Engram

In the original paper’s inference-time ablation, factual-knowledge benchmarks retained just 29–44% of their original performance, while reading comprehension retained 81–93%. This is due to the training–inference mismatch. The resulting degradation therefore measures this trained model’s dependence on Engram, not the performance difference between models trained with and without it.

In our ablations, suppressing Engram worsens token likelihood across all evaluated domains, especially encyclopedia text and several code corpora. Surprisingly, GSM8K accuracy stays within measured run-to-run variation and removing Engram has no effect.

Engram is not a detachable dictionary beside an unchanged MoE. Removing it changes downstream features and expert selection.

We tested whether rerouting hurts or compensates by holding tokens fixed in a teacher-forced experiment on CRUXEval, a code-reasoning benchmark of small Python functions where the model predicts a function’s output from its code and an input, and scoring the reference answer.

Removing Engram raised answer loss from 0.2848 to 0.3093 bits/token. Forcing the ablated model to use the original Engram-on expert choices made it worse still, at 0.3375 bits/token.

Rerouting partially compensates for the missing memory. Memory features and expert selection work together, rather than following a clean “memory stores facts; experts reason” division.

On the same CRUXeval, removing Engram during either phase reduced accuracy and increased generated tokens; removing it throughout produced the largest changes.

Keeping Engram for prefill leads to more correct answers than keeping for decode likely due to semantically richer KV cache transferred to decode workers, allowing it to mitigate some of the performance loss.

Agentic Inference Serving Performance

Engram’s table is large, but each lookup is small. DeepSeek-V4.1-Flash requests 24 rows at each of two Engram layers, about 12.4 KiB per processed token position across the model, or 3.1 KiB per GPU when split across four GPUs.

Currently as of Day 7 since Model Release, MI355X is still 2-4x worse performance per dollar compared to B200 even when normalized by Mi355X’s lower TCO. Our full total cost of ownership breakdown comes from our AI Cloud TCO Model along with monthly market surveys of over 100+ gpu clouds & gpu cloud customers.

On the Day 0 release of DeepSeekv4.1 Flash, NVIDIA vLLM works out of the box with zero issues across all 6 SKUs: H100, H200, B200, B300, GB200, GB300! This was thanks to the amazing work by the NVIDIA & Interact teams! In comparison, AMD vLLM did not work on day 0 for DeepSeekv4.1 Flash.Source: SemiAnalysis InferenceX

AMD’s vLLM documentation points to using vllm/vllm-openai-rocm:deepseekv41-flash-0909, but from hour 0 of the model release to hour 23, AMD has not publicly released the image. AMD claims, “SPEED IS THE MOAT,” yet it has still not released it by the 23rd hour. We wish that, going forward, the AMD team has a better process for hour 0 model releases.

Eventually when they did publicly release for “day 0” image support, performance-wise, it is currently up to 14.8x worse perf per dollar than H200 and up to 42x worse perf per dollar than B200/B300.

The power of the CUDA MOAT is NVIDIA’s collaboration with its massive 6 million-developer community ecosystem including most of the vLLM & SGLang & Tokenspeed maintainers which means that CUDA is optimized on day 0.

Overall, AMD did make significant improvements but the performance per dollar is still currently 2-4x worse than B200.

AgentX Engram DRAM Offloading Improving Performance

The HBM and DRAM offload use the same GPU kernel to select and dequantize rows. With HBM, it reads GPU memory; with Unified Virtual Addressing (UVA), it reads pinned host memory directly.

Both support full decode graphs. Moving the table into HBM accelerates only the sparse lookup, leaving decoder computation and communication unchanged, resulting in little overall benefit while consuming memory otherwise available to KV cache.

Another benefit of Engram offloaded to DRAM which means you can reduce the communication overhead by using less HBM GPUs per replica. For example, when enabling Engram offloading on B300, we are able to switch from TP4 to now TP2 which improves the pareto curve by up to 1.6x.