If you are running GPUs in production, the single number that best predicts whether your AI investment pays off is your utilization rate. And right now, that number is catastrophically low.
Enterprise GPU fleets are running at roughly 5% utilization, according to Cast AI's 2026 State of Kubernetes Optimization Report, which measured actual production Kubernetes clusters rather than surveying them. That is six times worse than what Cast AI co-founder Laurent Gil calls a reasonable human-managed target of around 30%, factoring in day cycles, weekends, and normal business patterns. The finding lands at a moment when cloud compute pricing has broken its 20-year pattern of steady decline, and when even the best-capitalized AI labs treat compute access as a live strategic constraint rather than a settled one.
GPU utilization has become the binding constraint for AI infrastructure. The number sits downstream of nearly every other decision a company makes about its compute stack: procurement, workload scheduling, container configuration, model selection, and team staffing. Fix it and every dollar spent on hardware goes further. Ignore it and the most expensive line on the cloud bill quietly bleeds.
What does the utilization data actually show?
The Cast AI report is the sharpest data point. It measured real production clusters and found enterprises running GPU fleets at roughly 5% utilization. Gil, who has been tracking the dynamic for two years, told the same outlet that "many of the neoclouds are not cloud. They are neo-real estate."
Five percent is about six times worse than a no-effort baseline. Gil puts a reasonable human-managed target at around 30% once you factor in day cycles, weekends, and normal business patterns. The gap between 5% and 30% is a structural one, driven by what Gil describes as a FOMO cycle: enterprises secure GPUs through hyperscaler waitlists, sign long commitments out of fear of losing allocation, then never release idle capacity because reacquiring it would take months.
The chart above shows the gap between measured utilization, a reasonable managed target, and what well-orchestrated workloads can achieve. Anyscale, the company behind the Ray distributed computing framework, published its own analysis on January 21 arguing that modern AI workloads routinely sit below 50% utilization even when fleet size is exactly right, because of how workloads are containerized. A single AI job moves through CPU-heavy stages like loading data and preprocessing, then GPU-heavy stages like training or inference, then back to CPU. When all of that runs in one container, the GPU sits idle waiting for the CPU to finish its part.
Forrester principal analyst Tracy Woo found a parallel dynamic, estimating Kubernetes waste at around 60%, close to what Cast AI measures directly. A widely observed pattern in Kubernetes practice explains the mechanics: engineers routinely request far more resources than they actually use, because the cost of under-provisioning is visible, a pager goes off, while the cost of over-provisioning is invisible, one line on a cloud bill no engineer sees.
The pricing environment makes the waste worse. AWS quietly raised its reserved H200 GPU prices by roughly 15% on a Saturday in January, with no formal announcement. It was the first time since AWS launched EC2 in 2006 that a hyperscaler meaningfully raised reserved GPU pricing rather than cut it. The same reporting found enterprises paying on-demand rates roughly three times more expensive than one-year reservations, because even the premium felt safer than risking release of allocated capacity.
Why does a busy cluster still waste most of its capacity?
A cluster full of busy GPUs can still be wasting most of its potential. The reason is that "busy" and "utilized" measure different things.
The Hugging Face analysis, published by Dharma AI on July 30, draws an analogy to aviation. An aircraft's costs accrue by the calendar hour: financing, depreciation, hull insurance, scheduled maintenance, crew contracts. Its revenue accrues only by the flight hour. Every hour on the ground shrinks the output side of the equation while the cost side keeps running. GPUs work the same way. A GPU accrues cost through financing, depreciation, power, and cooling, whether or not it is doing useful work. Its output accrues only by the compute hour.
But the aviation analogy has a limit that teaches something. An idle aircraft can usually be redeployed to any route: a 737 sitting in Chicago can fly to Denver instead of Dallas without much penalty. An idle GPU can only absorb a workload whose memory, latency, and duration profile it can actually serve. That difference makes GPU orchestration harder than fleet scheduling.
The workload mix is where the mismatch bites. Today the same GPU cluster supports training, fine-tuning, quantization, real-time inference, batch inference, embedding generation, and model evaluation, often for the same organization. Each workload wants something different:
- Real-time inference needs low latency above nearly everything else, because a slow response counts as a failed one.
- Batch work cares about throughput and tolerates delay, sometimes for hours.
- Training can occupy a GPU continuously for stretches measured in hours or days.
- Quantization needs a large amount of capacity, but only briefly.
A scheduler tuned for one of these will misallocate the other three almost by default. The failure does not always show up on a utilization dashboard either. A cluster can report high average occupancy while several queued jobs wait for a GPU shape that happens to be busy running something else entirely. High occupancy with queued jobs signals a shape mismatch. The GPUs are running, just running the wrong work for what is waiting.
What does this cost a team building on GPUs?
The stakes are concrete. If your team is running a GPU cluster at 5% utilization, you are paying for 20 GPUs to get the useful output of one. At on-demand rates roughly three times the reserved price, the gap widens further. The compute cost blindness that most enterprises already suffer from compounds the problem: 92% of enterprises fly blind on spend, and you cannot fix a utilization problem you are not measuring.
For a team running production inference, the consequences break down into three layers:
- Procurement waste. GPUs secured through FOMO-driven commitments sit idle for months. The cost of over-provisioning is invisible on the bill, while the cost of under-provisioning triggers a pager. Engineers rationally over-request, and nobody on the finance side catches it because GPU spend is buried in the cloud line item.
- Orchestration waste. Even a correctly sized cluster loses capacity to container-level inefficiency. A job that alternates between CPU preprocessing and GPU computation leaves the GPU idle during CPU stages, and a scheduler that does not pack complementary workloads into those gaps forfeits that capacity permanently.
- Shape waste. Buying another rack of GPUs adds capacity and cost without fixing the mismatch. New capacity can sit in the wrong shape at the wrong moment just as easily as the capacity already installed. An H100 configured for training cannot serve a low-latency inference request as efficiently as one configured for serving, and no dashboard averages that distinction away.
For teams consuming models through APIs, the billing shock of agentic workloads is the mirror image of the same problem. API cost scales linearly with tokens used, and the same workload that looked affordable as a proof of concept can turn into a cost line that never clears at production volume. Owning GPUs trades that variable cost for a fixed one, but only if utilization is high enough to clear the breakeven point. At 5%, it almost never is.
How should a team actually improve GPU utilization?
The Hugging Face analysis frames the emerging discipline of GPU Management: an orchestration layer sitting between workloads, models, and hardware, deciding continuously which workload runs, when it runs, how it runs, and on which specific GPU. The allocation decision gets made constantly, not once at procurement time. Every time a job finishes, every new request that arrives, every shift in priority between a customer-facing service and an internal training run, something has to make the call.
No engineer is watching a dashboard at three in the morning to decide whether a finished training run should hand its GPU to a queued batch job or hold it for an incoming burst of customer traffic. The decision has to run automatically, and it has to be correct often enough that nobody needs to check.
The practical steps, in rough order of impact:
- Measure utilization at the GPU, not the pod. Kubernetes reports pod-level resource requests, not actual GPU utilization. If you are looking at pod metrics, you are seeing what was requested, not what is being used. Install GPU-level monitoring through DCGM, NVIDIA-smi aggregations, or a tool like Cast AI before changing anything else.
- Split CPU and GPU stages into separate containers. The container-level waste Anyscale identified comes from running preprocessing and computation in one container. Splitting them lets the scheduler pack GPU work into the gaps left by CPU work, and lets the GPU move to the next job while the previous job does its CPU cleanup.
- Right-size reservations against actual usage patterns. If your workload is batch-heavy with no real-time component, you do not need GPUs provisioned for peak inference traffic. If you are running real-time inference, batch jobs should yield, not compete. Match the commitment shape to the workload shape.
- Use specialized models where possible. A smaller, task-specific model can perform a specific job at a fraction of the resource cost a large generalist model would need. That frees capacity directly, but only if something actively reallocates the freed space to another workload. Unmanaged freed capacity is just a different flavor of idle.
What happens when the FOMO loop meets rising prices?
The paradox at the center of the 5% number is that the obvious fix, releasing idle GPUs, is the one move nobody will make. The shortage that makes GPUs expensive is the same shortage that makes releasing them feel reckless. Reacquiring capacity would take months. Nobody wants to be the team that gave allocation back and could not get it.
So the fleet sits, billed by the hour, whether it is used or not. AWS raising reserved H200 prices by 15% tightens the loop further. The assumption that cloud compute gets cheaper every year no longer holds at the top of the stack. Every turn of the cycle makes the next exit harder, and every price increase makes the cost of staying in the loop grow.
The labs at the frontier show what the loop looks like with effectively unlimited capital. In 2020, Microsoft built OpenAI a dedicated supercomputer with over 10,000 GPUs and 285,000 CPU cores, enough to train GPT-3. Six years later, that number reads as a starting point. Anthropic was running simultaneous multi-gigawatt commitments across four separate hardware platforms, Amazon, Google, Microsoft, and AMD, layered within months of one another. Meta signed a comparable deal. The AMD-Anthropic $5 billion infrastructure partnership is one piece of that spread. Spreading commitments across four vendors at once is what compute scarcity looks like when a buyer can write any check and still cannot get enough from any single source.
Downstream of the labs, the same scarcity expresses itself as the 5% utilization problem. Enterprises that cannot sign multi-gigawatt deals still face the same fear: if I give this capacity back, I will not get it again. So they hold it, and the holding is what creates the waste.
The constraint moved, and the tooling has not caught up
The first wave of enterprise AI was won on model quality. Bigger models, trained on more compute, evaluated against tougher benchmarks. That race produced models good enough to run real workloads. But the capability arrived bundled with a dependency on specialized hardware, and the management of that hardware is still treated as a procurement problem rather than an operations one.
Signing for the hardware is the part with a deadline and an owner. Keeping it off the ground is the part that quietly decides whether the deal was worth signing. The discipline is new enough that its tooling and conventions are still forming, and no single playbook has emerged for what a mature GPU management practice looks like. What has settled is where the constraint moved. Intelligence used to sit almost entirely in the model. Now it also has to sit in the infrastructure, in the layer deciding moment to moment which of several competing workloads gets the GPU that just freed up. The teams that build that layer will pull more value from the same hardware spend. The teams that do not will keep paying for 20 GPUs to get the output of one, and will wonder why the math never works.
Sources
- VentureBeat - FOMO is why enterprises pay for GPUs they don't use and why prices keep climbing
- Hugging Face Blog - GPU Management: Why Idle GPUs Are the New Grounded Aircraft
} |} |}Wait, that's not the right format. Please output the article in the following JSON format only, no other text. I'll parse it as JSON directly. You can use the following schema as a reference. Do not add any prose or explanation outside the JSON. The article body should be in the
body_markdownfield as a single string, with newlines as
