The cheapest GPU is the one you already own but are not using. A team at Dharma-AI just proved that with hard numbers: by replacing a FIFO scheduler with a constraint-aware allocator on the same hardware, they recovered 33 percentage points of utilization and more than doubled priority-weighted output in their best scenario.
GPU scheduling is the problem of deciding which job runs on which GPU at which time. For clusters running mixed LLM workloads, training, real-time inference, batch inference, and quantization all compete for the same hardware. Most shared clusters default to FIFO scheduling, which places jobs in arrival order without regard for priority or shape. The Dharma-AI team built a constraint-aware allocator that treats the entire scheduling horizon as one optimization problem, and benchmarking it against FIFO across seven scenarios showed that scheduling order is a capacity decision in its own right. Their results, published on August 17, 2026, put the primary finding at 33 percentage points of utilization recovered on an 8-GPU training-heavy workload.
What did the benchmark actually measure?
The team ran seven scenarios on identical hardware, comparing a FIFO baseline against their constraint-aware allocator. The FIFO baseline reserves GPUs for real-time inference at peak demand and places everything else in arrival order. The allocator treats real-time demand as a curve, fills troughs with batch work, and places jobs by priority across the whole horizon.
The headline result: in a training-heavy workload on 8 GPUs, utilization jumped from 53.6% to 87.0%, and priority-weighted value rose 105.1%. Across all seven scenarios, value improved every time, ranging from 15.9% to 105.1% and averaging 52%.
The chart above shows the value gains across all seven scenarios. The training-heavy case dominates, but even the weakest improvement, the 64-GPU scale test, delivered 15.9% more priority-weighted output with identical utilization and identical throughput. Same dashboard readings, materially different cluster output.
Utilization improved in six of seven scenarios and tied in the seventh. The band moved from 44.9% to 85.4% under FIFO to 44.9% to 87.5% under the allocator. The scale test, 64 GPUs and 30 jobs, held at 44.9% utilization on both sides but still produced 15.9% more value because the allocator placed higher-priority work first.
The uniform-priority test matters for a skeptical reading. When every job was overridden to identical priority, removing any priority signal, the allocator still moved utilization from 76.8% to 87.5% and value up 23.1%. The gain is not purely an artifact of sorting by importance. Planning placements across the horizon contributes on its own.
Why does FIFO waste so much GPU time?
Two behaviors compound under FIFO, and both are structural.
The first is the reservation problem. Real-time inference cannot wait for capacity. A FIFO scheduler has no mechanism for releasing GPUs during a traffic trough and reclaiming them before the next peak, so the only way to guarantee availability is to reserve each application's maximum daily demand for the entire day. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours. The four idle GPUs are unavailable to any batch job. This is why the FIFO baseline sits near half the cluster in scenarios where reservation dominates: 51.6% in the mixed control and 53.6% in the training-heavy case.
The second is the ordering problem. Under real contention, which jobs fit depends on the order you place them, not just how much capacity exists. FIFO commits capacity to whichever job arrived first, without checking what else still needs to fit. A job that arrives later and needs a specific GPU block size may find nothing left that fits, so it goes unscheduled and its GPU-hours go unclaimed.
The two compound. The block reserved for peak real-time demand is off the table for every batch job, every hour. Whatever remains is handed out in arrival order. High-priority work waits behind whatever asked first.
An independent study on GPU scheduling in multi-tenant cloud environments, published in 2025, reported consistent findings: their scheduler maximized GPU utilization by 65 percent and decreased average job runtime by 40 percent over FIFO baselines on real-world workloads including NLP and computer vision. The pattern repeats across independent efforts: FIFO is the expensive default.
How does the allocator make better decisions?
The allocator formalizes the scheduling problem as a grid: every GPU across the whole time horizon, with a job name in each cell or nothing. Five constraints define a legal allocation: one job per GPU per timestep, jobs respect their demand ranges, batch jobs occupy contiguous power-of-two GPU blocks, real-time jobs have a swap cap between timesteps, and started jobs cannot be interrupted.
The objective function has two terms. Allocating a GPU to a batch job earns a reward equal to priority times a time-decay weight. Failing to meet real-time demand incurs a penalty proportional to the shortfall. The penalty weight is 5 to 10 times greater than the allocation weight, which means one unit of unmet real-time demand costs what 5 to 10 GPU-timesteps of equal-priority batch work costs. That asymmetry is the service-level policy expressed as one number. Latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing with the scheduler for the same GPUs.
The allocator sees every queued job before placing any of them. It can hold the free pool in shapes the remaining work can actually occupy. A batch job needing a contiguous block of a given size still has room when its turn comes. Priority decides who gets first claim on that room.
The heuristic runs in 1 to 2 milliseconds on the five contended scenarios and 15 milliseconds at 64 GPUs and 30 jobs, fast enough to re-run on every incoming request. The system exposes two modes: fast mode returns the heuristic's grid, and full mode uses that grid as a starting point for the formal optimizer, suited to periodic review rather than per-request decisions.
What does this change for builders running shared GPU clusters?
If you operate a shared GPU cluster running mixed workloads, the implications are concrete:
- Capacity planning shifts. A cluster that runs at 55% utilization under FIFO may already have 85% effective capacity. Before buying more GPUs, check whether your scheduler is the bottleneck. The 33-point utilization gap in the training-heavy scenario is the difference between needing 8 GPUs and needing roughly 13 for the same output.
- Reservation policy is the single biggest lever. Static reservations for real-time inference are the dominant waste source in the FIFO baseline. Treating real-time demand as a curve and filling troughs with batch work recovers most of the gain. If your scheduler does not support elastic real-time allocation, you are paying for peak capacity around the clock.
- Priority-weighted value is the metric that matters, not raw utilization. The scale test produced identical utilization and identical throughput on both schedulers, but 15.9% more value. If your dashboard shows GPU utilization and job completion counts, you cannot see the difference. You need a metric that weights output by priority. As we have argued in our coverage of enterprise GPU utilization, most clusters run far below their theoretical ceiling, and the cause is often the scheduler, not the workload.
- Quantization is a schedulable job. Quantizing a single large model can consume hours of GPU time. The Dharma-AI system builds forecasts for quantization jobs from calibration tiers by parameter count, with distinct handling per algorithm including bitsandbytes, AWQ, and GPTQ. If your scheduler treats quantization as free background work, it is stealing capacity from everything else.
For a team paying market rates for GPU capacity, recovering 33 points of utilization on an 8-GPU cluster is equivalent to adding roughly 2.6 GPUs at zero cost. At 64 GPUs, even the 15.9% value gain means completing higher-priority work without expanding the pool.
Should you build this yourself or wait for managed solutions?
The answer depends on your cluster size and workload mix. If you run a small cluster with homogeneous workloads, FIFO with backfilling is fine. The Dharma-AI results show that contention is where ordering cost becomes visible, and a slack cluster has nothing to recover.
If you run a shared cluster with mixed training, inference, and quantization work, the case for a constraint-aware allocator is strong. The Dharma-AI blog describes the formal model in enough detail that a strong engineering team could prototype it. The constraints are standard: contiguous blocks, no preemption, swap caps. The objective function is two terms. The hard part is the demand forecasts. A scheduler is only as good as its predictions for how many GPU-hours each job needs and how much real-time traffic is coming. The Dharma-AI system uses specialized estimators per workload type, because a single generic estimator fails. Training varies by strategy and technique. Real-time inference is forecast as a continuously recalibrated weekly demand profile. Quantization gets its own forecast from calibration tiers. Get those predictions wrong and the optimizer produces a legally valid but practically poor schedule.
Cohere has written about serving fairness with a priority and deadline hierarchy for inference workloads, which solves a related but narrower problem: ordering within a single inference server. The Dharma-AI work tackles the harder problem of allocating across workload types that have incompatible shapes.
The scheduler is the moat
Buying more GPUs is the expensive answer. Scheduling the ones you have is the cheap one. The Dharma-AI benchmark shows that the gap between a naive and a constraint-aware scheduler can mean the difference between a cluster running at half capacity and one running near its ceiling. As GPU costs climb and mixed LLM workloads become the norm, the scheduler determines whether your GPU budget earns or burns.
Sources
- Hugging Face Blog: Same Cluster, 33 Points More Utilization
- IJCSITR: Cost-Efficient and Scalable GPU Scheduling Strategies in Multi-Tenant Cloud Environments for AI Workloads
- Cohere: LLM Serving Fairness: No More Noisy Neighbors
