# A Grace Blackwell scheduler that idles two nodes per rack gets 27% more usable GPUs

> SkyPilot argues NVL72 racks need 16-node slices with two hot spares and rack-local placement. Full-rack allocation and naive placement leave GPUs idle or stuck on InfiniBand.

- Published: 2026-10-29
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/grace-blackwell-scheduler-two-spares-27-percent-capacity
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy

## The short version

- At the roughly 2% node unavailability SkyPilot reports on production NVL72 clusters, NVIDIA's model shows 16-node slices yield 2,037 usable nodes versus 1,602 for full 18-node racks, a 27% gain.
- A naive scheduler that splits multi-node jobs across racks pushes communication onto InfiniBand, and Hugging Face measured a 43% training throughput drop when tensor parallelism moved off NVLink onto InfiniBand.
- Rack-aware, slice-based placement raised GR00T N1.7 pre-training throughput by 14.4% on 512 GB300 GPUs, cutting iteration time from 1.59 s to 1.39 s.
- Commenters on Hacker News note that topology-aware gang scheduling is long-established. A commenter speaking for SkyPilot said its scheduler is built on Kueue, whose elastic-workload mode does not support topology-aware scheduling.

On NVL72 racks, usable GPUs get stranded in three ways. Allocating whole 18-node racks means one node failure makes the entire rack unschedulable until it is repaired. Naive schedulers split multi-node jobs across racks, which forces NVLink-bound traffic onto InfiniBand, and jobs finishing at different times scatter free nodes across racks. SkyPilot's case for a Grace Blackwell scheduler is to place 16-node slices on a single rack and hold two spares per rack. NVIDIA's capacity model shows this gives 27% more usable capacity than full-rack allocation (2,037 vs. 1,602 nodes) at a 2% node unavailability rate.

The argument comes from a SkyPilot blog post published October 29, 2026, which spends most of its length on why rack locality matters. A shorter section names the scheduler's components: Topology-Aware Scheduler (TAS), GPU Manager, Frontier Trainer and Auto Defrag. The 27% figure combines an NVIDIA model of a 128-rack cluster with SkyPilot's measured unavailability rate, so it is not a measured before-and-after on one cluster. The measured result is a separate 14.4% throughput gain that NVIDIA's GR00T team reported for a rack-aware, slice-based policy like SkyPilot's.

## Why naive placement strands NVL72 GPUs on InfiniBand

An NVL72 rack connects 72 GPUs through 18 NVSwitch chips. Each GPU has 18 NVLink ports at 100 GB/s bidirectional, for 1.8 TB/s per GPU and 130 TB/s aggregate inside the rack. The fabric is non-blocking, so any GPU reaches any other in the rack with uniform latency. That advantage holds only inside one rack. Naive schedulers split multi-node jobs across racks, and workers on different racks communicate over InfiniBand at 400 Gb/s per direction.

The second failure mode is fragmentation. Jobs finish at different times, so free nodes end up scattered across racks. The cluster then has enough free GPUs for a rack-scale job but cannot place it. In SkyPilot's example, 16 nodes are free but split between two racks, and a 16-node hero run waits.

## Tensor and expert parallelism must stay on NVLink

Two parallelism strategies depend most on rack locality, because their communication sits on the critical path. For tensor parallelism, Megatron-LM splits each block's first GEMM column-wise and its second row-wise. That still leaves two all-reduces per transformer layer in the forward pass, and they cannot be fully overlapped with computation. Hugging Face measured a 43% drop in training throughput when tensor parallelism moved off NVLink onto InfiniBand.

For expert parallelism, each MoE layer needs two all-to-all steps in the forward pass: dispatch and combine. With 64-way expert parallelism on standard 8-GPU servers, that traffic moves onto InfiniBand, where unoptimized communication can consume up to 60% of training time. On NVL72, all 64 GPUs stay within one NVLink domain, which the post says effectively eliminates that bottleneck.

Inference has the same constraint. On 8-GPU servers, an expert-parallel group can span at most eight GPUs before it falls back to InfiniBand, where per-GPU bandwidth drops 18x. SkyPilot cites GB200 NVL72 serving DeepSeek-R1 at interactive speeds with 32-way expert parallelism, delivering up to 4.4x higher throughput per GPU than 8-GPU B200 servers capped at 8-way. Wider sharding also frees memory for KV cache: in one vLLM benchmark with Llama-3.1-70B on H100s, two-way tensor parallelism raised KV cache blocks 13.9x and throughput 3.9x.

## Why 16-node slices beat full 18-node racks

The reliability arithmetic drives the choice of slice size. Meta measured a mean time between interruptions of roughly 490 days per 4-GPU node on its gigawatt-scale supercluster. Making the 72-GPU rack the atomic unit increases the fault blast radius 18x, so a rack sees an interruption roughly every 490 / 18, or about 27 days. If jobs are allocated entire racks, a 128-rack cluster loses a rack for full-rack jobs roughly every 5 hours.

NVIDIA modeled usable capacity on a 128-rack, 2,304-node cluster for different slice sizes as node unavailability rises. SkyPilot says it measured around 2% unavailability on production NVL72 clusters. At that rate, 16-node slices yield 2,037 usable nodes against 1,602 for 18-node slices. They also beat 9-node slices by 4%: with 9-node slices each failure costs half a rack, while with 16-node slices the first two failures in a rack are fully absorbed.

Against 8-node slices, 16-node slices give each job twice the GPUs per NVLink domain. They give up less than 1% of usable capacity until node unavailability exceeds about 3.3%. Sixteen nodes is also 64 GPUs, a power of two, which matches how parallelism groups are usually sized.

The 27% advantage depends on the unavailability rate. Teams with a different failure profile should read the figure as one point on NVIDIA's curve, not a constant. The post's text states values only at 2% unavailability and the roughly 3.3% crossover against 8-node slices.

## How the Grace Blackwell scheduler places and repairs slices

SkyPilot's TAS splits a multi-rack job into 16-node slices; the size is configurable. It places each slice on its own rack, leaving two spare nodes. When GPU Manager detects a GPU failure, Frontier Trainer hot-swaps the faulty node for a spare in the same rack while the remaining nodes keep running. If the rack has no spares left, Frontier Trainer reschedules that slice onto a healthy rack. The post does not say how long a hot swap takes or how much training state is lost.

The measured evidence comes from NVIDIA's GR00T team, which pre-trained GR00T N1.7, a multimodal foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes. A naive scheduler spread the nodes over an arbitrary number of racks. A rack-aware, slice-based policy put the job on exactly 8 racks and raised throughput 14.4%, from 1.59 s to 1.39 s per iteration. SkyPilot extrapolates this to roughly $16.7K saved per training hour on a 10,000-GPU cluster, or about $400K per training day.

Commenters on Hacker News pushed back on two points. One said gang scheduling and rack-aware scheduling have been around since HTCondor and are used at most large GPU deployments, and another said HPC schedulers have always been topology-aware. A commenter whose team built a custom topology-aware scheduler said the post leaves out the most interesting parts of such a project; they said their own next iteration targets O(millions) of GPUs with millisecond scheduling cycles. Another commenter pointed to Kueue's topology-aware scheduling as prior art. A reply said the SkyPilot Platform built its scheduler on Kueue and had to work around Kueue's elastic workloads listing "No Topology-Aware Scheduling (TAS) support," which it said makes hot swapping impossible for large jobs on NVL72 racks.

> jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps
>
> - A commenter describing SkyPilot's scheduler, Hacker News

## Is an 11% spare reserve too generous?

Two idle nodes out of 18 is about 11% of each rack. One commenter called that a bit generous for such an expensive cluster. They suggested pooling spare capacity across units instead, the way a data center with four zones might keep 7 backup generators rather than eight or twelve. The sources do not test a pooled-spare design. SkyPilot's design keeps the spare in the same rack as the failed node, which keeps the replacement inside the rack's NVLink fabric.

NVIDIA's model compares only fixed slice sizes against each other. On those terms, 16-node slices beat both 18-node and 9-node slices at 2% unavailability. Whether a cross-rack spare pool could do better is not answered by the sources.

## Auto Defrag preempts to free whole racks

TAS bin-packs workloads within a rack when it schedules them. Shared clusters running pre-training, RL rollouts and batch inference still fragment over time as jobs finish at different times. SkyPilot's Auto Defrag starts when a job is blocked by topology rather than by total capacity. It looks for the least disruptive set of workloads to preempt so that an entire rack becomes free.

The available text of the post does not say how disruption is scored or what happens to preempted jobs, and it gives no fragmentation measurements. A Hacker News commenter speaking for SkyPilot said further posts on topology-aware scheduling, defragmentation and hot swapping are coming. Until then, anyone evaluating an NVL72 scheduler for a mixed cluster should treat preemption policy as the open question, because that is where rack-scale jobs and smaller tenants compete.

## Questions this raises

### Is the 27% capacity gain a measured result?

No. It combines NVIDIA's model of a 128-rack cluster with SkyPilot's measured 2% node unavailability rate. The measured result is a separate 14.4% throughput gain reported by NVIDIA's GR00T team for a rack-aware, slice-based policy.

### Why use 16-node slices instead of full 18-node NVL72 racks?

With full-rack allocation, one node failure makes the whole rack unschedulable until repair. A 16-node slice leaves two spare nodes per rack, so the first two failures in a rack are absorbed by hot-swapping. Sixteen nodes is also 64 GPUs, a power of two that matches typical parallelism group sizes.

### How does the scheduler relate to Kueue?

A Hacker News reply said the SkyPilot Platform built its scheduler on Kueue. It had to work around Kueue's elastic workloads lacking topology-aware scheduling support, which the reply said makes hot swapping impossible for large jobs on NVL72 racks.

## Key facts

- At 2% node unavailability, 16-node slices yield 2,037 usable nodes vs. 1,602 for full 18-node racks on NVIDIA's 128-rack, 2,304-node model.
- 16-node slices beat 9-node slices by 4% and trail 8-node slices by under 1% until unavailability exceeds about 3.3%.
- NVIDIA's GR00T team measured a 14.4% throughput gain (1.59 s to 1.39 s per iteration) on 512 GB300 GPUs with rack-aware slicing.
- With full-rack allocation, a 128-rack cluster loses a rack roughly every 5 hours, based on Meta's ~490-day per-node interruption interval.
- Hugging Face measured a 43% training throughput drop when tensor parallelism moved from NVLink to InfiniBand.

## Sources

1. [Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks](https://skypilot.ai/blog/grace-blackwell-needs-a-new-scheduler) - skypilot.ai

Discussion read (practitioners' reactions, not a source): https://news.ycombinator.com/item?id=50290912