COMPUTE AND INFRASTRUCTURE

Custom GRES as an escape hatch from automatic hardware discovery

Event source: AWS PCS adds custom GRES, topology and MIG scheduling ↗

In this analysis
THE RETROSPECTIVE TAKE

Custom GRES extends the scheduler representation; it changes resource descriptions rather than the physical capacity of a node.

Custom GRES in AWS PCS provides a way to correct the scheduler’s picture of a machine when automatic discovery is insufficient. The September 11, 2026 update matters because an automated inventory and an accurate resource contract are not necessarily the same thing.

Automation needs a defined boundary

The original PCS announcement on August 28, 2024 centered on managed Slurm clusters. Service management reduces repetitive infrastructure work, but users still need a representation of resources that matches the workload. A managed scheduler cannot infer every application-specific constraint from a device count.

That leaves a legitimate role for administrator declarations. An override is not automatically evidence that discovery failed. It can deliberately select a more useful scheduling abstraction for the particular host and job mix.

New hardware does not erase that need

The March 18, 2025 RTX PRO announcement expanded professional GPU options before the later PCS change. Hardware evolution supplies capabilities, while a scheduler configuration determines which capabilities become allocatable names and constraints.

The causal connection is representational. As devices and host layouts become more varied, a single default view becomes less likely to express every intended allocation policy. This does not mean newer hardware is inherently harder to operate or that AWS reported such a failure.

The explicit override path

The September update introduced custom generic resources and hardware topology support with Slurm 26.05. It also describes automatic affinity and interconnect detection, with topology overrides such as treating L3 cache domains as sockets.

Automatic discovery and explicit configuration therefore coexist. The meaningful comparison is between their resulting resource descriptions. An override that silently changes the meaning of a familiar resource name could make accounting less clear even if jobs continue to launch.

An illustrative change record

Suppose an administrator replaces the default topology abstraction for a specialized CPU preparation stage. The review artifact can separate intention from observation:

RecordContent
Previous viewAutomatically discovered resource topology
Intended changeThe scheduling boundary the workload requires
New declarationThe custom resource or topology representation
Validation evidenceAllocation output and application behavior

This is an example of a configuration record, not a claim about an internal AWS process. Its value is explaining why an override exists. A declaration without that explanation can survive long after the workload that motivated it disappears.

The operational implication

Custom GRES is an escape hatch from a default abstraction, not permission to invent physical capacity. The scheduler’s resource name, the actual hardware and the application’s interpretation still need to agree. A declarative label cannot create missing memory or connectivity.

Corporate GPU inventory supplies the physical accounting context. NUMA and GPU servers explains why topology semantics matter. The announcement adds a supported place to express those distinctions instead of forcing every host into an apparently uniform resource list.

← All retrospectives