Coordinate Triton model reloads, control modes and version policies

Validate Triton repository updates together with model-control mode and version policy. A new version directory does not guarantee immediate inference switching. NONE, POLL, and EXPLICIT react differently to changes and management requests. Preserve initial state, selected action, and readiness of the particular version. This is repository lifecycle management, separately from directing canary traffic or comparing answer quality.

Establish control mode

The model-management guide defines the three modes. NONE ignores repository changes while running, EXPLICIT requires management actions, and POLL discovers changes through its mechanism. Do not combine them into one assumed reload procedure. Record effective startup mode and permitted actions. A rejected load request can be expected for that mode rather than evidence of damaged model files. Successful directory copying likewise does not establish that the server noticed or began preparing a replacement instance.

Inspect version policy

The model-configuration reference distinguishes All, Latest, and Specific policies. Latest concerns numerical versions, not file modification time. A lower-numbered directory does not become the selected newest version simply because it was copied later. Preserve policy explicitly. Where a client omits version selection, verify selection rules rather than infer the version from the public model name. Model identity and repository version are separate elements of the request contract and need separate observations.

Exercise a controlled reload

For a test model, prepare two known versions and a policy permitting the intended one. Record initial state, the applicable load/unload or repository-update action, and subsequent readiness of each version. This is an experiment design, not a seamless-reload promise for every backend. Follow documented EXPLICIT update ordering rather than assuming file replacement triggers reload. Keep management status and a functional request selecting the version explicitly. Process readiness cannot establish readiness of the required model version.

Diagnose unsuccessful updates

If the new version is unavailable, preserve its diagnostic and verify whether the previous version remains available in that scenario. Do not generalize one mode’s behavior to all others. With POLL, account for observation of partially prepared repositories and follow an appropriate publication procedure. Use an isolated test repository rather than exposing production to unfinished copies. A policy mistake can resemble loading failure, so compare available versions against the declared selection rule before changing model files or runtime dependencies.

Preserve management evidence

Keep control mode, version policy, directory layout, management requests, readiness, and functional fixtures. Distinguish files updated, load accepted, version ready, and client actually using it. Each needs separate evidence. Restore policy along with artifacts during rollback, otherwise old files may remain inaccessible. The deliverable reproduces one version’s transitions without guessing. It does not promise automatic canaries, unchanged latency, or flawless hot swapping of every model; those need separate backend and traffic validation after the basic repository lifecycle is established.

Updating an LLM on a GPU: testing a new version with a small rollout; vLLM is running but not ready: health checks and the first request.