In this article
Understand InferencePool and endpoint picking for LLM workers, including compatibility, stale metrics and the boundary between routing and model execution.
Ordinary load balancing can distribute requests among healthy endpoints without understanding how long their prompts are or which context they already cache. An inference-aware gateway adds information relevant to model serving. Kubernetes Gateway API Inference Extension supplies APIs and integration points for this task; it does not itself execute a language model.
State what ordinary balancing cannot see
List the load information currently available to the gateway: connections, request counts, readiness or backend-specific metrics. Long-running generation requests and short requests can have very different remaining work despite each counting as one connection.
The Inference Extension introduction explains its inference-oriented role. Define the actual missing decision before adding it. If a simple replica set already meets the workload objective, a more elaborate picker is a hypothesis to test rather than an automatic upgrade in useful capacity.
Understand InferencePool and the picker
The InferencePool documentation describes a group of serving endpoints and its relationship with endpoint picking. The resource declares serving membership and integration information; it is not a model checkpoint, GPU allocation or execution engine.
Draw the gateway, picker and model workers separately. Mark where ordinary HTTP handling occurs and where a decision uses inference state. An upstream model or provider router is another boundary again. Clear ownership prevents the mistaken assumption that installing a gateway component also installs a supported LLM runtime.
Match actual API and implementation versions
Use the API reference for the selected resource version and compare it with the gateway implementation and picker image. A resource accepted by Kubernetes does not prove that the intended controller interprets every field or exposes every optional scoring feature.
Record supported metrics, authentication paths and discovery behavior. Verify that endpoints advertise the correct model and precision rather than only responding successfully to a health probe. A healthy but ineligible worker should not receive requests for a model it cannot serve.
Test decisions with stale and missing state
Use a trace containing short chats, long prompts, popular prefixes and unique inputs. Compare the inference-aware policy with the existing balancer on the same worker pool and arrival schedule. Inspect predicted load against actual queueing and first-token latency.
Then interrupt metrics delivery, restart a worker and add capacity. Observe how quickly decisions reflect those changes and what fallback applies without telemetry. Cache affinity should not keep sending work to an overloaded or unavailable endpoint merely because it once held a useful prefix.
Measure a service outcome
Record useful completions, latency tails, worker imbalance and routing overhead. Keep the success rule fixed across policies. A more even request count is not necessarily better if requests require different amounts of computation or have different cache opportunities.
Document the qualified gateway, extension, picker and engine releases together. For a provider, explain this as internal worker routing, complementary to an upstream platform selecting a provider. Avoid presenting the extension as a competing token marketplace or as evidence that every GPU in a distributed network can execute one tightly coupled model.
Related reading
Prefix caching: when repeated context helps a GPU pool, Bound vLLM request queues and prefill token backlogs.