In this article
Restored connectivity and a successful model request are separate events; neither should be inferred solely from the other.
A GPU node network outage log should show the sequence of observed events: the last successful check, detection, restored connectivity and a successful service test. One duration labelled server down is insufficient to explain the event or improve operation.
The record should describe actual availability. Do not invent lost token sales when request demand and retry behaviour during the interruption are unknown.
Identify the observation point
A local process, a site-network check and an external request can observe different conditions. Record where each check ran, its target and whether it tested a connection, HTTP response or completed model request.
Prometheus distinguishes jobs and instances used for monitoring. A scrape target’s availability refers to that target, not automatically the entire user journey. Correlate the observations before declaring a complete host failure.
Record several timestamps
| Event | Example UTC time | Evidence |
|---|---|---|
| Last successful external check | 10:00 | Monitor record |
| First detected failure | 10:03 | Failed check |
| Site responds | 10:15 | Support reference |
| Connection restored | 10:30 | Successful connection |
| Model request completes | 10:38 | Control response |
The application became ready after connectivity returned. Without more detailed evidence, the interruption began somewhere between the last success and first detection, not necessarily at either timestamp.
Preserve time zones and uncertainty
Use UTC for the common event line and retain local zones where needed for support hours. Python zoneinfo documents named calendar zones. Avoid applying one manually guessed offset to every historical event.
Write observations before causes: an external request failed, a local process continued, or the site reported a network change. Add a confirmed cause later, preserving the original observation and the date of the new explanation. Label a hypothesis as a hypothesis.
Close the event with one practical action
Record the responsible person, readiness check and specific improvement. It might be an additional external test, corrected support contact or agreed recovery route. Put the change into operating documentation rather than leaving it as a promise in a conversation.
Do not multiply the interruption by peak GPU throughput and call that lost income. Technical recovery decisions need observed demand and service evidence. A realistic operating record is useful even when its limitations are explicit.
Continue with GPU Idle Reports: Separate Waiting, Outages and Reservations and Compare Remote Hands Costs for Your GPU Server.