# OpenTelemetry Collector drops data during backend outages: queues and limits

Diagnose openTelemetry Collector drops data during backend outages: queues and limits. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/otel-collector-backpressure/
Language: en

A Collector can lose telemetry through queue overflow, expired retries or restart without a persisted buffer. Locate the failed transition first. Sending queues and memory_limiter protect different boundaries and cannot guarantee endless delivery during a backend outage.

## Locate the loss boundary

Draw the actual SDK → agent → gateway → backend path. Every transition can have its own queues, timeouts and retries. Preserve receiver acceptance or refusal, exporter failures and queue size/capacity using metrics appropriate to the version. A running Collector does not prove the intended pipeline is active. Confirm components are enabled under service pipelines and data reaches the selected receiver. Successful local debug output does not establish remote backend receipt.


[Official reference 1](https://opentelemetry.io/docs/collector/resiliency/).

## Separate queue and memory limits

A sending queue buffers data before export. Overflow and exhausted retry duration create loss conditions. Memory_limiter controls Collector pressure and may refuse intake that upstream must handle. An oversized queue can itself push the process toward its container memory limit. Queue capacity is therefore coupled to RAM. Record processor ordering, retries and workload limits before selecting a larger buffer.


[Official reference 2](https://opentelemetry.io/docs/collector/troubleshooting/).

## Run a bounded outage test

Send synthetic numbered events in a test environment and temporarily stop the test backend. Observe queue growth and upstream behavior. After recovery, compare generated and received identifiers without claiming exactly-once delivery. Test Collector restart separately. An in-memory queue cannot retain its contents through a crash. Persistent sending queues using a storage extension change that behavior, but remain constrained by capacity, retry policy and storage correctness.

| Test | Observable evidence |
| --- | --- |
| Backend unavailable | Export failures and growing queue |
| Queue full | Refusal or loss of new data |
| Collector restarted | Behavior of retained backlog |
| Backend restored | Catch-up delivery and gaps |

## Choose the tradeoff explicitly

Define the outage duration and volume to absorb, then budget resources from measured input. Critical transitions may justify persistent queues or a messaging layer with additional operational work. That is not a universal requirement for every local agent. Complexity introduces failure points too. Record responsibility for exporter retries and upstream retries so backpressure is not silently discarded by an intermediate component.

## Verify return to normal

Recovery includes decreasing backlog, no continuing refusal or drops, and an identified interval of possible loss. Absence of new errors does not prove delivery of history. Preserve topology, versions, queue settings and the synthetic outage result. Repeat after changing memory limits or telemetry frequency. This keeps capacity tied to actual volume and makes the Collector observable as a component whose failures cannot be hidden by the GPU service's own apparent health. Review both application and telemetry paths when interpreting missing incident evidence.

Related operational reading: [gpu monitoring scorecard](/journal/gpu-monitoring-scorecard/); [network and uptime](/journal/network-and-uptime/).