# Prometheus remote_write falls behind: WAL, queues and shards

Diagnose prometheus remote_write falls behind: WAL, queues and shards. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/prometheus-remote-write-wal-backlog/
Language: en

Local scraping can remain healthy while remote_write falls increasingly behind. Inspect the WAL reader, pending samples, shard queues and remote receipt separately. Raising max_shards without checking memory and backend capacity can worsen overload instead of restoring delivery.

## Separate local and remote freshness

Compare one series in local Prometheus and remote storage, accounting for query-layer behavior. Record the relevant destination because remote-write queues for different destinations are independent. Up=1 describes target scraping, not successful onward delivery of all telemetry. Preserve export errors, timestamps and network evidence. A stale remote dashboard can also arise in querying, so first establish actual ingestion delay.


[Official reference 1](https://prometheus.io/docs/practices/remote_write/).

## Follow the WAL and shards

Remote write reads the WAL and distributes samples into shard queues that send batches. A full shard queue can block WAL reading for other shards of that destination queue. Prometheus adjusts parallelism using incoming rate and sending performance. Capacity describes queued samples, while max_samples_per_send bounds a batch; neither is retention. Shards and queue sizes increase memory use alongside the series-ID-to-label cache.


[Official reference 2](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).

## Examine a short backend outage

In a test, remote receipt becomes unavailable while local scrapes continue and pending samples rise. After recovery, inspect decreasing lag rather than only the absence of new errors. Catch-up traffic competes with new samples. If sending does not exceed incoming volume, backlog cannot clear. The WAL is not an infinite archive: prolonged delay and compaction can lose unsent data. Check actual boundaries against the installed version and configuration.

| Observation | Stage being checked |
| --- | --- |
| Fresh local points | Scraping and local storage |
| Increasing pending samples | Delivery queue |
| Endpoint errors | Backend availability and responses |
| Decreasing lag | Delivery catching up with input |

## Choose a bounded adjustment

Inspect sender CPU and networking, then receiver latency and throttling. Review queue_config and backoff afterward. A larger batch can help one backend and harm another, requiring a controlled test. Excessive shards create additional requests and RAM overhead. Avoid universal sizes borrowed from examples. Write relabeling can reduce future unwanted remote traffic but does not make an existing backlog delivered.

## Verify completeness after recovery

Following the test outage, compare local and remote time ranges, pending samples and lag reduction. Record possible gaps separately from successful new sends. Preserve destination, queue settings, input rate and recovery chronology. A successful endpoint HTTP response cannot prove complete delivery of accumulated data. Repeat the test when backend limits change so queue configuration remains connected to measured behavior rather than defaults. This distinction keeps current availability separate from historical completeness when reviewing an incident.

Related operational reading: [gpu monitoring scorecard](/journal/gpu-monitoring-scorecard/); [inference capacity planning](/journal/inference-capacity-planning/).