# Prometheus relabel_configs vs metric_relabel_configs: where to filter data

Diagnose prometheus relabel_configs vs metric_relabel_configs: where to filter data. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/prometheus-target-vs-metric-relabeling/
Language: en

Relabel_configs modifies or filters targets before scraping; metric_relabel_configs processes samples before ingestion. The stage determines which costs disappear. Dropping a metric after receiving the response does not cancel the scrape request or automatically reduce exporter work.

## Draw the data path

Record discovery, target relabeling, scraping, metric relabeling and local storage. Remote write has a separate write_relabel_configs stage before transmission. Similar configuration formats apply to different objects at different times. Preserve the job and effective labels. Do not move a rule between stages solely because YAML keys look similar: discovery labels may exist before scraping but not during sample processing, while __name__ belongs to a metric series.


[Official reference 1](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).

## Select the intended effect

Filter a target when it should not be polled at all. Use metric relabeling when the exporter is needed but some samples should not be stored. Consider write relabeling when local series remain useful but only a subset should reach remote storage. That does not delete accepted local history. Metric relabeling does not apply to automatically generated series such as up. Thus up may remain even when selected exporter samples are no longer ingested.


[Official reference 2](https://prometheus.io/docs/concepts/jobs_instances/).

## Examine an illustrative drop rule

Suppose a job needs host metrics but excludes a debug_* family. A metric_relabel_configs rule matching __name__ can discard those samples before ingestion. The exporter still receives the scrape and generates its response. Compare scrape_samples_scraped with scrape_samples_post_metric_relabeling, then inspect required series. This demonstrates the filtering stage rather than promising lower network usage or exporter CPU without changes at the source.

| Stage | What may stop |
| --- | --- |
| Target relabeling | Polling the excluded target |
| Metric relabeling | Ingesting selected samples into TSDB |
| Write relabeling | Sending selected samples remotely |
| Exporter configuration | Generating unwanted source data |

## Preserve label uniqueness

Removing a label does not aggregate series. If formerly distinct series become identical after labeldrop, the result can be conflict rather than a valid sum. Investigate the actual labelset and job scope. Validate configuration with the available promtool and test a bounded group. Avoid broad regular expressions that remove alerting or instance-identity labels. A rule change also does not instantly erase historical samples; separate new ingestion policy from existing retention.

## Verify the chosen boundary

After reload, compare discovery, up, scraped versus post-relabel sample counts and required series. For write relabeling, inspect remote receipt independently of local storage. Preserve rule, stage, intended effect and observed evidence. If the goal was lower exporter load but only local ingestion changed, the correction does not meet that goal. This record explains exactly what filtering changed and which pipeline stages continue operating as before.

Related operational reading: [gpu monitoring scorecard](/journal/gpu-monitoring-scorecard/); [inference capacity planning](/journal/inference-capacity-planning/).