# Web pages for RAG: remove navigation without losing source evidence

Clean webpage text for RAG should retain the article's actual evidence while excluding navigation and unrelated page furniture.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/webpage-clean-text-rag-trafilatura/
Language: en

Clean webpage text for RAG should retain the article's actual evidence while excluding navigation and unrelated page furniture. A shorter extraction is not automatically better: tables, captions and publication metadata may be necessary to interpret the answer and identify the correct version of a page.

## Distinguish the page from its main content

The [Trafilatura paper](https://aclanthology.org/2021.acl-demo.15/) addresses web text extraction, while the [primary repository](https://github.com/adbar/trafilatura) provides the implementation. HTML combines content with menus, promotional blocks, comments and repeated navigation. Indexing everything can make shared furniture dominate search. Removing too aggressively can erase the condition or caption that qualifies a technical claim. Define the intended content boundary before comparing extracted character counts.

## Choose extraction options for the evidence

The official [Python usage documentation](https://trafilatura.readthedocs.io/en/latest/usage-python.html) describes extraction and metadata options. Decide whether tables, links and other elements are required for the workload. Preserve the fetched source or an appropriate immutable capture alongside the extracted result. A later live page may differ from the version that produced the indexed text, so a bare URL is insufficient for reproducing an old answer.

## Test an article with competing page regions

Choose a documentation page containing a procedure, a sidebar for another product and a table of conditions. Ask a question requiring the procedure and its table qualifier. Inspect whether the extractor keeps both while excluding unrelated navigation. Then check a page with a similar layout but different content, because a rule tuned to one article can accidentally remove the main body elsewhere.

| Page element | Ingestion decision |
| --- | --- |
| Main procedure | Retain as answer-bearing text |
| Conditions table | Preserve relationships and caption |
| Shared navigation | Exclude from primary evidence |
| Date or revision | Retain as source metadata |

## Keep extraction and rendering failures separate

A fetched HTML response may not contain content that a browser obtains later through scripts. An empty or incomplete extraction can therefore reflect acquisition rather than text-cleaning quality. Inspect the input HTML before changing extraction rules. Do not claim that a text extractor automatically renders every application or bypasses access requirements. Record whether the available source actually contained the expected evidence.

## Validate the indexed result against the capture

Keep URL, retrieval time, source revision where available, extraction configuration and text hash. Resolve selected retrieval hits back to the captured page and verify their scope. Review duplicated templates and missing qualifications separately. This produces a readable corpus with traceable evidence, rather than a collection that merely looks clean while silently discarding the tables, exceptions or version information necessary for trustworthy technical answers.

Related reading: [A local RAG server: placing components around the GPU](/journal/rag-server-local-architecture/); [Batch text embeddings on your own GPU](/journal/text-embeddings-batch/).