Remove HTML Boilerplate in Python Without Dropping Useful Content

Remove HTML boilerplate only after defining which source content must survive. A parser that deletes navigation but also drops the required code or table is not an acceptable conversion for that task. Use owned fixtures and content assertions before selecting a cleaning rule.

Start with an explicit main-content boundary

from bs4 import BeautifulSoup

html = "<nav>Navigation only</nav><main><h2>Fixture</h2><p>Required text.</p><pre>print(1)</pre></main>"
main = BeautifulSoup(html, "html.parser").select_one("main")
if main is None:
    raise ValueError("No expected main-content boundary")
text = main.get_text("\n", strip=True)
assert "Required text." in text
assert "print(1)" in text
assert "Navigation only" not in text

This is an owned HTML fixture with a known main element. Real pages need a verified content boundary; hard-coding main for every site is not an extraction algorithm.

Compare a general extractor with the fixture requirement

The installed BeautifulSoup and trafilatura test preserved the tested main text and code while excluding fixture navigation. That observation covers the specific fixture only. Add a table, nested lists, repeated interface labels and missing-content cases to the application’s own set.

Use a parser configuration that preserves the structures the downstream task requires. Keep the source snapshot and cleaned result so a regression can be located. A plain-text output is not suitable when the application depends on table relationships or code indentation unless those structures are separately preserved.

Reject a cleaner that only makes the string smaller

• Assert important paragraphs, table cells and code tokens.
• Report content loss alongside remaining boilerplate.
• Keep unavailable and empty sources out of accepted output.
• Use a permitted rendering route only when the required runtime content is missing.

Reference documentation: www.crummy.com — /software/BeautifulSoup/bs4/doc; trafilatura.readthedocs.io — /en/latest.

Related implementation and validation guides

• Clean Web Scraping Data Python: Reduce HTML Noise