Changelog

Main-content extraction now measures the page’s explicit semantic scope

When a page declares one, extraction now selects the semantic scope — the <main> element, the [role=main] landmark, or the <article> element — instead of a whole-document heuristic. Every record now also seals what it measured: which element was selected, why, and how much of the document it covered.

When no semantic scope exists, the fallback heuristic excludes landmark regions (navigation, banners, footers, aside content) rather than sweeping the whole document.

robots.txt 404/410 is a fact, not an abstention

A robots.txt that responds 404 or 410 is now recorded as NO_POLICY_FILE, so the audit completes instead of abstaining: no policy file is a state of the site. A 401 or 403 still abstains as “policy unreadable” — the file exists but refused our fetch, so its rules remain unknown to us.

Quota enforcement single-sourced to plan entitlements

Scan quota enforcement now reads from one source — the plan entitlements — instead of parallel checks.