Why recorded protocols beat a universal score
A network failure, an intentional training opt-out, a browser-only content deficit, and invalid JSON-LD are not interchangeable. The result keeps these observations separate, preserves unknowns, and derives a verdict only from required production protocols.
Protocol 1
Direct HTTP acquisition
Records: HTTP status, final URL, bounded redirects, declared content type, byte length, selected response headers, and artifact hash.
Does not prove: That a provider used the same network path or accepted the page downstream.
Protocol 2
Robots policy acquisition
Records: The public /robots.txt response as a separate, hashed policy artifact.
Does not prove: That an absent policy file caused a provider to crawl the site.
Protocol 3
Actor-specific robots decision
Records: The matching group, longest applicable allow/disallow rule, actor purpose, and policy decision for the audited path.
Does not prove: Actual access, indexing, training, or compliance by the provider.
Protocol 4
Raw and rendered representation
Records: Distinct raw HTTP, rendered browser, and no-JavaScript browser artifacts with mechanical hashes and byte counts.
Does not prove: Semantic equivalence or a universal crawler representation.
Protocol 5
Main-content extraction
Records: Deterministic extraction outcomes, word counts, hashes, and material raw-to-rendered deficits.
Does not prove: Content quality, truth, usefulness, or universal language readability.
Protocol 6
Structured-data consistency
Records: JSON-LD parse diagnostics, schema types, raw availability, and bounded consistency for selected material string properties.
Does not prove: Eligibility, correctness of every property, rich-result display, or downstream use.
Protocol 7
Machine-readable dates
Records: Strict ISO dates, supported meta fields, JSON-LD dates, time elements, and HTTP Last-Modified evidence.
Does not prove: That undated content is stale or that a declared date is factually true.
Protocol 8
llms.txt acquisition
Records: The public /llms.txt response as a separate hashed artifact: presence, bounded byte length, media type, and the required title heading.
Does not prove: That any provider fetched, parsed, or used the file.
Protocol 9
AI-usage header policy
Records: Site-declared x-robots-tag AI tokens and TDM reservation headers read from the captured raw HTTP response for each audited actor.
Does not prove: That a provider honored the declared signals or that the headers constrain downstream use.