Blog
Notes on AI crawler access, parse failures, and what an evidence-first audit can — and cannot — tell you about AI system readability.
October 25, 2026
AI was blocked from our own site. Only our own scanner caught it.
Our scheduled self-audit returned AI Access Blocked for generativemetrics.com: all four required AI actors received HTTP 403. Six sealed runs, published.
October 1, 2026
What we got wrong, and how to check our work
The usa.gov example we published on 2026-09-19 was wrong — our extractor under-counted the page. The wrong numbers verbatim, the corrected sealed gm-ai-consumer-compat/4.0.0 scan, and the signed public errata register.
September 16, 2026
We probed 500 popular domains for AI access signals. Nearly half did not answer one request.
One bounded observation of 500 domains sampled from the Tranco top-1M list (every 2000th rank, list N2PYW): 264 answered within the 3.2-second page bound, 169 served a parseable robots.txt, 55 served llms.txt, and 5 sent an X-Robots-Tag header — with the method, the non-response accounting, and the limits stated.
September 16, 2026
We reviewed how our own site presents to AI systems. Here is what we fixed.
We ran our own audit checks against generativemetrics.com, found six gaps, fixed them in two commits on September 16, 2026, and re-ran the checks — with an honest account of what that does and does not prove.
September 15, 2026
On September 15, Cloudflare changes the default. Will you know whether AI crawlers still see your site?
The blocking that exists is concentrated where organizations have paid attention. Everyone else is running an unexamined default — and most site owners cannot answer the question the moment forces.
New posts are added as we publish audits and findings.
Subscribe via RSS