· Blog
On September 15, 2026, Cloudflare flips a switch. For a defined set of Cloudflare-managed properties, AI crawlers classified as Training or Agent will be blocked by default on pages that display ads — while Search-classified crawlers stay allowed. Customers can opt out, and sites with existing preferences will not be silently flipped. It is a meaningful but a narrower change than many headlines suggest.
That is not really the point, though. The point is what the moment exposes: most site owners cannot answer the question it forces — which AI systems can actually access, parse, and extract my site's content — right now, per crawler, and how do I know?
- Can GPTBot, ClaudeBot, or PerplexityBot fetch your pages at all?
- Can they parse them? A crawler that does not execute JavaScript may be reading a nearly empty page even if your site looks perfect in a browser.
- Is your robots.txt doing what you think it is — or only what you assume it says?
If your honest answer is “I assume it's fine,” keep reading.
Why site owners were blindsided
For twenty years, website owners dealt with one crawler regime: search engines. Googlebot was the whole conversation, robots.txt was a solved problem. That world is gone: AI systems now crawl through a roster of per-purpose actors — GPTBot, ClaudeBot, PerplexityBot, and others — with their own user agents and policies, and Cloudflare's per-purpose defaults are one visible piece of that shift.
How much blocking already exists? The honest, sourced picture is specific:
- 60.0% of reputable news sites disallow at least one AI crawler (Steinacker, Gosain and Dao, ACM WWW '26, using May 2025 data). That figure is often quoted without its scope — it is a finding about news sites, not the web as a whole.
- On the general web, only 9.33% of Tranco top-1M domains block a major AI crawler (Zenodo census, June 2026). Even within news and media, the census finds 80.6%.
Read those two numbers together and you get the real problem: most of the web has never examined its AI-access posture. The blocking that exists is concentrated where organizations have paid attention — everyone else is running an unexamined default.
The same peer-reviewed work found something worse than ignorance: sites' active user-agent blocking diverges from their declared robots.txt, and robots.txt-based defenses are widely violated and easily circumvented. In other words, reading your own robots.txt does not tell you what actually happens. And “just asking the AI” doesn't either — the AI vendors' own documentation says answer inclusion is editorial and non-deterministic, and 2026 research shows identical queries returning different answers and citations across runs.
The uncomfortable truth: you can't fix what you can't see
Here is the part that should be uncomfortable no matter how good your site is.
A website owner's instinct is to ask: “Am I visible in ChatGPT / Claude / Perplexity?” That question is not answerable with evidence — not hard,unanswerable — because model behavior varies run to run, and nobody, including the tool vendors who promise rankings and citations, can make it reliably measurable. Investors have reportedly poured $227 million into AI visibility-tracking services, and one veteran SEO voice put the skeptical case plainly: if a service does not admit that success in AI visibility is mostly good fundamental SEO, it is selling you something else.
The answerable question sits upstream of all of that: can AI systems access, parse, and extract your content? That is an execution question about your property — your server, your robots policy, your rendered and raw HTML, your structured data. It is answerable with evidence, deterministically, today. And if the answer is no, no amount of “GEO optimization” fixes it, because the content never reaches the model in the first place.
What a real audit finds — in about 30 seconds
A real scan of usa.gov, run on 2026-09-12, returned FAIL with one named blocker: RAW_MAIN_CONTENT_DEFICIT. The site's main content is built by JavaScript. The raw HTML a non-JS AI crawler receives exposes 36 words of main content; the browser-rendered page exposes 412. A crawler that does not run a browser — which is how most AI extraction crawlers fetch — reads a page that is almost empty, and there is nothing visibly wrong from a human visitor's point of view.
That is what “can it parse my content?” looks like when someone actually checks: one verdict (PASS / FAIL / INCOMPLETE), one named protocol observation, and a downloadable evidence bundle showing exactly what was fetched and what was extracted.
Your site may pass, or fail something else entirely. Either way, the finding is on the table in about thirty seconds — and, importantly, it is derived from observed behavior of your site, not from a heuristic score.
What our audit does — and explicitly does not do
We built Generative Metrics on a simple axiom: the evidence trail is the product. Here is what the audit actually is.
What it does:
- Fetches and analyzes access per documented AI actor (GPTBot, ClaudeBot, PerplexityBot, and others), classifying what each actor is permitted to do against your declared policy — robots directives, AI-usage headers, and UA-gated rules.
- Runs both a rendered and a no-JavaScript extraction of your content, so you find out whether a non-browser fetcher sees your article — or 36 words of it.
- Checks your llms.txt (labeled for exactly what it is: a convention; some vendors honor it, some do not), your robots policy, and your structured data.
- Issues a verdict that is an execution state — PASS, FAIL, or INCOMPLETE — never a score, never a rank. Every verdict is backed by a sealed, downloadable JSON evidence bundle: real, logged observations, no LLM in the audit path, zero fabrication. If a check could not be completed, the audit says INCOMPLETE rather than claiming anything.
What it does NOT do. Read this list twice; we mean every word of it.
- It does not tell you whether you appear in any AI system's answers.
- It does not tell you whether you are indexed, ranked, or cited.
- It does not predict or measure what any AI model does or will do.
- It does not improve your traffic by itself. It is a diagnostic. If it says FAIL, the fix is yours to make — in your code, your CMS, or your policy — and a rescan verifies whether the fix actually landed.
We are not selling a leaderboard or a promised seat at the top of an AI answer. We are telling you whether the door is unlocked, per crawler — and giving you the evidence to hold. What happens after the door is open is your business, and anyone who promises you a specific outcome there is guessing.
Know, don't assume
On September 15, the default changes for some crawl behavior at the platform level. The sounder default for every site owner is the one this post opened with: know, don't assume.
Run a free audit at generativemetrics.com — 5 free scans per month, no credit card, and the PDF report and evidence bundle are downloadable on every tier, including free. You get the facts, per actor, with receipts.
Whether the result is PASS, FAIL, or INCOMPLETE, you will be answering the September 15 question with something better than an assumption: evidence.
Run your first audit now — it takes about 30 seconds.