· Blog
September 15 has passed, and the Cloudflare default change described in this post took effect on that date. This post was published on September 15, 2026; the sentences that described the change as upcoming now read in the past tense.
Title shortened and citation-tracking framing replaced with the deterministic upstream acquisition question.
On September 15, 2026, Cloudflare flipped a switch. For a defined set of Cloudflare-managed properties, AI crawlers classified as Training or Agent are now blocked by default on pages that display ads — while Search-classified crawlers stay allowed. Customers can opt out, and sites with existing preferences are not silently flipped. It is a meaningful but a narrower change than many headlines suggest.
That is not really the point, though. The point is what the moment exposes: most site owners cannot answer the question it forces — which documented AI actor is allowed to fetch my site's content — right now, per crawler, and how do I know?
- Can GPTBot, ClaudeBot, or PerplexityBot fetch your pages at all?
- Can they parse them? A crawler that does not execute JavaScript may be reading a nearly empty page even if your site looks perfect in a browser.
- Is your robots.txt doing what you think it is — or only what you assume it says?
If your honest answer is “I assume it's fine,” keep reading.
Why site owners were blindsided
For twenty years, website owners dealt with one crawler regime: Googlebot was the whole conversation, robots.txt was a solved problem. That world is gone: AI systems now crawl through a roster of per-purpose actors — GPTBot, ClaudeBot, PerplexityBot, and others — with their own user agents and policies, and Cloudflare's per-purpose defaults are one visible piece of that shift.
How much blocking already exists? The honest, sourced picture is specific:
- 60.0% of reputable sites disallow at least one AI crawler (Steinacker, Gosain and Dao, ACM WWW '26). The paper's term is “reputable sites” — primarily news publishers. That figure is often quoted without its scope — it is a finding about those sites, not the web as a whole.
- On the general web, only 9.33% of Tranco top-1M domains block a major AI crawler (Zenodo census, June 2026). Even within news and media, the census finds 80.6%.
Read those two numbers together and you get the real problem: most of the web has never examined its AI-access posture. The blocking that exists is concentrated where organizations have paid attention — everyone else is running an unexamined default.
The same peer-reviewed work found something worse than ignorance: sites' active user-agent blocking diverges from their declared robots.txt, and robots.txt-based defenses are widely violated and easily circumvented. In other words, reading your own robots.txt does not tell you what actually happens. And “just asking the AI” doesn't either — the AI vendors' own documentation says answer inclusion is editorial and non-deterministic, and 2026 research shows identical queries returning different answers and citations across runs.
The uncomfortable truth: you can't fix what you can't see
Here is the part that should be uncomfortable no matter how good your site is.
The question owners ask first — 'am I visible in ChatGPT / Claude / Perplexity?' — is not reliably answerable with evidence: model behavior varies run to run. The question that is answerable, deterministically, today, sits upstream: what is each documented AI actor allowed to fetch, and what does a no-JavaScript fetch actually receive? If the answer is 'not your content', no amount of GEO optimization fixes it, because the content never reaches the model.
What a real audit finds — in about a minute
Correction, 2026-09-29: an earlier version of this section illustrated the audit with a usa.gov scan whose raw-extraction numbers our own raw-extraction arm under-counted. Independent re-measurement found 443 raw words and no deficit. We removed the figures; the corrected numbers were published from the fixed pipeline on the sample-report page, and the wrong scan is recorded in the public errata register.
A real audit answers “can it parse my content?” with recorded evidence when someone actually checks: one verdict (PASS / FAIL / INCOMPLETE), one named protocol observation, and a downloadable evidence bundle showing exactly what was fetched and what was extracted.
Your site may pass, or fail something else entirely. Either way, the finding is on the table in about a minute — and, importantly, it is derived from observed behavior of your site, not from a heuristic score.
What our audit does — and explicitly does not do
We built Generative Metrics on a simple axiom: the evidence trail is the product. Here is what the audit actually is.
What it does:
- Fetches and analyzes access per documented AI actor (GPTBot, ClaudeBot, PerplexityBot, and others), classifying what each actor is permitted to do against your declared policy — robots directives, AI-usage headers, and UA-gated rules.
- Runs both a rendered and a no-JavaScript extraction of your content, so you find out exactly what a non-browser fetcher sees of your article.
- Checks your llms.txt (labeled for exactly what it is: a convention; some vendors honor it, some do not), your robots policy, and your structured data.
- Issues a verdict that is an execution state — PASS, FAIL, INCOMPLETE, or AI Access Blocked — never a score, never a rank. Every verdict is backed by a sealed, downloadable JSON evidence bundle: real, logged observations, no LLM in the audit path, zero fabrication. If a check could not be completed, the audit says INCOMPLETE rather than claiming anything.
What it does NOT do. Read this list twice; we mean every word of it.
- It does not tell you whether you appear in any AI system's answers.
- It does not tell you whether you are indexed, ranked, or cited.
- It does not predict or measure what any AI model does or will do.
- It does not improve your traffic by itself. It is a diagnostic. If it says FAIL, the fix is yours to make — in your code, your CMS, or your policy — and a rescan verifies whether the fix actually landed.
We are not selling a leaderboard or a promised seat at the top of an AI answer. We are telling you whether your declared policy and access controls treat each documented actor as allowed — and giving you the evidence to hold. What happens after the door is open is your business, and anyone who promises you a specific outcome there is guessing.
Know, don't assume
On September 15, the default changed for some crawl behavior at the platform level. The sounder default for every site owner is the one this post opened with: know, don't assume.
Run a free audit at generativemetrics.com — 5 single-page audits per month plus one full site audit (up to 25 URLs), once, no credit card, and the PDF report and evidence bundle are downloadable on every tier, including free. You get the facts, per actor, with receipts.
Whether the result is PASS, FAIL, INCOMPLETE, or AI Access Blocked, you will be answering the September 15 question with something better than an assumption: evidence.
Run your first audit now — it takes about a minute.