· Blog
We record what servers return to bounded requests so people can see how their sites present to automated consumers. To see what a single bounded observation returns across many sites instead of one, we sampled 500 domains from the Tranco top-1M list and probed each one once on September 16, 2026, using the same probe engine that backs our free quick check.
264 of the 500 answered (52.8%) and 236 did not (47.2%): 163 requests hit the 3.2-second page bound, 40 failed DNS validation, and 33 failed before a response arrived. That split is the first thing this study reports, because every other number depends on it. The figures below are exactly what the results file recorded; counts are whole numbers and percentages are rounded to one decimal place.
The sample and the method
The sample source is the Tranco top-1M daily list, list ID N2PYW (data window 2026-08-17 to 2026-09-15, downloaded 2026-09-16). We took every 2000th rank — 2000, 4000, 6000, and so on to 1,000,000 — deduplicated to registrable domains, and kept the first 500. No other filter was applied.
Each domain received one HTTPS GET with one user agent, up to 4 redirects, a 3.2-second bound for the page, a 2.6-second bound for auxiliary files, and a 6-second total budget. The probe then requested /robots.txt and /llms.txt from the origin that served the page. There were no retries. The collector script and the full results file are kept outside the public site; this article reports the aggregates and the method.
A probe counts as usable in this study when it recorded a page HTTP response or a fetched-and-parsed robots.txt. In this run the 169 parsed robots.txt files all came from domains that also answered, so the usable count equals the answered count: 264.
Non-response was 47.2%
236 of 500 domains did not answer: 163 timeouts, 40 DNS failures, and 33 transport failures. Timeouts were spread across the list rather than concentrated in one part of it: 33, 32, 28, 29, and 41 timeouts per 100-domain rank bucket, from the highest-ranked bucket to the lowest.
A timeout is not evidence that a site is unavailable, that it blocks anyone, or that its access posture is bad. It means one request from one vantage point received no response inside a 3.2-second bound. We did not re-check those sites with a longer bound or another network path, so they simply reduce the denominator here. Any percentage computed on 500 probes includes 236 non-responses in it, which is why the split is stated before the results.
What the 264 answers contained
Of the 500 domains, 196 answered with a 2xx status, 57 with a 4xx, and 11 with a 5xx (0 ended on a redirect status; redirects were followed). A 4xx or 5xx response is still an answer — the server responded — but the request did not receive a successful page.
169 of 500 (33.8%) served a robots.txt the probe could fetch and parse. For the generic (*) group and the exact URL probed, the parsed decisions were: 38 explicitly allowed, 22 explicitly disallowed, and 109 with no matching rule. Another 75 returned other client-error responses (403 or 404; recorded as no policy file), 4 were indeterminate, and 252 produced no robots result at all. A decision here concerns the generic group only, at one moment, for one URL: it is not a statement about any named AI provider, and a robots rule is not evidence that any crawler actually fetched or was blocked.
55 of 500 (11%) served an llms.txt with a 2xx response. Of the rest, 181 returned client-error responses (403 or 404; recorded as not found), 11 returned another error status, 15 timed out, and 238 were not reached. Publishing the file is a choice a site makes; its presence is not evidence that any AI system fetches or uses it, and its absence is not proof that none does.
5 of 500 (1%) carried an X-Robots-Tag header on the response the probe received. The recorded values included noindex and nofollow directives, a snippet-length directive, and one all. A header on one response describes that response, not a site-wide policy.
What the raw HTML contained
262 of 500 (52.4%) returned HTML the probe could read. Of those 262 pages, 252 had a <title> (96.2%), 156 had a meta description (59.5%), and 155 had an <h1> (59.2%). Two answered pages returned non-HTML bodies (one XML, one with no declared content type), so they had no HTML signals to read.
After removing scripts, styles, comments, and markup, the text a non-executing fetcher would see had a median of 2,947 characters (first quartile 143, third quartile 6,025, minimum 0, maximum 32,973, over the 262 HTML pages). These are characters, not words, and they describe what arrived in the response body. A low count is not a quality judgment about a page; it is a delivery observation about the bytes a fetcher received without running a browser.
What one observation does not prove
This was one probe per site at one moment from a single vantage point. A different time, network path, or page-timeout budget would move sites between the answered and not-answered groups, and the non-response share is itself a result of those conditions rather than a property of the sites. The not-answered domains are not shown here to be unavailable, blocked, or badly configured — only not observed within the bound.
The robots.txt numbers cover the generic (*) group for the probed URL. Named AI crawlers can be addressed by different groups, and a parsed rule is not evidence of any crawler's actual behavior. Nothing in this study is an audit of any sampled site and no verdict about any of them follows from it. This is not evidence about any AI provider, and it does not measure indexing, retrieval, training, citation, ranking, or use. A signal that was not observed in one bounded check is not proof that it does not exist.
Finally, the sample is a systematic slice of one popularity list, not a census of the web and not representative of it. The percentages describe these 500 domains under this method on this date. They say nothing about sites outside the list, and the HTML-signal counts say nothing about content quality or about how any system will treat a page.
What the study does provide is a dated, reproducible method and the raw counts it produced — including the non-responses, which are part of the result rather than a footnote to it.
Run the same checks on your own site.