Generative MetricsAudit · Diagnose · Verify

Crawler actor registry

The actors the scanner probes, published as open data. The machine-readable document is at /registry.json; this page renders the same record.

This registry is what Generative Metrics probes with. Vendor documentation is linked for every actor. It describes our scanner, never any provider's behavior.

18 actors, generated from lib/evidence/actors.ts (newest recorded vendor-document review date: 2026-10-02). The machine-readable document is /registry.json.

Registry actors with provider, purpose, robots-policy state, probe identity, vendor documentation, and default-profile requirement
Actor	Provider	Purpose	Robots policy	Probe identity	Vendor docs	Default profile
Amazonbot amazon:amazonbot	Amazon	SEARCH Search or discovery crawling	APPLIES The vendor documents robots.txt directives for this actor.	Amazonbot Documented product token (no fixed vendor-published string)	Documentation	Not required
Claude-SearchBot anthropic:claude-searchbot	Anthropic	SEARCH Search or discovery crawling	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/clausearchbot) Vendor-published string	Documentation	Required
Claude-User anthropic:claude-user	Anthropic	USER_ACTION Fetches on a user’s behalf	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected]) Vendor-published string	Documentation	Not required
ClaudeBot anthropic:claudebot	Anthropic	TRAINING Model training collection	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) Vendor-published string	Documentation	Not required
Generative Metrics direct HTTP probe gm:direct-http-probe	Generative Metrics	UNSPECIFIED No purpose recorded	NOT_APPLICABLE The scanner’s own direct HTTP probe, which fetches as itself.	GenerativeMetrics/1.0 (+https://generativemetrics.com) Vendor-published string	—	Not required
Google-Agent google:google-agent	Google	USER_ACTION Fetches on a user’s behalf	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	Google-Agent Documented product token (no fixed vendor-published string)	Documentation	Not required
Google-Extended google:google-extended	Google	TRAINING_AND_GROUNDING Training and grounding collection	CONTROL_TOKEN A robots.txt control token with no crawler of its own; it is never probed.	Google-Extended Documented product token (no fixed vendor-published string)	Documentation	Not required
Googlebot google:googlebot	Google	SEARCH Search or discovery crawling	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Vendor-published string	Documentation	Required
Meta-ExternalAgent meta:meta-externalagent	Meta	TRAINING_AND_GROUNDING Training and grounding collection	APPLIES The vendor documents robots.txt directives for this actor.	meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) Vendor-published string	Documentation	Not required
Meta-ExternalFetcher meta:meta-externalfetcher	Meta	GROUNDING Grounding or product functions	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) Vendor-published string	Documentation	Not required
MistralAI-User mistralai:mistralai-user	Mistral AI	USER_ACTION Fetches on a user’s behalf	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) Vendor-published string	Documentation	Not required
ChatGPT-User openai:chatgpt-user	OpenAI	USER_ACTION Fetches on a user’s behalf	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot Vendor-published string	Documentation	Not required
GPTBot openai:gptbot	OpenAI	TRAINING Model training collection	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) Vendor-published string	Documentation	Not required
OAI-AdsBot openai:oai-adsbot	OpenAI	ADS_VALIDATION Validates pages submitted as ads	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot Vendor-published string	Documentation	Not required
OAI-SearchBot openai:oai-searchbot	OpenAI	SEARCH Search or discovery crawling	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot) Vendor-published string	Documentation	Required
Perplexity-User perplexity:perplexity-user	Perplexity	USER_ACTION Fetches on a user’s behalf	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Vendor-published string	Documentation	Not required
PerplexityBot perplexity:perplexitybot	Perplexity	SEARCH Search or discovery crawling	APPLIES The vendor documents robots.txt directives for this actor.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Vendor-published string	Documentation	Required
xAI-SearchBot xai:xai-searchbot	xAI	SEARCH Search or discovery crawling	CONTEXT_DEPENDENT The vendor documents no robots.txt control for this actor; the directive is recorded as an observation.	xAI-SearchBot Documented product token (no fixed vendor-published string)	Documentation	Not required

What the recorded states mean

Purpose and robots-policy values are the registry’s recorded states, published as declared. “Required” under Default profile means the default audit profile (public-ai-discovery) names this actor as required; “Not required” covers informational and unlisted actors.

Probe identity source

VENDOR_PUBLISHED: the string is the operator’s published identity for that crawler. PRODUCT_TOKEN_FALLBACK: no fixed vendor-published string exists, so the documented product token is sent and the fallback is recorded on each observation.

Vendor-document review dates

2026-07-26, 2026-09-21, 2026-10-02 — the dates a vendor page was actually read for the entries above. A date is never invented for an entry.

Deprecated actor ids: none recorded.

Create account Audit protocols Allow the scanner