Skip to main content
Generative Metrics logo
Status loading…Create account

Allow the Generative Metrics scanner

Serving AI crawlers and blocking them are both legitimate choices. Here is how to let the scanner measure a protected page without opening your site to everything.

Audit

Record what each named protocol observed.

Diagnose

Trace each production finding to evidence.

Verify

Re-run the protocol required by the remediation.

Allowing and blocking are both legitimate choices

CDNs, WAFs, and bot-management services routinely challenge automated clients, and choosing which automated clients to serve is the site owner's call. A site that serves AI crawlers and a site that blocks them are both clean, coherent paths; the audit records the wall it meets as a finding and never treats blocking as a fault. When our scanner is challenged or denied, the audit reports that it could not measure the page. To have a protected page measured, allow the scanner as a known, verified client.

Allow the scanner in two minutes

The verified, domain-scoped token is the supported path — it is verified and scoped to your domain. The generic per-provider recipes further down this page are second.

  1. 1. Create a free account and open Protected Sites.
  2. 2. Add your domain and choose your provider. You get a one-time scan token — copy it immediately, it is shown only once.
  3. 3. Add the allow rule below to your WAF or bot management, above any block or challenge rule.
  4. 4. Press Verify Setup. When it succeeds, the domain is marked active and future scans carry the token.

What the scanner sends

Direct HTTP evidence (no JavaScript)
A branded user agent: GenerativeMetrics/1.0 (+https://generativemetrics.com)
Or for unauthenticated single-page quick checks: GenerativeMetricsQuickCheck/1.0 (+https://generativemetrics.com)
For registered domains with a scan token, direct-HTTP acquisition also carries X-GM-Scan-Token.
Scanner egress addresses (declared)
Our scans are sent from 144.91.112.237 and 2a02:c207:2303:7469::1. (These addresses can change; the per-domain token path is the stable way to allow us.)
Browser-rendered capture and protected scans
A normal browser user agent plus the registered scan-token header: X-GM-Scan-Token: <your token>

The token is scoped to your domain and is never forwarded to any other host during redirects. Named AI-crawler probes are separate: they send that crawler's own published token, because they test how your access controls treat requests carrying that name. Each scan also sends one ordinary request as our published scanner identity, under the same crawling rules as every request we send, so we can tell AI-specific blocking from general robot blocking. Every scan arrives as a first-time visitor: we send no cookies, store no consent state, and click nothing, including consent screens. We measure what an unconsented machine visitor receives, which is what AI systems without stored consent experience. A machine-readable copy of this identity is published at /gm-scanner.json (and /.well-known/gm-scanner.json).

Generic provider recipes

Second to the token path above: manual recipes if you prefer to allow by rule without registering a domain.

Cloudflare

  1. Log in to Cloudflare and select your domain.
  2. Go to Security > WAF > Custom Rules.
  3. Create or edit a rule named "Allow Generative Metrics Scanner".
  4. Set expression: any(http.request.headers["x-gm-scan-token"][*] eq "YOUR_TOKEN_HERE")
  5. Set action to Skip.
  6. Under components to skip, select "All remaining custom rules".
  7. Place this rule above any block or challenge rule for the domain, then save and deploy.

Akamai

  1. Open your Akamai security configuration for this property.
  2. Add an allow/bypass rule for requests with a custom header match.
  3. Condition: X-GM-Scan-Token equals "YOUR_TOKEN_HERE".
  4. Set action to allow/bypass bot and WAF checks, then activate the property.

DataDome

  1. Open your DataDome dashboard for this protected domain.
  2. Create an allow rule based on an HTTP header.
  3. Condition: X-GM-Scan-Token equals "YOUR_TOKEN_HERE".
  4. Apply and deploy the rule so scanner requests bypass blocking challenges.

Vercel

  1. Log in to Vercel and select your project.
  2. Go to Settings > Security > Firewall (or Attack Challenge Mode).
  3. Under Custom Rules, add a new rule named "Allow Generative Metrics Scanner".
  4. Condition: Header "X-GM-Scan-Token" equals "YOUR_TOKEN_HERE".
  5. Action: Set action to Bypass or Allow.
  6. Save and deploy the rule to your production domains.

AWS WAF

  1. Open the AWS WAF console and select your Web ACL.
  2. Go to Rules > Add rules > Add my own rules and rule groups (Rule Builder).
  3. Inspect: Single header. Header field name: "x-gm-scan-token".
  4. Match type: Exactly matches string. Value: "YOUR_TOKEN_HERE".
  5. Action: Set rule action to Allow.
  6. Ensure this rule is positioned with priority above any rate-limiting, block, or challenge rules.
  7. Save the rule and Web ACL.

Nginx

  1. Open your Nginx configuration file for this server block.
  2. In your server or location block, inspect the request header: $http_x_gm_scan_token.
  3. Match condition: if ($http_x_gm_scan_token = "YOUR_TOKEN_HERE") { set $bypass_security 1; }
  4. Bypass rate limiting or challenge rules when the token matches.
  5. Run "nginx -t" to test configuration syntax, then reload Nginx ("systemctl reload nginx").

Apache

  1. Open your Apache virtual host configuration or .htaccess file.
  2. Check the incoming request header: SetEnvIf X-GM-Scan-Token "^YOUR_TOKEN_HERE$" GM_SCANNER_ALLOWED=1
  3. Or using mod_rewrite: RewriteCond %{HTTP:X-GM-Scan-Token} =YOUR_TOKEN_HERE
  4. Bypass ModSecurity, bot challenges, or access restrictions when the token matches.
  5. Test configuration with "apachectl configtest", then reload Apache.

Generic WAF

  1. For platforms not listed in the kit, open your reverse proxy, CDN, or WAF configuration.
  2. Create an allow or bypass rule matching the custom request header "X-GM-Scan-Token".
  3. Condition: Request header "X-GM-Scan-Token" equals "YOUR_TOKEN_HERE".
  4. Action: Allow the request or bypass bot protection, challenges, and rate limits.
  5. Position this allow rule with priority above any blocking or challenge rules, then save and deploy.

If you cannot add a rule

  • Some hosts gate custom firewall rules behind a higher plan. If your WAF cannot match request headers, allow the branded scanner user agent on the paths you want audited, or ask your provider to allowlist the scanner as a verified client.
  • The token changes whether we can measure the page. It never changes what the audit reports, and it does not affect other crawlers or visitors.
  • Removing the token or disabling the domain restores your original access controls immediately.
Create free accountRead scope FAQContact support