AI Search Analyzer
Crawler policy

Our crawler: AISearchAnalyzerBot

AISearchAnalyzerBot reads public web pages when someone runs a scan on AI Search Analyzer. It identifies itself with this user agent:

AISearchAnalyzerBot/1.0 (+https://aisearchanalyzer.com/bot)

How it behaves

  • It only fetches publicly accessible pages, never logs in, never submits forms and never takes actions on your site.
  • It reads at most 50 pages for a free scan and 200 for a deep audit, a few at a time.
  • It follows robots.txt rules for its own user agent.
  • It does not run JavaScript. It sees the same raw HTML most AI crawlers see, which is the point of the scan.
  • To check whether your firewall treats AI agents differently, a scan also requests your homepage once with a browser user agent and once with an AI agent’s user agent, and compares the two.

Let it in

If a scan says your site turned us away or asked us to prove we’re human, your firewall or bot protection is blocking automated visitors. It’s very likely blocking ChatGPT, Perplexity and other AI tools the same way, which keeps your business out of their answers. Fixing it for us usually fixes it for them.

  • Cloudflare: in Security, open WAF, add a custom rule where the user agent contains AISearchAnalyzerBot, and set the action to Skip. While you’re there, check that the bot and AI crawler blocking settings match what you want.
  • Other firewalls and hosts (Sucuri, Wordfence, Akamai, Vercel, Netlify and similar): add AISearchAnalyzerBot to the allowed user agents or bot allowlist.
  • A password on the whole site (common on staging sites and some site builders): remove it, or scan the public address instead.
  • robots.txt: make sure it doesn’t block AISearchAnalyzerBot, or every user agent with User-agent: * and Disallow: /.

Then run the scan again.

How to block it

Add this to your robots.txt:

User-agent: AISearchAnalyzerBot
Disallow: /

Exclusion and removal

To have your site excluded from scans run by other people, or stored snapshots removed, email hello@aisearchanalyzer.com from an address at your domain. Our Acceptable Use and Crawler Policy has the details.