Can AI Search Tools Read Your Website? Robots.txt, AI Crawlers and Accidental Blocking

Quick answer: AI search tools use their own crawlers (like OAI-SearchBot for ChatGPT search, PerplexityBot and Claude-SearchBot) to read websites. If your robots.txt file, hosting firewall or security service blocks them, those tools cannot read your pages and have to rely on what other sites say about you. Most small businesses should allow the search crawlers, then decide separately whether to allow crawlers used for AI training.

Business owners spend money on content and SEO, then wonder why AI tools never cite their site. Sometimes the answer is technical and surprisingly simple: the site is blocking the bots that AI search tools send. It can happen because of an old robots.txt rule, a security plugin, or a hosting setting that started blocking AI crawlers by default. Here is how to check, what to allow and what the tradeoffs are.

Search crawlers vs. training crawlers

This is the most important distinction, and it confuses a lot of people. AI companies typically run more than one crawler, each with a different job:

• Training crawlers collect content that may be used to train future AI models. Examples include OpenAI's GPTBot and Anthropic's ClaudeBot.
• Search crawlers index pages so the AI tool can find and cite them when answering questions. Examples include OAI-SearchBot (ChatGPT search), PerplexityBot and Claude-SearchBot.
• User-triggered fetchers visit a page because a user asked the AI to look at it, such as ChatGPT-User, Perplexity-User and Claude-User.

Each company documents its own crawlers and how they treat robots.txt, and the names and rules do change, so check the official documentation before making changes. The key point: you can block training while still allowing search. Blocking everything means you are invisible to AI search.

What about Google and Bing?

Google's AI Overviews and AI Mode use content Google's regular crawler (Googlebot) already indexes. Google also has a separate control called Google-Extended, which it says governs use of content for training and grounding its Gemini models but does not affect inclusion or ranking in Google Search. Microsoft Copilot leans on the Bing index, so Bingbot access matters. Never block Googlebot or Bingbot unless you mean to disappear from search.

How to check your robots.txt in two minutes

1. Type your domain followed by /robots.txt into a browser (for example, yourbusiness.com/robots.txt).
2. Look for lines that say "User-agent:" followed by a bot name, and "Disallow: /" under it. That combination blocks the bot from the whole site.
3. Watch for "User-agent: *" with "Disallow: /". That blocks every bot that follows the rules, including Google. It is sometimes left over from when a site was in development.
4. If there is no robots.txt at all, bots are generally allowed by default.

A simple, AI-search-friendly robots.txt for most small business sites allows everything except private areas (like an admin login) and lists your sitemap.

Blocking that doesn't show up in robots.txt

Robots.txt is a polite request. Some blocking happens at a different layer and will not appear in that file:

• CDN and firewall settings. Some content delivery and security services offer, and in some cases default to, blocking known AI crawlers. Cloudflare, for example, began blocking AI crawlers by default for new domains in 2025, with options to allow specific ones. Check your dashboard for bot or AI crawler settings.
• Security plugins. WordPress security tools sometimes block unfamiliar user agents or rate-limit aggressively.
• Website builders. Some platforms add their own AI crawler toggles in settings.
• Bot protection pages. "Checking your browser" screens and CAPTCHAs can stop crawlers from seeing content.

If you are not sure, check your server or CDN logs for visits from these bots, or ask whoever manages your site.

Should a small business allow AI training crawlers?

This is a business decision, not a technical one. Some things to weigh:

• Allowing training may help AI models "know" your business in the long run. For a local service business with mostly marketing content, the downside is usually small.
• Blocking training makes sense if your site contains original work you sell (courses, proprietary guides, premium research) and you do not want it absorbed into models.
• Either way, allow search crawlers if you want to be cited in AI answers.

There is no proven ranking benefit from allowing training crawlers, so don't let anyone tell you it guarantees AI visibility.

Other things that stop AI tools from reading your pages

• Content loaded only by JavaScript. Many AI crawlers do not run scripts the way a browser does. Important text should be in the HTML.
• Text inside images. Prices, service lists and hours placed in a graphic are invisible to most crawlers.
• Noindex tags on pages you actually want found.
• Slow or unreliable hosting that times out.

A related topic is llms.txt, a proposed file that summarizes your site for AI tools. It is optional and not a substitute for being crawlable. See what llms.txt is and whether you need one.

A quick checklist

• robots.txt does not block Googlebot, Bingbot, OAI-SearchBot, PerplexityBot or Claude-SearchBot.
• CDN, firewall and security plugin settings allow AI search crawlers.
• Your sitemap is listed and submitted in Google Search Console and Bing Webmaster Tools.
• Key content is plain HTML text, not images or script-only.
• No leftover "Disallow: /" from development.

For the full strategy beyond access, see AI SEO: how to optimize your website for AI search engines.

What a sensible setup looks like

For most small business websites, a sensible setup is straightforward: allow all the major search crawlers (Googlebot, Bingbot and the AI search bots), block only private areas like admin logins or checkout pages, and list your sitemap at the bottom of robots.txt. Then confirm your CDN or security tool isn't overriding that with its own bot blocking.

If you decide to block AI training while staying visible in AI search, you would add rules for the training crawlers only, such as GPTBot and ClaudeBot, and leave the search and user-triggered crawlers alone. Keep a note of what you blocked and why, because crawler names change and new ones appear. Review the file once or twice a year, or any time you change hosting or security providers.

Frequently Asked Questions

Does blocking GPTBot remove me from ChatGPT search?

According to OpenAI's documentation, GPTBot (training) and OAI-SearchBot (search) are controlled separately, so blocking GPTBot alone should not remove you from ChatGPT search results. Blocking OAI-SearchBot would.

Do all AI crawlers obey robots.txt?

The major companies say their crawlers do, though user-triggered fetchers may be treated differently. Some smaller or unknown bots ignore it, which is why firewall-level controls exist.

Will allowing AI crawlers slow down my site?

For a typical small business site, crawler traffic is minor. If you see heavy load from a specific bot, you can rate-limit it rather than block it entirely.

Not sure what your site is blocking? Our team checks this as part of every SEO and AI search engagement. Get in touch and we'll take a look.