Services

Web Design and BuildSEOGEOAEOGoogle Business ProfilePaid AdsContent and VideoAI Systems

Company

IndustriesMarketsPricingResourcesInsightsAboutContact (210) 729-4008
Home / Answers / AI Search

Should I block AI crawlers from my website?

For most local businesses, no. Blocking AI crawlers removes you from answers your customers are already asking for. Blocking mainly makes sense when your content is the product you sell, which is rare for a contractor, clinic or firm.

The longer answer

You can block them. Robots.txt rules for GPTBot, Google-Extended, ClaudeBot, PerplexityBot and others are widely documented, and the major crawlers honor them. The real question is what you gain. Publishers who sell subscriptions or license their archives have a genuine argument to make. A roofing company whose website exists to get the phone ringing does not. Blocking there simply makes you harder to find and harder to recommend.

Know the difference between crawler types as well. Some fetch pages to train future models, and some fetch pages live to answer a question somebody is asking right now. Blocking the live retrieval crawler is the one that directly costs you visibility, because that fetch is what produces a citation and a clickable link. Blocking only the training crawler while allowing live retrieval is a middle position, and it is a defensible compromise.

Whatever you decide, decide it deliberately. Plenty of sites are blocking by accident through a security plugin, an aggressive firewall rule, a bot protection setting, or a hosting default, and the owner has no idea it is happening. Check your robots.txt and your firewall logs before you assume you are open. If you are a Heights Digital client this is part of the monthly technical work, and you can reach us at (210) 729-4008.

What to take from this

  • Blocking AI crawlers mostly costs local businesses visibility
  • Publishers who license content have a real reason to block
  • Separate training crawlers from live retrieval crawlers
  • Blocking live retrieval removes you from cited answers
  • Many sites block by accident through plugins or firewalls
More answers

People also ask.

Start here

Find out what you are actually missing.

Send the site over. We will tell you what is broken, what you could rank for, and whether AI search knows you exist. No charge and no pitch deck.

  • A real look at your site and your Search Console
  • What your competitors are ranking for that you are not
  • Whether ChatGPT and AI Overviews mention you
  • Pricing in plain numbers, on the call

Or call or text (210) 729-4008. A person answers.

Got it.We will come back to you with something useful, not a pitch deck.
CallLet's talk