← Back to all articles

Key takeaways

  • AI companies run separate crawlers for search answers, model training and live user requests.
  • Blocking a search crawler can quietly remove your business from AI answers customers rely on.
  • Firewalls and security settings can block AI crawlers even when robots.txt allows them.

Most small businesses should let in the AI crawlers that power search answers, such as the ones behind ChatGPT search and Perplexity, and make a separate, deliberate choice about crawlers that collect content for model training. Blocking the wrong one quietly removes you from AI answers, and customers hear about your competitors instead.

60%

Google question searches that produced an AI summary

Pew Research Center, 2025

26%

Searches with an AI summary where users ended their session

Pew Research Center, 2025

18%

All Google searches in the study that produced an AI summary

Pew Research Center, 2025

What is an AI crawler, and what does robots.txt have to do with it?

A crawler is an automated program that visits websites and reads their pages. AI companies send crawlers to learn what businesses exist and what they offer.

Robots.txt is a small file on your website that tells crawlers which ones are welcome. Many sites have one they've never looked at, set up years ago by a developer or a website builder.

That file now decides something important. It can determine whether AI assistants are allowed to read your site at all.

Not every AI crawler does the same job

This is the part most business owners miss. AI companies don't run one crawler each. They run several, and each has a different purpose.

OpenAI is a clear example. According to OpenAI's crawler documentation, OAI-SearchBot is what puts websites into ChatGPT's search results, while GPTBot collects content that may be used to train its models. The two settings are independent, so a site can appear in ChatGPT search while opting out of training.

Perplexity says its PerplexityBot shows and links websites in its results and is not used to train AI models. Apple says Applebot-Extended only controls whether content can be used to train its models, and blocking it does not remove a site from Apple search results.

There is also a third kind. Some fetchers only visit a page when a person asks the assistant a question. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a user started the request.

Why blocking the wrong crawler costs you customers

Customers increasingly get answers without clicking through to websites. Pew Research Center's 2025 study found Google users clicked a traditional result on 8% of visits when an AI summary appeared, versus 15% when none did. They clicked a link inside the summary itself on just 1% of visits.

How often Google users clicked a link (March 2025)
Traditional link, no AI summary15%
Traditional link, AI summary shown8%
Link inside the AI summary1%

Source: Pew Research Center, 2025

So the answer itself is becoming the shop window. If an AI search crawler can't read your site, you can't be part of that answer. OpenAI is direct about this: sites that block OAI-SearchBot will not appear in ChatGPT search answers.

Plenty of sites are shutting the door anyway. Cloudflare's 2025 Radar Year in Review found AI crawlers were the user agents most often fully blocked in robots.txt files. Often that's a blanket decision made out of worry about training, without realizing it also affects search.

What this looks like for a real business

Picture a family dental clinic in Denver. During a website redesign, their developer added a rule blocking every AI bot, reasoning that it protected their content. Nobody mentioned it to the owner.

Months later, a parent asks ChatGPT for a gentle family dentist nearby. The assistant names two other clinics. The Denver clinic has better reviews, but the assistant had no permission to read its site, so it simply wasn't in the running.

Signs your business may have this problem:

  • Rivals get the mention. Customers say they found a competitor by asking an AI assistant.
  • Your site never appears. AI answers about your services cite other websites, never yours.
  • Nobody knows the settings. You can't say who last changed your site's crawler rules or why.
  • A redesign changed things. Your AI mentions seemed to drop after a new site or security update.

Robots.txt isn't the only gatekeeper

Even a perfectly welcoming robots.txt file can be undone by other settings. Perplexity warns that web application firewalls, the security filters many hosts add to block bad traffic, may need to allow its bots explicitly.

It also notes that Cloudflare's Bot Fight Mode may still challenge its crawlers regardless of custom rules. In other words, your site can say "come in" while your security layer quietly says "no".

Search engines matter too. Microsoft's Copilot Studio documentation says generative answers built on public websites draw on Bing search, and robots.txt controls Bingbot's access. Blocking a traditional search crawler can ripple into AI answers as well.

So which crawlers should you allow?

Our view at BeyondVisibl: if you want customers to find you through AI assistants, the crawlers that power AI search answers should be able to reach you. Training crawlers are a separate business decision, and it's reasonable to decide either way.

What "good" looks like is simple to describe. Your settings reflect a choice you actually made, search-facing crawlers can read your pages, and nothing in your security setup contradicts that. The right mix depends on your business, your platform and how your site is hosted, so there's no single answer that fits everyone.

Access is also only the entry ticket. Being readable doesn't mean an assistant will recommend you; it still has to understand and trust your business, as we explain in how ChatGPT decides which businesses to recommend. The first step is checking what your site currently allows, which a tool such as BeyondVisibl does as part of its site readiness check.

If you're new to this area, our guides to answer engine optimization and llms.txt explain the wider picture.

Frequently asked questions

If I block AI training, will I disappear from AI answers?

Not necessarily. OpenAI, Perplexity and Apple all separate training from search, so opting out of training doesn't have to remove you from their search results.

Why would my site block AI crawlers without me knowing?

Developers, website builders and security tools sometimes block AI bots by default or during a redesign. Firewalls can also block them even when your robots.txt file allows them.

Does robots.txt stop every AI visit?

No. OpenAI and Perplexity say their fetchers that act on a live user request may not follow robots.txt, because a person asked for the page.

How quickly do changes take effect?

OpenAI and Perplexity both say changes can take about 24 hours to show up. Being allowed in also doesn't mean you'll be recommended, since that depends on how well AI understands and trusts your business.


Is AI recommending your business?

BeyondVisibl checks whether ChatGPT, Gemini, Perplexity and Claude recommend you, and shows you what to fix. Results in about 2 minutes.

Check my AI visibility →

Last updated November 10, 2026 · By Vishrant, founder of BeyondVisibl