For most local businesses, the safe default is to allow the crawlers that power AI search features and decide separately about the ones used for model training. Blocking everything by habit can make you invisible in ChatGPT’s search results, which is the opposite of what a business that wants customers should do. Here is what the main AI crawlers do and how to decide.
What do OpenAI’s crawlers actually do?
According to OpenAI’s crawler documentation, there are three you’ll see in your logs:
| Crawler | Purpose | robots.txt effect |
|---|---|---|
| OAI-SearchBot | Surfaces sites in ChatGPT’s search features | Allowing it helps you appear in ChatGPT search answers; opting out means you won’t appear there, though navigational links may still show |
| GPTBot | Crawls content that may be used to train OpenAI’s models | Disallowing it signals your content shouldn’t be used for training |
| ChatGPT-User | Visits pages when a user asks ChatGPT to, or via GPT Actions | User-initiated, so robots.txt rules may not apply |
OpenAI says each setting is independent. You could allow OAI-SearchBot and disallow GPTBot. It also notes that if both are allowed, it may use the results of one crawl for both purposes.
What about Google?
Google’s Search crawler, Googlebot, follows robots.txt rules, and those rules govern whether your pages can be crawled for Search, including its AI features. Google’s AI features page says to allow crawling in robots.txt and in any CDN or hosting setup, and points to a separate control for limiting AI training and grounding in other Google systems.
What’s a sensible default for AI crawlers?
- Allow Googlebot and OAI-SearchBot. You want to be found in search, including AI search.
- Make a deliberate choice about training crawlers. A local business with ordinary service pages usually has little to protect. If you publish original, paid or sensitive content, you may prefer to disallow training crawlers.
- Don’t block by accident. Some security plugins and CDNs block unknown bots by default. Check your settings.
How do you check your site?
- Open yourdomain.com/robots.txt and read it. Look for broad “Disallow: /” rules.
- Check your hosting or CDN’s bot protection settings.
- Use Search Console’s URL Inspection tool to confirm Google can access key pages.
- Allow up to a day for OpenAI’s search crawler changes to take effect, per its documentation.
Is robots.txt a security measure?
No. It’s a request that well-behaved crawlers follow. It doesn’t protect private information. Anything confidential should sit behind a login.
Review your AI crawlers settings whenever your hosting or security plugin changes. Not sure what your robots.txt is doing? Our technical SEO checks cover it, or send us your domain and we’ll take a look.
Frequently Asked Questions
Will blocking GPTBot remove me from ChatGPT search?
Not according to OpenAI’s documentation. GPTBot relates to model training, while OAI-SearchBot controls inclusion in ChatGPT search features, and the settings are independent.
Should I block all AI bots to protect my content?
For most local businesses, no. Blocking search-related crawlers can reduce your visibility in AI answers. Block training crawlers only if you have a specific reason.
What is ChatGPT-User?
It is the agent that visits a page when a person asks ChatGPT to look at it. OpenAI says robots.txt rules may not apply because the user initiates the action.
Does robots.txt affect Google’s AI Overviews?
Google says Googlebot’s robots.txt rules govern crawling for Search, and to appear in AI features pages need to be indexed and eligible for snippets.
How do I edit robots.txt on WordPress?
Most SEO plugins let you edit it from the dashboard, or you can edit the file on your server. Back up the current version first and re-test after any change.