The fastest way to check whether ChatGPT can see your website in 2026 is to open yoursite.com/robots.txt and look for three tokens: OAI-SearchBot, ChatGPT-User, and GPTBot. If any of them sits under a Disallow rule, or if your Cloudflare bot settings block the AI crawler categories, ChatGPT cannot fetch your pages no matter how strong the content is. The most common reason the answer is no is a blanket block somebody added and forgot about. Roughly 25 percent of the top 1,000 websites now block GPTBot, up from about 5 percent in early 2023, and a large share of them blocked ChatGPT search access by accident on the way.
This post is the test itself. Five steps, five explicit pass or fail signals, real commands you can paste. The wider diagnosis of why a site stays invisible after the crawlers are confirmed working lives in why your website is not showing in AI search. Start here first, because there is no point tuning schema and content if OpenAI, Microsoft Bing, and Perplexity are getting a 403 at the door.
Which crawlers does ChatGPT use to see my website?
Three, and they do different jobs. Per OpenAI’s own bots documentation, OAI-SearchBot indexes pages for ChatGPT search, ChatGPT-User fetches a page live when a person asks ChatGPT to open a URL, and GPTBot collects content for model training. Each one is a separate robots.txt token with separate rules.
Here are the exact strings to search for:
OAI-SearchBot(full agent stringOAI-SearchBot/1.0). This is the one that decides whether you can appear in ChatGPT search results. Block this and you are out of ChatGPT search.ChatGPT-User(ChatGPT-User/1.0). Live retrieval when a user pastes your URL or asks ChatGPT to check your site. Block this and ChatGPT tells the user it cannot access your page.GPTBot(GPTBot/1.1). Training data collection only. Blocking this has no effect on ChatGPT search visibility.
OpenAI publishes machine readable IP range files for all three at openai.com/searchbot.json, openai.com/chatgpt-user.json, and openai.com/gptbot.json, so you can verify a request in your logs actually came from OpenAI and not a scraper wearing the name.
The distinction matters because the settings are independent. A rule that blocks GPTBot leaves OAI-SearchBot untouched, and a rule that blocks OAI-SearchBot still lets GPTBot train on you. Most sites get this backwards.
Crawler access is the cheapest AEO problem you will ever fix, and the one nobody checks. Get your free AI visibility audit and see which bots your site is turning away right now.
What is the five minute test to see if ChatGPT can reach my site?
Run these five steps in order. Each has a single pass or fail read. Do not skip to step 5. Most failures are found in steps 1 and 4.
Step 1. Read your robots.txt
Open a terminal and run this, substituting your domain:
curl -s https://yoursite.com/robots.txt
Or just visit https://yoursite.com/robots.txt in a browser. Then read the whole file, not the first block.
Fail signals. Any of these means ChatGPT search is blocked:
User-agent: OAI-SearchBot
Disallow: /
User-agent: *
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Pass signals. No mention of OAI-SearchBot or ChatGPT-User anywhere in the file, or an explicit Allow. A 404 on robots.txt is also a pass, since no file means no restrictions. A User-agent: GPTBot with Disallow: / is a pass for search visibility, because that token only governs training.
One trap: the agent token is case insensitive but exact on spelling. OpenAI-SearchBot and ChatGPTBot are not real tokens and do nothing. If you see either, whoever wrote the file was guessing.
Step 2. Ask ChatGPT to open your URL
Open ChatGPT and paste this prompt exactly, with your real URL:
Open https://yoursite.com/your-key-page/ and quote the H1 and the first full sentence of body copy, word for word.
Asking for a verbatim quote is the whole trick. It forces a live fetch through ChatGPT-User and makes bluffing visible.
Pass signal. ChatGPT returns text that matches your page exactly. That means ChatGPT-User reached your server, got a 200, and parsed real HTML.
Fail signals. Any of these:
- “I was unable to access that site” or “the site appears to be blocking access.”
- A summary that sounds right but contains no exact match to your copy. That is the model answering from training memory, not a live fetch.
- It quotes a competitor, an old version of the page, or your homepage instead of the URL you gave it.
Then confirm from your side. Within a minute of running the prompt, check your access log:
grep -iE "ChatGPT-User|OAI-SearchBot|GPTBot" /var/log/nginx/access.log | tail -50
You want to see a ChatGPT-User/1.0 line with a 200 status. A 403 or 429 in that line is a hard fail and points you straight to step 4.
Step 3. Check whether Bing has you indexed
Go to bing.com and run this query:
site:yoursite.com
ChatGPT search leans on Bing’s index as its documented retrieval layer, with OAI-SearchBot supplementing it with fresh crawls. If Bing does not have your pages, one of the two main paths into ChatGPT search is closed.
Pass signal. The result count is in the same ballpark as your real page count. A 40 page site returning 30 to 50 results is healthy.
Fail signals. Zero results, or a handful for a site with hundreds of pages. Open Bing Webmaster Tools, run URL Inspection on a page you care about, and read the reason. Common ones are “Discovered but not crawled,” a blocked robots.txt rule, or the site was never submitted. The fix path is in how to get indexed on Bing.
Step 4. Check your CDN and WAF
This is where most real blocks live. robots.txt is a request. A web application firewall is enforcement, and it can return a 403 to OAI-SearchBot while your robots.txt says everything is fine.
Test the enforcement layer directly:
curl -A "OAI-SearchBot/1.0; +https://openai.com/searchbot" -I https://yoursite.com/
curl -A "ChatGPT-User/1.0; +https://openai.com/bot" -I https://yoursite.com/
Pass signal. HTTP/2 200 on both.
Fail signals. 403, 401, 429, 503, a redirect chain that never resolves, or an HTML challenge page in the body instead of your content. Compare against a normal browser user agent. If a plain curl returns 200 and the OAI-SearchBot curl returns 403, your firewall is the problem, not your content.
Then check the dashboard. On Cloudflare, go to your domain, then Security, then the bot and AI crawler controls. Cloudflare began blocking AI crawlers by default for new domains in July 2025, and it is moving to a three category model covering Search, Agent, and Training that takes effect September 15, 2026. Under that model, Training and Agent are blocked by default on ad supported pages for new domains, new sites on existing accounts, and free tier accounts that never changed their settings, while Search stays allowed.
Two things to watch. If your site launched recently on Cloudflare, assume you are blocking something until you have looked. And Cloudflare has warned that blocking the Training category also blocks multipurpose crawlers including Googlebot, Bingbot, and Applebot, because those bots crawl for more than one purpose under a single agent, so that block costs you classic search too. Then check Security Events filtered by user agent for the last seven days. If OAI-SearchBot or ChatGPT-User shows up in the blocked list, you have your answer. Fastly, AWS WAF, Akamai, and most managed WordPress hosts have equivalent bot rule sets. Same test, same read.
Step 5. Check whether your content survives without JavaScript
AI crawlers fetch HTML. They do not run your front end. An analysis of hundreds of millions of AI crawler fetches published by Vercel found no evidence of JavaScript execution by GPTBot at all. GPTBot downloads JavaScript files in roughly 11.5 percent of requests and Claude’s crawler in roughly 23.8 percent, but neither executes them. As of 2026 this holds for OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, meta-externalagent, and Bytespider. Google’s Gemini is the exception, since it inherits Googlebot’s rendering service.
Run this on a page that matters:
curl -s https://yoursite.com/your-key-page/ | grep -i "your key phrase here"
Or in Chrome: open DevTools, press Command+Shift+P, type “Disable JavaScript,” press Enter, then reload the page.
Pass signal. Your H1, body copy, service list, attorney or physician bios, phone number, and address are all present in the raw HTML.
Fail signals. The curl returns nothing, or the page with JavaScript off is a blank shell, a spinner, or an empty div plus a script bundle. Reviews, pricing, or FAQ accordions that only appear after hydration do not exist as far as ChatGPT is concerned. The fixes, mostly server side rendering or static generation, are in can AI crawlers read JavaScript.
What robots.txt should I use if I want ChatGPT to see my site?
If you want maximum ChatGPT visibility, say so explicitly. Silence works, but an explicit Allow survives the next person who edits the file:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
If you want to appear in ChatGPT search but keep your content out of model training, split it:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Google-Extended works the same way for Gemini training and does not affect Googlebot or Google’s search index. Applebot-Extended is Apple’s training opt out and is separate from Applebot. Always keep the Sitemap line. OAI-SearchBot and Bingbot both use it for discovery.
Does blocking GPTBot also block ChatGPT search?
No. GPTBot governs training data collection only. OAI-SearchBot governs whether your pages can be indexed for ChatGPT search, and ChatGPT-User governs live retrieval. OpenAI treats all three as independent settings, so a GPTBot block leaves ChatGPT search access fully intact.
This is the most misunderstood point in the category, and it cuts both ways. Publishers who blocked only GPTBot to protect training data are still fully crawled by OAI-SearchBot, which is usually what they wanted. Site owners who copied a “block AI bots” snippet off a forum in 2023 often blocked all three, or used a User-agent: * wildcard, and quietly removed themselves from ChatGPT search for two years without noticing.
What about PerplexityBot, ClaudeBot, and the other AI crawlers?
Same test, different tokens. Every assistant runs its own agents, and blocking one does not block the others. Check your robots.txt and firewall for all of them, not just OpenAI’s.
The current roster worth naming: PerplexityBot and Perplexity-User for Perplexity, ClaudeBot and Claude-SearchBot for Anthropic’s Claude, Google-Extended for Gemini training, Googlebot for Google search and AI Overviews, Bingbot for Bing and Microsoft Copilot, Amazonbot for Alexa, Applebot and Applebot-Extended for Apple, meta-externalagent for Meta AI, and Bytespider for ByteDance.
Googlebot and Bingbot are the load bearing two, because Google AI Overviews and AI Mode run on Google’s index while ChatGPT search and Copilot run on Bing’s. Blocking either takes out classic search and AI search in one move. Bytespider and Amazonbot are the most commonly blocked for bandwidth reasons, and blocking them costs almost nothing in Western markets.
How long after I fix the block will ChatGPT see my site?
Not instantly, and the three paths move at different speeds. ChatGPT-User is immediate, since it fetches live the moment somebody gives ChatGPT your URL, so retest that within minutes of removing a firewall rule.
OAI-SearchBot has to recrawl and Bing has to index or reindex you, which runs days to weeks depending on your crawl history. Submit your sitemap in Bing Webmaster Tools the same day you lift the block, request indexing on your five most important pages, then check your access logs weekly for OAI-SearchBot hits returning 200. The log is your ground truth. Once OAI-SearchBot is fetching you and getting 200s, access is solved and everything after that is a content and authority problem.
Frequently asked questions
Does ChatGPT-User respect robots.txt?
Yes. OpenAI documents ChatGPT-User as an agent that reads robots.txt, so a Disallow rule naming ChatGPT-User will stop live fetches even when a user pastes the URL directly into ChatGPT. In practice ChatGPT then tells the user it cannot access the page and answers from training memory instead, which usually means an outdated version of your business or a competitor’s page getting cited in your place.
Can I appear in ChatGPT search if Bing has not indexed my site?
It is much harder. Bing’s index is the documented retrieval layer behind ChatGPT search and Microsoft Copilot, and OAI-SearchBot supplements it with fresh crawls rather than replacing it. Treat Bing indexing as a prerequisite. Verify with a site:yoursite.com query on bing.com, then submit your sitemap in Bing Webmaster Tools. Bing indexing is free, fast to set up, and most sites have simply never done it.
How do I verify a crawler hit really came from OpenAI?
Two checks, both required. First, confirm the user agent token matches exactly: OAI-SearchBot/1.0, ChatGPT-User/1.0, or GPTBot/1.1. Second, confirm the source IP falls inside OpenAI’s published ranges at openai.com/searchbot.json, openai.com/chatgpt-user.json, or openai.com/gptbot.json. User agent strings are trivially spoofed, so scrapers routinely impersonate GPTBot. Only the IP check settles it.
Does an llms.txt file make ChatGPT see my site?
No. As of 2026 no major AI platform, OpenAI included, has confirmed that it reads llms.txt as a crawl directive. It is a proposed convention, not a standard the crawlers act on. It costs nothing to publish and can help humans and some third party tools, but it will not unblock a crawler, get you into Bing’s index, or override a firewall rule. Fix robots.txt and your WAF first.
Will a noindex tag keep my page out of ChatGPT?
Mostly yes, through the Bing path. Bing honors noindex, so a noindex page will not enter the index that ChatGPT search retrieves from. ChatGPT-User can still fetch the page live if a user hands over the exact URL, since that is a direct request rather than an index lookup. If you want a page out of AI answers entirely, pair noindex with a robots.txt Disallow covering OAI-SearchBot and ChatGPT-User.
My robots.txt looks fine but ChatGPT still says it cannot access my site. What now?
Go straight to step 4. A clean robots.txt with a failing fetch almost always means a firewall, a bot management rule, a rate limit, or a geo restriction. Run the curl test with the OAI-SearchBot user agent and compare the status code against a normal browser agent. Then check Cloudflare Security Events, or the equivalent log in Fastly, AWS WAF, or Akamai, filtered to the last seven days by user agent.
Five checks, one afternoon, and you stop guessing. Get your free AI visibility audit and we will run every one of them against your domain and hand you the results.
Nothing in AEO compounds if the door is locked. Schema markup, press placements, review velocity, and content depth all assume a crawler can reach the page and read it in raw HTML. Get five clean passes and every hour you spend on AI visibility after that actually lands. Skip the test and you are optimizing a page ChatGPT has never once opened. At Subscribe PR the crawler check is always step one, because it is the only part of this work where one line in a text file decides whether anything else counts.
Tagged