How often do AI engines recrawl your site in 2026

AI engines recrawl on no fixed schedule in 2026: OpenAI’s GPTBot and Anthropic’s ClaudeBot hit a typical site in bursts spaced weeks or months apart, PerplexityBot returns every few days on linked pages, and Googlebot (which feeds AI Overviews and AI Mode) revisits active sites daily or faster. Cadence matters less than live retrieval, because ChatGPT-User, Perplexity-User and Claude-User fetch pages at query time, and Cloudflare’s 2025 Radar Year in Review measured user triggered fetching growing more than 15 fold in one year. Per Vercel’s network data, GPTBot alone made 569 million requests in a single month; the question is whether your server lets it in and whether it sees anything worth citing.

Which AI bots actually visit your site, and what does each one do?

Roughly a dozen user agents matter, split across three jobs: training crawls, search index crawls, and live fetches for a user. By engine family:

1. OpenAI (ChatGPT)

GPTBot crawls for model training and is the heaviest by volume. OAI-SearchBot builds the ChatGPT search index and decides whether your page can appear as a linked source. ChatGPT-User fires when a live ChatGPT session opens your URL. OpenAI publishes IP ranges for each and says robots.txt changes take about 24 hours to apply.

2. Anthropic (Claude)

ClaudeBot crawls for training, Claude-SearchBot indexes for search, and Claude-User is the live fetch. All three honor robots.txt. Cloudflare Radar’s crawl to refer ratio for Anthropic was 70,900:1 for the week of June 19 to 26, 2025, the highest of any platform.

3. Perplexity

PerplexityBot builds the search index. Perplexity-User fetches at query time, and Perplexity’s documentation says it generally ignores robots.txt because a human requested the page. Perplexity’s Radar ratio was under 200:1 in 2025, the best among pure AI companies, because its whole product is retrieval.

4. Google (AI Overviews, AI Mode, Gemini)

There is no separate AI crawler. Googlebot feeds classic Search, AI Overviews and AI Mode from one fetch. Google-Extended is a robots.txt token, not a user agent: it controls Gemini training and grounding, and blocking it does not remove you from AI Overviews. Cloudflare’s 2025 Year in Review put Googlebot above 28% of all verified bot traffic.

5. Microsoft (Copilot, Bing)

Bingbot feeds Bing search and Microsoft Copilot from one crawl, and Bing is the primary engine behind IndexNow, your only push button recrawl lever among the major AI platforms.

6. Everyone else

Applebot crawls for Siri and Spotlight; Applebot-Extended is a control token for Apple Intelligence training. Amazonbot feeds Alexa and Rufus. Bytespider (ByteDance) is the most aggressive training crawler on many logs. Meta-ExternalAgent trains Meta’s models. CCBot (Common Crawl) archives the web monthly for training sets.

Want to know which of these bots are hitting your site, which are blocked, and whether ChatGPT can cite you? Get your free AI visibility audit and we will pull the log data.

How often does each engine recrawl in practice?

No engine publishes a per site schedule, so the answer comes from log data and network scale studies. Vercel’s one month network analysis counted 569 million GPTBot requests, 370 million from Anthropic’s Claude crawlers and 24.4 million from PerplexityBot, against 4.5 billion from Googlebot. Cloudflare’s 2025 Year in Review found training crawls ran at up to 8 times the volume of search index crawls and 32 times the volume of user triggered fetches.

On a single mid sized site, per our client logs and published log studies (estimates, not guarantees):

• Googlebot: daily on established sites, hourly on frequently updated URLs, minutes after a Search Console URL Inspection request. Google says crawl budget only constrains sites above roughly one million pages changing weekly or 10,000 changing daily, so a law firm or clinic site is never budget limited.
• Bingbot: every few days organically, within 24 hours of a validated IndexNow ping per Microsoft’s guidance.
• GPTBot: bursty. Hundreds or thousands of URLs in a session, then silence for two to eight weeks. One log study found it recrawled about 42% of pages it had already visited.
• OAI-SearchBot: lighter and more selective. The same study saw it recrawl only 27% of visited pages.
• ClaudeBot: same burst pattern, often heavier per visit. Vercel found more than 34% of GPTBot and Claude requests hit 404 pages, so these crawlers follow stale links and clean their queues slowly.
• PerplexityBot: every few days on linked or sitemapped pages, weekly or slower on orphans.

Bigger, better linked, more frequently updated sites get shorter gaps.

Why does “recrawl” matter less than crawlability for AI answers?

Because most AI answers that cite you are assembled at query time from a live fetch, not a cached crawl, so what matters is whether your page loads in a few seconds and parses without JavaScript. Cloudflare’s data is the proof: user action crawling (ChatGPT-User, Perplexity-User, Claude-User) grew more than 15x during 2025, the fastest growing category on its network.

The index crawl (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) decides whether your URL is a candidate. The live fetch (the -User agents) decides what the model reads when it writes the answer. If OAI-SearchBot last saw your pricing page in April but ChatGPT-User fetches it today, the answer reflects today’s page. If a Cloudflare WAF rule blocks ChatGPT-User, the answer reflects whatever a competitor says about you.

So the real questions are: Can the index crawler discover the URL? Can the live fetcher load it fast? Do freshness signals (Last-Modified, sitemap lastmod, visible dates) say the content is current? We cover the third in our guide to content freshness for AI search.

How do you see AI recrawls in your own server logs?

Grep the access log for the user agent strings and count hits per day per bot, on Apache, Nginx or any host that exposes raw logs:

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User" access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

That gives you OpenAI visits per day; swap in “PerplexityBot|Perplexity-User” and print the URL field to see which pages Perplexity cares about. Without raw logs (managed WordPress, Squarespace, Webflow), you have three other windows:

• Cloudflare: the Bots section and AI Audit dashboard list every AI crawler by name, request count and allow or block status.
• Screaming Frog Log File Analyser: segments verified bots automatically, including GPTBot, ClaudeBot and PerplexityBot, and flags bot hits that return 404 or 5xx.
• Vercel and Netlify: both break bot traffic out in analytics; Vercel’s Bot Observability separates AI agents from search crawlers.

Verify hits, since anyone can spoof a GPTBot string. OpenAI, Anthropic, Perplexity and Google publish IP ranges or reverse DNS methods, and Cloudflare’s verified bot list does the check for you.

What speeds up a recrawl?

Four levers, and only IndexNow is a direct push.

1. IndexNow and Bing Webmaster Tools

One HTTP POST with your changed URLs, up to 10,000 per call. Bing, Yandex, Naver, Seznam and Yep share the signal, and Bing typically crawls within 24 hours of a validated submission. Submit only URLs that changed; repeated pings on stable URLs earn 429 responses. Because Bingbot feeds Copilot, and ChatGPT search has historically drawn on Bing’s index, IndexNow is the closest thing to a recrawl button for two AI platforms at once. Google does not participate. Bing Webmaster Tools also accepts up to 10,000 URLs per day through its own submission API.

2. XML sitemap with honest lastmod

Googlebot, Bingbot, PerplexityBot and OAI-SearchBot all read sitemaps. Google trusts lastmod when the values are accurate and ignores them sitewide when every URL claims to have changed today.

3. ETag and Last-Modified headers

Crawlers that send If-Modified-Since get a 304 Not Modified when nothing changed and spend the visit on your changed pages instead. Sites that return full HTML on every request get revisited less.

4. Internal links from pages the bots already like

GPTBot and PerplexityBot discover new URLs by following links from pages already in their queue. Link a new page from the homepage, your most crawled post and the relevant service page the day it publishes. In our logs, that earns a first PerplexityBot hit in one to three days; an orphan can wait weeks.

What silently blocks AI crawlers?

Four failure modes, and the first affects every site that joined Cloudflare after mid 2025. On July 1, 2025, Cloudflare began blocking AI training crawlers by default for new domains and launched Pay Per Crawl, a marketplace where publishers set a price per fetch.

1. Cloudflare managed rules and Bot Fight Mode. Check the AI crawlers toggle under Security, Bots. You can allow OAI-SearchBot and the -User agents while blocking training bots; blocking everything blocks the search index crawlers too.
2. WAF and rate limits. Rules that challenge non browser user agents serve a JavaScript challenge or 429 to every AI bot, and none of them solve challenges. Whitelist the published IP ranges from OpenAI, Anthropic and Perplexity.
3. JavaScript only rendering. Vercel confirmed that none of the major AI crawlers render JavaScript, including GPTBot, ClaudeBot and PerplexityBot. Client side React without server rendering is an empty shell to them; see whether AI crawlers can read JavaScript for the fix.
4. robots.txt mistakes. A stray Disallow: / under User-agent: *, a security plugin that auto blocks “AI” strings, or a copied block list that includes OAI-SearchBot alongside GPTBot. Test with curl using each user agent string.

How long until a new page shows up in ChatGPT, Perplexity and AI Overviews?

Estimates from our client work, assuming the page is linked, sitemapped and unblocked: Google AI Overviews in one to seven days, Perplexity in two to ten days, Bing Copilot in one to three days with IndexNow, and ChatGPT search in one to four weeks for a stable index entry. These are observations, not published SLAs.

• Google AI Overviews and AI Mode: same timeline as Google indexing. Request indexing in Search Console; established sites are indexed within hours and eligible once the page ranks near the top.
• Perplexity: PerplexityBot picks up linked pages in days, and Perplexity-User can pull the page live the same day.
• Microsoft Copilot: IndexNow ping, Bingbot within roughly 24 hours, Copilot eligibility as soon as Bing ranks it.
• ChatGPT search: the slowest stable path. OAI-SearchBot favors pages with existing authority signals; a new domain can wait a month or more. Press coverage shortens it, because OAI-SearchBot discovers URLs from pages it already trusts.

Publish, link internally, ping IndexNow and request indexing in Search Console within the same hour, then check logs on day three and day ten.

Frequently asked questions

Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI’s training crawler. ChatGPT search citations come from OAI-SearchBot’s index and ChatGPT-User live fetches, so you can disallow GPTBot in robots.txt and still appear in ChatGPT search answers, provided those two are allowed. Confirm your Cloudflare or WAF rules match your robots.txt, because a firewall block overrides anything robots.txt says.

What is Google-Extended and does it affect AI Overviews?

Google-Extended is a robots.txt token, not a crawler. It controls whether your content trains Gemini models and grounds Gemini apps. Google has stated it does not affect inclusion or ranking in Search, and AI Overviews and AI Mode are built from Googlebot’s regular index, so blocking it leaves AI Overviews eligibility untouched. Applebot-Extended works the same way for Apple Intelligence.

Can I force ChatGPT or Perplexity to recrawl a specific page?

Not directly; neither OpenAI nor Perplexity offers a submission tool like Google Search Console. Your indirect levers are IndexNow (which reaches Bing, whose index ChatGPT search has drawn on), fresh internal links from high traffic pages, an updated sitemap lastmod, and asking ChatGPT or Perplexity about the URL, which triggers a ChatGPT-User or Perplexity-User live fetch. That fetch does not update the index, but it shows what the model sees today.

How do I tell a real GPTBot visit from a spoofed one?

Check the IP against OpenAI’s published ranges at openai.com/gptbot.json and the matching files for OAI-SearchBot and ChatGPT-User. Anthropic and Perplexity publish equivalent lists, and Googlebot supports reverse DNS verification. Cloudflare’s verified bot directory flags impersonators automatically. “GPTBot” traffic from a residential IP block is a scraper borrowing the name.

Does IndexNow help with Google AI Overviews?

No. Google has tested IndexNow but does not consume it. IndexNow reaches Bing, Yandex, Naver, Seznam and Yep, and through Bing it reaches Microsoft Copilot. For Google, use Search Console’s URL Inspection with Request Indexing, keep sitemap lastmod accurate, and rely on internal links. Indexing is only step one; the page also needs to rank, since AI Overviews pull from top results.

Should I use a crawl-delay directive for AI bots?

Only Bingbot honors crawl-delay reliably; Googlebot ignores it, and OpenAI, Anthropic and Perplexity do not document support. If ClaudeBot or Bytespider is hammering your server, rate limit by verified IP range at the CDN level instead, leaving enough headroom for a search index crawler to fetch a few hundred pages per session.

The takeaway

Recrawl frequency in 2026 is a function you influence, not a schedule you wait on. Googlebot and Bingbot return daily if you give them fresh, linked, fast pages; GPTBot and ClaudeBot return in bursts you cannot summon; OAI-SearchBot and PerplexityBot return for pages they already trust; and the -User agents from OpenAI, Anthropic and Perplexity return the moment a real person asks about you, as long as your server lets them in. Get the plumbing right (unblocked bots, server rendered HTML, honest lastmod, IndexNow on every change) and cadence stops being the bottleneck. Earned coverage in publications the engines already crawl shortens the gap for the slowest bots, which is why our AEO and PR retainers pair the two. Not sure whether GPTBot has visited your site this quarter, or whether Cloudflare is quietly turning it away? Request your free AI visibility audit and we will read your crawl logs and tell you which engines can see you.