Why Is ChatGPT Not Crawling My Site? 12 Checks and Fixes

If ChatGPT is not crawling your site, the cause is usually one of four things: the wrong OpenAI bot is being tested, robots.txt blocks it, a firewall or CDN challenges it, or the useful page content is not available in the response the bot receives. Our ecommerce SEO services include crawler and rendering diagnosis, but you can isolate most access failures yourself with the checks below.

Quick diagnosis: first decide whether you want ChatGPT search discovery, user-requested page retrieval, or model training access. These use different OpenAI user agents. Then check robots.txt, the bot’s real HTTP response, server logs, raw HTML, canonical tags and internal discovery. A normal browser loading the page does not prove an automated agent can reach it.

ChatGPT crawling is not one single process

The phrase “ChatGPT crawler” hides three separate jobs. OpenAI’s official crawler documentation lists different user agents for different purposes. Mixing them up can produce a false diagnosis.

You can allow OAI-SearchBot while blocking GPTBot. That is a valid policy choice. It also explains why changing only the GPTBot rule may do nothing for a ChatGPT search visibility problem.

1. Check the exact bot in robots.txt

Open https://yourdomain.com/robots.txt in a private browser window and find every rule that could apply to OAI-SearchBot and ChatGPT-User. A broad User-agent: * block can matter even when there is no OpenAI-specific section.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This example allows ChatGPT search discovery while declining GPTBot training access. Do not paste it blindly. Keep any private account, cart, checkout, internal-search and staging paths protected according to your own policy.

2. Look for conflicting robots rules

Robots files often accumulate rules from plugins, hosts, security tools and old staging configurations. Check for duplicate OpenAI sections, an accidental sitewide Disallow: /, environment variables that generate different files, and a robots.txt served from a CDN cache after the origin changed.

Also confirm the file returns a normal 200 response and plain text. A login page, redirect loop or HTML error document at the robots.txt URL is not a valid instruction file.

3. Test the response outside your normal browser

Your browser carries cookies, executes JavaScript and may be trusted by the firewall. An automated request may receive a different response. Test representative product, category, article and homepage URLs using a plain HTTP client and an OpenAI user-agent string. Record the status code, final URL, response size and first part of the body.

Look for 401, 403, 429 and 503 responses, redirects to a challenge page, a body that says “enable JavaScript,” or a successful 200 status that contains only a bot-check screen. OpenAI’s crawler access guidance specifically calls out firewalls, CDNs, bot mitigation, JavaScript challenges, CAPTCHAs, authentication and geographic restrictions as possible blockers.

4. Check the CDN and web application firewall

Cloudflare, hosting security, WordPress protection plugins and enterprise WAF rules can override robots.txt. Review bot-management events around the exact time of a test. Search for the requested URL, response code, user agent and source address.

Do not create a rule that trusts any visitor merely because it claims to be OAI-SearchBot. User-agent strings can be copied. Use the verification method and published network information provided by the crawler operator, then scope the allow rule to the minimum needed public content.

5. Read server logs instead of guessing

Logs answer three different questions: did a request arrive, what did your server return, and which URLs were requested? If no verified request appears, the page may not have been discovered yet. If requests appear with 403 or 429 responses, access is the problem. If they receive 200 responses, move on to content, canonical and discovery checks.

A useful log review includes timestamp, user agent, verified source, requested path, status, bytes returned and response time. One homepage request does not prove deeper product URLs are reachable.

6. Remove access gates from public pages

Password protection, age gates, consent walls, country selectors and mandatory location popups can stop retrieval. So can a staging flag accidentally left on production. Test without an existing session and from more than one region if the store applies location rules.

Keep checkout and customer data private. The goal is not to expose protected content. It is to ensure public pages that should be discoverable can be requested without a human completing an interaction first.

7. Put the important answer in the delivered HTML

If the first response contains only an empty application shell, a crawler may have little to interpret. Compare “view source” with the rendered page. Product names, descriptions, price context, availability, category relationships, policies and article answers should be present in accessible text whenever the platform allows it.

This is also good search practice. Google’s technical requirements emphasize crawlable pages, successful responses and indexable content. AI visibility does not replace those foundations. Our ecommerce technical SEO guide shows how rendering, canonicals and crawl controls fit together.

8. Check canonical tags and redirects

A bot may crawl a URL yet treat another URL as the preferred version. Inspect the final response after redirects and the page’s canonical tag. Watch for canonicals pointing to staging, a different market, the homepage, an HTTP version, or a filtered URL that is not the intended answer.

Redirect chains add another failure point. Use one direct permanent redirect when a page has moved. Do not redirect every missing product to the homepage, because that removes the relationship between the old and new resource.

9. Make the page discoverable through real links

Allowing a bot is not the same as giving it a route. Important pages should be linked from navigation, categories, related content or other crawlable pages using ordinary HTML links. An XML sitemap helps discovery, but it should support rather than replace the site’s architecture.

For a deeper audit, use our guide to ecommerce site architecture and check whether priority products sit several clicks deeper than low-value filters and tags.

10. Separate crawling from citation

A successful fetch does not guarantee that ChatGPT will cite or recommend the page. Access is only the first gate. The page still needs to answer a relevant question, identify the business or product clearly, support claims, remain current and fit the user’s request.

This distinction saves time. If verified logs show successful access, repeatedly editing robots.txt is unlikely to fix a recommendation problem. Move to product data, evidence, entity consistency and query fit. The companion guide on why stores are missing from AI product recommendations covers that second layer.

11. Do not treat llms.txt as a magic switch

An llms.txt file can summarize useful resources for systems that choose to read it, but it is not a substitute for accessible HTML, internal links, accurate structured data or crawler permission. Publishing the file does not force a model to crawl, index, cite or rank a site.

Start with observable failures. Fix denied requests, unusable responses and broken discovery before adding optional machine-readable summaries.

12. Retest one controlled URL

Choose a stable public page with useful text and one clear canonical. Record the current robots response, WAF event, server response, raw HTML and internal links. Make one change, purge the relevant cache, and repeat the same test. This prevents five simultaneous changes from hiding which fix worked.

Then monitor logs over time. Crawling is scheduled by the external service, so a clean technical test cannot promise when a new visit, citation or search appearance will happen.

A fast decision tree

1. No request in verified logs: confirm the correct bot, robots permission and discoverable links.
2. Request receives 403, 429 or a challenge: fix the CDN, WAF or rate-limit rule.
3. Request receives 200 but empty or gated HTML: fix delivery, rendering or access gates.
4. Useful HTML receives 200 but another canonical is declared: repair canonical and redirect signals.
5. Everything is accessible but there is no citation: investigate answer quality, product evidence and query fit.

What to send a developer

A useful ticket is specific: “Verified OAI-SearchBot requests to these five public URLs receive HTTP 403 from the CDN while ordinary requests receive 200. Allow verified bot traffic to public product, collection and article paths, keep account and checkout paths protected, purge robots.txt cache, and return before-and-after response evidence.”

That is implementable. “Make us visible in ChatGPT” is not. If the fault crosses WordPress, Shopify, hosting and CDN settings, an ecommerce SEO fix sprint can turn the confirmed failure into a scoped implementation and acceptance test.

FAQs

Does blocking GPTBot stop my site appearing in ChatGPT search?

Not necessarily. GPTBot is associated with model improvement, while OAI-SearchBot is used for ChatGPT search discovery. Review each user agent separately against your policy.

Why can Google crawl my site while ChatGPT cannot?

Your robots.txt, CDN or firewall may have different rules for each crawler. A site can allow Googlebot and challenge OAI-SearchBot or ChatGPT-User.

How long does ChatGPT take to crawl a site after robots.txt changes?

There is no dependable public recrawl deadline. Confirm the new file is live and uncached, test access, then monitor verified server logs rather than assuming an immediate visit.

Will Product schema make ChatGPT crawl my ecommerce site?

No. Structured data can clarify product facts after access, but it does not override robots.txt, authentication, a firewall denial or poor discovery.

Can I allow search access but block AI training?

Yes. OpenAI documents separate user agents, so site owners can express different rules for OAI-SearchBot and GPTBot.