Making Your Website Readable to AI Crawlers

Know which crawlers you are dealing with

The major AI companies publish the names their crawlers use, and several run more than one. OpenAI documents GPTBot (used to gather training data), OAI-SearchBot (used for search features in ChatGPT) and ChatGPT-User (used when a person asks ChatGPT to open a page). Anthropic documents ClaudeBot along with separate agents for search and for user-requested fetches. Perplexity documents PerplexityBot and Perplexity-User.

Google AI Overviews are built on Google Search, so the crawler that matters there is ordinary Googlebot. Google also documents a separate control called Google-Extended, which governs whether your content is used to improve Gemini models; Google states it does not affect inclusion in Search. Microsoft's Copilot draws on Bing, so Bingbot matters too.

• OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
• Anthropic: ClaudeBot and its search and user agents
• Perplexity: PerplexityBot, Perplexity-User
• Google: Googlebot (Search and AI Overviews), Google-Extended (Gemini training control)
• Microsoft: Bingbot

Read your robots.txt with fresh eyes

Open yoursite.com/robots.txt in a browser. Look for any 'User-agent' line that names one of the crawlers above followed by 'Disallow: /'. Look also for a blanket 'User-agent: *' with 'Disallow: /', which blocks every crawler that respects the file. Old staging rules and copied templates are a common cause.

Blocking is a legitimate choice. Some businesses decide they do not want their content used for training and block GPTBot or Google-Extended while still allowing the search crawlers. That split is possible precisely because the companies separate the agents. What you want to avoid is blocking by accident, or blocking a search crawler while you are hoping to appear in that tool's answers.

Check the layer below robots.txt

A robots.txt file can say 'allow' while your hosting provider or security service refuses the request anyway. Bot protection products, firewalls and some CDN settings challenge or block unfamiliar crawlers by default. The crawler never gets past the challenge page, and nothing in your analytics tells you.

If you use a service like this, look for a setting covering verified or known bots, and review whether AI crawlers are included. Your server logs, or your host's log viewer, will show requests from these user agents and the status codes they received. A run of 403 responses is a clear sign something is in the way.

Put the words in the HTML

Many crawlers read the HTML your server sends and do little or no JavaScript processing. If your service descriptions, prices, hours or reviews only appear after a script runs in the browser, a crawler may see an empty shell. Googlebot does render JavaScript, but not every AI crawler is documented as doing so, and it is safer not to depend on it.

A quick test: view the page source (not the inspector, the raw source) and search for a sentence you can see on the page. If it is missing, that text is being added by JavaScript. Content inside tabs, accordions or 'read more' toggles is usually fine as long as it is present in the source.

llms.txt: useful, but not a switch

llms.txt is a proposed convention: a plain text file at the root of your site that summarises what the site is and links to its most useful pages, written for language models. It is cheap to add and harmless. It is also a community proposal rather than an adopted standard, and there is little public confirmation that the major AI tools read it when answering questions.

If you publish one, keep it accurate and short, and make sure it agrees with your homepage. Do not treat it as a substitute for readable pages and an open robots.txt.

Then watch what changes

Fixing a blocked crawler does not change answers overnight. Crawlers revisit on their own schedules, and models update on theirs. Found in AI needs nothing installed on your site; it reads what the tools answer. Run your tracked questions before and after a fix, and look at the trend rather than a single scan.