AI Visibility Checklist for Your Website: How to Make Sure Your Website Is Visible to AI Crawlers

In my research, I've found eight checks that show whether ChatGPT, Google, Claude, Perplexity, and other AI systems can reach and read your website. This page lists each check with the tool or step to test it. Run them in order, since a crawler blocked by the first two never reaches the rest.

TL;DR

Run these eight checks to see whether web crawlers can reach, read, and index your site:Robots.txt must be set to allow web crawlers to crawl your site.Your Content Delivery Network (CDN) cannot be blocking web crawlers.Your pages must be indexed and allowed to show a snippet.JSON-LD should label the content on your site.JavaScript rendering or interactive elements shouldn't hide content crawlers need to read.Headings and paragraphs should be structured so each section makes sense on its own.Your server logs should show AI crawlers visiting.llms.txt is optional, since adoption by the LLMs is not consistent.

How to Check Robots.txt

Robots.txt is a file on your website that contains rules to allow or block specific web crawlers (robots or bots) from accessing your website (read more from Google). Each AI company runs separate crawlers for search, training, and user requests, and robots.txt sets rules for each one separately.

You can see which web crawlers have access to your site by opening your robots.txt file.

• Open a browser and go to the URL https://yourwebsite/robots.txt
• Look for:

User-Agent: *
Allow: /

• Or look for specific website crawlers set as allowed or disallowed. Start with the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot. The Related Questions section lists each crawler by name.

Search crawlers decide whether your site can appear in AI search answers. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI crawler docs). Anthropic says disabling Claude-SearchBot may reduce a site's visibility in Claude search results (Anthropic). For Google, the robots.txt rules for Googlebot control how your site is crawled for Search, which includes AI Overviews and AI Mode (Google).

After you change robots.txt, OpenAI says its search systems can take about 24 hours to adjust.

How to Check CDN (Content Delivery Network)

Your CDN might be set to block specific crawlers, even when robots.txt allows them. A CDN has servers in different geographic areas that store website files so they can load faster. Popular CDNs are Cloudflare and Amazon Web Services. Some CDNs block AI bots by default.

You can check whether your CDN is blocking bots by using tools like aicrawlercheck.com, crawlercheck.com, or by checking your CDN's logs and settings.

If your CDN has bot or AI crawler settings, allow the search crawlers. Google lists robots.txt and CDN or hosting infrastructure as the places where crawling must be allowed (Google). OpenAI recommends allowing OAI-SearchBot and its published IP ranges. Perplexity recommends allowing PerplexityBot and Perplexity-User by user agent and published IP address in your firewall (Perplexity docs). Cloudflare customers can set an AI bot policy in the Cloudflare bot settings.

How to Check That Your Pages Are Indexed and Can Show a Snippet

Google requires a page to be indexed and eligible to show a snippet before the page can appear as a supporting link in AI Overviews or AI Mode. Google lists no additional technical requirements (Google).

You can check both in Google Search Console:

• Open the URL Inspection tool and enter the page URL.
• Confirm the page is indexed.
• View the page's HTML and search it for these tags:

<meta name="robots" content="noindex">
<meta name="robots" content="nosnippet">

A noindex tag keeps the page out of Google's index. A nosnippet tag limits what Google shows from the page. Remove either one from pages you want cited. Also look for data-nosnippet and max-snippet, which limit snippets too (Google's robots meta tag documentation).

How to Check for JSON-LD

JSON-LD (JavaScript Object Notation for Linked Data) is code added to the HTML on webpages on your site. It labels your content and tells web crawlers what's on the page.

You can check whether your site has JSON-LD by going to:

• validator.schema.org

Google says structured data should match the visible text on the page, and that no special schema is required to appear in AI Overviews or AI Mode (Google). iPullRank's AI Search Manual recommends schema types that match your content, such as Organization, Person, Article, FAQPage, and Product, and warns that markup applied to content that isn't on the page can be ignored (iPullRank, Chapter 9).

How to Check for JavaScript Rendering or Interactive Elements

JavaScript is code on a website that makes pages interactive. But some web crawlers don't run that code, which means content that depends on JavaScript running is invisible to them. For example, if you have important FAQ information in dropdowns or accordions on a webpage, that information might be invisible to the bots.

Google runs JavaScript when it crawls, but Google's documentation says not all bots can, and it recommends server-side or pre-rendering for that reason (Google JavaScript SEO basics). Google also recommends making sure important content is available in text form (Google).

You can run a quick test in Chrome:

• Open your page and view the page source (Ctrl+U on Windows, Cmd+Option+U on Mac).
• Search the source for a sentence from the page.
• If the sentence is missing from the source, the page loads it with JavaScript.

You can use these tools to check for problems preventing web crawlers from seeing content on your site:

• llmrefs.com's AI Crawl Checker
• Google's Lighthouse Chrome Extension
• The rendered HTML in Google's URL Inspection tool

How to Check Your Page Structure

AI systems pull sections of a page, such as a paragraph, a list, or a table row, to build answers, so each section needs to make sense on its own. iPullRank's AI Search Manual describes this as semantic chunking (iPullRank, Chapter 9).

You can check a page against this list:

• Headings follow a clear order: one H1, then H2s for each section.
• Each section opens with its answer.
• Each paragraph covers one idea.
• Each section names its subject instead of starting with "this" or "it."
• Facts include specific numbers and full dates.
• Comparisons sit in tables.

Read one section by itself. If a stranger can't tell what the section is about from its first sentence, rewrite that sentence.

How to Check Server Logs for AI Crawler Visits

Your server logs show whether AI crawlers visit your site, because each request records the crawler's name. iPullRank's AI Search Manual lists server log analysis as a way to track AI bot activity (iPullRank, Chapter 13).

• Ask your web host, CDN, or developer for your access logs.
• Search the logs for the crawler names:

grep -i -E "OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot" access.log

• Check that the visits are real by comparing the IP addresses to the published lists from Anthropic, OpenAI, Perplexity, and Google.

A crawler that appears in your logs with 403 or 429 responses is being blocked by your CDN or firewall. A crawler that never appears might be blocked in robots.txt, or it might not have visited yet.

How to Check for llms.txt

llms.txt is a proposed standard text file for your website that tells LLMs and AI agents what your website is about. Adoption by the LLMs is not consistent, but there's no penalty for adding it. Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode (Google).

You can check whether your site has an llms.txt file by opening a browser window and entering your site with llms.txt at the end, like the example below:

• Open https://yourwebsite/llms.txt

If you add one, the H1 with your site name is the only required section. A minimal file looks like this (llmstxt.org):

# Your Site Name

> One or two sentences on what your website is about.

## Key Pages

- [Page title](https://yourwebsite/page): What the page covers

Chrome's Lighthouse includes an llms.txt check in its agentic browsing audits (llmstxt.org).

Related Questions

Which web crawlers should have access to my site and should be listed in my robots.txt file?

You can allow all web crawlers to access your site by setting the User-Agent to User-Agent: *.

Here's a list of specific bots from OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Google, and others:

If GPTBot is blocked but OAI-SearchBot isn't, does that stop training but still let ChatGPT cite the site in search?

Yes. GPTBot is training, OAI-SearchBot is search, and ChatGPT-User is the live browse when a user's question sends ChatGPT to a specific page. Blocking one doesn't block the others. Check robots.txt for OAI-SearchBot and GPTBot separately. A robots.txt that blocks training and allows search looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The site loads fine in a browser. Why would a CDN block a bot at all?

Because bot protection is often set to be defensive by default. Not all bots are useful, and some can be harmful. You want to let the good bots in and keep the ones you don't want out.

If I block ChatGPT-User in robots.txt, does ChatGPT stop visiting my pages?

It depends on the company. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user's question starts each visit. Perplexity says Perplexity-User generally ignores robots.txt rules for the same reason. Anthropic says disabling Claude-User in robots.txt prevents Claude from retrieving your content in response to a user query (OpenAI, Perplexity, Anthropic).

How long does it take for ChatGPT search and Perplexity to see a robots.txt change?

OpenAI says ChatGPT search can take about 24 hours to adjust after a robots.txt update. Perplexity says changes can take up to 24 hours to apply.

Do I need llms.txt to show up in Google's AI Overviews?

No. Google says there are no additional requirements to appear in AI Overviews or AI Mode, and that you don't need to create new machine readable files, AI text files, or markup (Google).

More checklists like this go out as I write them.

Subscribe to Search Seeker