How do you get ChatGPT, Google’s AI answers or Microsoft Copilot to mention your website? You can’t force it, and nobody can promise it. What you can do is make sure these tools can reach your pages, can read them easily, and find something on them worth quoting. This guide walks you through eleven checks in plain English.
You don’t need to be technical to follow it. I explain any jargon the first time it appears.
Key takeaways
- OpenAI, Anthropic, Perplexity and Google each run their own crawlers, programs that visit web pages automatically and read them. OpenAI says a site that opts out of its search crawler, OAI-SearchBot, won’t appear in ChatGPT search answers (OpenAI, 2026).
- For Google, a page has to be indexed (stored in Google’s list of pages) and eligible to show with a snippet (a short extract). Google says there’s no special markup or file to add for its AI features, and that meeting its requirements doesn’t guarantee your page will be shown (Google, 2026).
They don’t all work the same way. Each company runs its own crawlers. Usually there’s one for search, one for training (using your content to teach its AI models), and one that fetches a page when a person asks a question. Blocking one doesn’t block the others, so it helps to know which is which.
| Company | Search crawler | Training crawler | Fetches a page when a person asks |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User. OpenAI says robots.txt rules may not apply to it. |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | None. Perplexity says PerplexityBot isn’t used for AI training. | Perplexity-User. Perplexity says it generally ignores robots.txt. |
| Googlebot, which also covers its AI features | Google-Extended, a robots.txt token that controls whether content is used for training Gemini models and for grounding. Google says it doesn’t affect Search (Google, 2026). | Google lists separate user-triggered fetchers, such as Google Read Aloud. Google says these generally ignore robots.txt (Google, 2026). |
Google works differently. Its AI Overviews (the AI-written summaries at the top of some searches) and AI Mode show links to supporting websites, and a page must be indexed and eligible to be shown in Google Search with a snippet to be one of them. Googlebot is Google’s crawler for Search, and it follows the rules in your site’s robots.txt file, which is a small file that tells crawlers which parts of your site they may visit. Google says robots.txt directives for Googlebot are how site owners manage how their sites are crawled for Search (Google, 2026). Google also says AI Overviews are only shown when its systems determine they are additive to classic Search.
Google recommends allowing crawling in robots.txt and through any CDN or hosting layer. A CDN is a network that sits between your site and its visitors to speed things up.
OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP addresses, which are the addresses its crawlers visit from (OpenAI, 2026). Perplexity says the same about PerplexityBot (Perplexity, 2026). Anthropic says blocking its crawlers by IP address may not work correctly, because it can stop a crawler reading your robots.txt file (Anthropic, 2026). OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust, and Perplexity says it may take up to 24 hours for its systems to reflect changes.
Some services make these choices for you. Cloudflare announced changes to its AI crawler controls on 15 September 2026. It sorts crawlers into Search, Agent and Training. Its “Block” setting now also stops mixed-use crawlers such as Googlebot and Bingbot (Bing’s crawler), so it can stop them reaching your site, search included, as well as blocking AI training. If you want to stay in search while opting out of training, Cloudflare says to choose “Disallow AI Training” instead. It also says existing customers’ settings carry over on their own (Cloudflare, September 2026).
OpenAI says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, because each of its settings is independent (OpenAI, 2026).
WordPress has a setting called “Discourage search engines from indexing this site”, under Settings, then Reading. When it’s ticked, WordPress adds a “noindex, nofollow” instruction to your pages, which asks search engines to ignore your site (WordPress.org, 2024).
“Noindex” is a small tag that asks search engines to leave a page out of their listings.
Also look for snippet limits. Google says that to limit what’s shown in its AI features you can use nosnippet, data-nosnippet, max-snippet or noindex, which are tags that restrict how much of a page appears in results (Google, 2026).
Google says a page has to be indexed and eligible to appear in Google Search with a snippet before it can be considered for its AI features. It also recommends verifying your site in Search Console (Google, 2026). Search Console is Google’s tool that shows how Google sees your site. Bing Webmaster Tools is Bing’s version.
“Indexed” simply means the search engine has stored the page so it can be shown. The Page indexing report in Search Console gives the reason for any page that isn’t indexed (Google, Page indexing report, 2026).
Google says to keep important content in text form, and to support it with images or video where that helps (Google, 2026). Microsoft says to avoid putting key information only in images. AI systems can sometimes read text in images, but it adds complexity and often reduces accuracy. Microsoft says to always provide alt text, a short written description of each image, or present critical details in HTML, which means ordinary text on the page (Microsoft Advertising, October 2025).
In practice, that means writing your services, your prices and your answers out as text, not only as images.
To see what Google actually receives for a page, paste the page address into the URL Inspection tool in Search Console. It shows the HTML, which is the page’s underlying code, that Googlebot received (Google, 2026).
Google says it can be helpful to have a unique viewpoint that stands out, such as a first-hand review based on personal experience, and that a summary of existing content simply restates information already available elsewhere (Google, 2026).
Microsoft recommends clear questions with direct, self-contained answers, because assistants can often lift a question and its answer straight into a response (Microsoft Advertising, October 2025).
Google says generating many pages mainly to manipulate AI responses can break its scaled content abuse policy, its rule against churning out pages to game the results (Google, 2026).
Microsoft says AI systems break content into smaller pieces, and it recommends sentences that make sense even when pulled out of context. Use descriptive H2 and H3 headings (the section titles on a page), lists for steps, and tables for comparisons, without overusing lists (Microsoft Advertising, October 2025).
Google says it’s generally a good idea to use semantic HTML when possible, meaning page code that labels what is a heading, a list and so on. It says this helps users such as screen readers parse and navigate a page more easily (Google, 2026).
Google recommends internal links, which are links between your own pages, so your content is easy to find (Google, 2026). Microsoft says backlinks, which are links from other websites to yours, remain part of the basics too (Microsoft Advertising, October 2025).
Google says seeking inauthentic mentions of your business across the web isn’t as helpful as it might seem (Google, 2026).
Google says that if your business information isn’t accurate, your profile might not show up for relevant searches (Google, 2026).
Microsoft describes the content it wants as clear, current, comprehensive content that AI systems can surface with confidence (Microsoft Advertising, October 2025).
Don’t change a date just to look fresh. Google lists that among the warning signs of content written for search engines first (Google, Creating helpful content, 2026).
Structured data is code that labels what’s on a page, such as a business, an article or a question, so a machine can tell what each part is. Google says it isn’t required for its AI features, and that there’s no special schema.org markup for them. If you use it, Google says to make sure it matches the visible text on the page (Google, 2026).
Microsoft recommends schema markup, usually written in a format called JSON-LD, so systems can label content such as products, reviews, FAQs and events (Microsoft Advertising, October 2025).
Bing Webmaster Tools has an AI Performance report. It shows where your content is cited in AI-generated answers in Microsoft Copilot, Bing and select partner experiences, and the grounding queries behind those citations. Bing describes grounding queries as the key phrases the AI used when retrieving content that was referenced in AI-generated answers. It launched in public preview on 10 February 2026 (Bing Webmaster Blog, February 2026).
Microsoft says its Citation Share figure doesn’t represent traffic share (Bing Webmaster Blog, June 2026). Google says traffic from AI Overviews and AI Mode is counted in the standard Web results in the Search Console Performance report (Google, 2026).
Google says clicks from search results pages with AI Overviews tend to be higher quality, meaning users are more likely to spend more time on the site (Google, 2026). But Ahrefs, looking at desktop click data for 300,000 keywords, found that an AI Overview now correlates with a 58% lower average click-through rate for the top-ranking page, comparing December 2023 with December 2025. Ahrefs calculates the 58% against a predicted click-through rate for what would have happened without AI Overviews (Ahrefs, February 2026). I explain the wider shift in why your website traffic dropped.
Google’s guidance says you can ignore the following.
| Term | What it means |
|---|---|
| Crawler | An automated visitor that reads web pages. |
| robots.txt | A file on your site that tells crawlers which parts they may visit. |
| Index | The search engine’s stored copy of your pages, used to show results. |
| Snippet | The short extract shown with a search result. |
| noindex | An instruction asking search engines not to store a page. |
| Structured data | Code that labels what’s on a page, such as a business or a question. |
| Citation | When an AI answer names or links your page as a source. |
| Grounding query | The key phrases the AI used when retrieving content that was referenced in an AI answer, as Bing describes them in its AI Performance report. |
| Check | What good looks like |
|---|---|
| 1. Crawlers | Crawling is allowed in robots.txt and through any CDN or hosting layer. |
| 2. Site settings | The WordPress search visibility box is unticked, and there are no accidental noindex or snippet limits. |
| 3. Indexing | Key pages show as indexed in Search Console and Bing Webmaster Tools. |
| 4. Readable text | Services, prices and answers are written as text, and the text appears in the page source. |
| 5. Original content | Content with a unique point of view, such as first-hand experience, not just a summary of what’s already out there. |
| 6. Customer questions | Clear questions with direct, self-contained answers. |
| 7. Structure | Clear headings, lists for steps and tables for comparisons. |
| 8. Links and details | Internal links make your content easy to find, and your business information is accurate. |
| 9. Accuracy | Content is current, and dates are not changed just to look fresh. |
| 10. Structured data | If used, it matches the visible text on the page. |
| 11. Measuring | Bing’s AI Performance report and Search Console’s Performance report checked. |
No. Google says that meeting its requirements doesn’t guarantee crawling, indexing or serving (Google, 2026).
No. OpenAI says each crawler is a separate setting. OAI-SearchBot controls whether you appear in ChatGPT search, and GPTBot relates to training its models (OpenAI, 2026).
Not for Google. It says Search doesn’t use llms.txt files, and that it won’t affect your Search visibility either way (Google, 2026).
Google says there are no extra technical requirements for AI Overviews or AI Mode beyond being indexed and eligible to show in Search with a snippet (Google, 2026).
Bing Webmaster Tools’ AI Performance report shows citations across Copilot, Bing and select partner experiences (Bing Webmaster Blog, June 2026).
Not on its own. OpenAI, Anthropic and Perplexity each describe separate crawlers for search and for training. Anthropic does warn that blocking its search crawler, Claude-SearchBot, or its user crawler, Claude-User, may reduce your visibility in those results (Anthropic, 2026).
Cloudflare says to choose “Disallow AI Training”, not “Block”. Block now applies to mixed-use crawlers such as Googlebot and Bingbot, so it can stop them reaching your site at all, search included (Cloudflare, September 2026).
If you’d rather have one person check your site against this list and fix what’s wrong, get in touch.

SushiWP builds and manages websites for small businesses across Hampshire and the UK.
Clear, jargon-free, and built to get you found and keep the enquiries coming.