A plain markdown file at the root of your site that gives language models a curated map of it: a title, a one-line summary, and lists of the pages actually worth reading. It exists because a model reading your rendered HTML gets navigation, cookie banners and boilerplate along with the content. It is a proposal rather than a ratified standard, and support varies, but it costs nothing to publish.

How is it different from sitemap.xml?

A sitemap is exhaustive and machine-oriented: every URL, with timestamps, for crawlers deciding what to fetch. llms.txt is selective and human-written: the handful of pages you would point someone at, with a sentence explaining each. One is an index, the other is a recommendation.

How do I stop AI companies training on my site?

Disallow the training crawlers in robots.txt. GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini, Applebot-Extended for Apple Intelligence, CCBot for Common Crawl, and meta-externalagent for Meta. The block-training preset here selects exactly those and leaves the rest alone.

Will blocking AI crawlers hurt my search ranking?

Not if you block the right tokens. Google-Extended is a training opt-out signal with no crawler behind it, so disallowing it removes you from Gemini training and leaves Google Search untouched. Disallowing Googlebot is what would remove you from Search, and the two are easy to confuse. The same split exists at OpenAI, where GPTBot trains and OAI-SearchBot indexes for ChatGPT search.

Should I block AI search crawlers too?

Usually not. AI search crawlers index your pages so answers can cite and link to you, which sends readers back. Training crawlers take the content with no return path. Blocking both is a real choice, but it is a different one from opting out of training, and this tool warns when your selection crosses that line.

Does robots.txt actually stop anyone?

It stops the well-behaved. robots.txt is a request that operators choose to honour, not an access control, and a scraper that ignores it faces no technical barrier. If you need enforcement rather than a request, that belongs at your CDN or firewall, blocking by user agent and IP.

Where do these files go?

Both at the root: yoursite.com/robots.txt and yoursite.com/llms.txt, served as plain text. On most frameworks that means dropping them in the public or static folder.

llms.txt and robots.txt Generator

Write both files that tell machines what to do with your site. Which crawler trains, which one indexes for AI search, and which one is fetching because a person asked, are three different things, and blocking each costs you something different.

Your site

Site URL

Sitemap path

Written out as an absolute URL, which is what the format requires.

Paths to keep crawlers out of

AI crawlers

GPTBotOpenAITrains models
Collects pages to train future models. Blocking costs you nothing in ChatGPT search.
OAI-SearchBotOpenAIAI search index
Indexes for ChatGPT search results. Block this and you stop appearing there.
ChatGPT-UserOpenAIFetches on request
Fetches a page because someone asked ChatGPT to open it. Blocking breaks that link for real users.
ClaudeBotAnthropicTrains models
Collects pages for training. The one to disallow if you want out of Claude's training data.
Claude-SearchBotAnthropicAI search index
Indexes for Claude's search results.
Claude-UserAnthropicFetches on request
Fetches a page on a person's request, the same way ChatGPT-User does.
Google-ExtendedGoogleTrains models
A training opt-out token, NOT a crawler. Disallowing it keeps you out of Gemini training and leaves Search ranking untouched.
PerplexityBotPerplexityAI search index
Indexes for Perplexity answers, which cite and link back.
Perplexity-UserPerplexityFetches on request
Fetches a page a person asked for.
Applebot-ExtendedAppleTrains models
Apple Intelligence training opt-out. Separate from Applebot, which powers Siri and Spotlight and is worth keeping.
CCBotCommon CrawlTrains models
Common Crawl feeds many models indirectly. Blocking it is the widest single training opt-out.
meta-externalagentMetaTrains models
Meta's AI training crawler.
BytespiderByteDanceTrains models
ByteDance's crawler. Widely reported to be aggressive about rate.
AmazonbotAmazonMixed
Feeds Alexa and Amazon's AI work.

No AI-specific rules. Every crawler is treated like any other.

llms.txt content

Site name

One-line summary

Becomes the blockquote. The line a model is most likely to quote.

robots.txt

# robots.txt
# A request, not a fence. Well-behaved crawlers honour it.

User-agent: *
Disallow: /admin
Disallow: /api

Sitemap: https://example.com/sitemap.xml

Both belong at the site root: /robots.txt and /llms.txt.

Checks

Looks sound

llms.txt has no site name, so the H1 will read as a placeholder.
No summary line. That blockquote is the one sentence a model is most likely to use.

Both files are written in your browser. Nothing about your site is sent anywhere.

Describe the site

Your URL, your sitemap, and any paths crawlers should skip.

Choose a policy

Allow everything, opt out of training only, or pick crawler by crawler.

Ship both files

Copy or download robots.txt and llms.txt, then serve them from the root.

Questions

What is llms.txt?: A plain markdown file at the root of your site that gives language models a curated map of it: a title, a one-line summary, and lists of the pages actually worth reading. It exists because a model reading your rendered HTML gets navigation, cookie banners and boilerplate along with the content. It is a proposal rather than a ratified standard, and support varies, but it costs nothing to publish.
How is it different from sitemap.xml?: A sitemap is exhaustive and machine-oriented: every URL, with timestamps, for crawlers deciding what to fetch. llms.txt is selective and human-written: the handful of pages you would point someone at, with a sentence explaining each. One is an index, the other is a recommendation.
How do I stop AI companies training on my site?: Disallow the training crawlers in robots.txt. GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini, Applebot-Extended for Apple Intelligence, CCBot for Common Crawl, and meta-externalagent for Meta. The block-training preset here selects exactly those and leaves the rest alone.
Will blocking AI crawlers hurt my search ranking?: Not if you block the right tokens. Google-Extended is a training opt-out signal with no crawler behind it, so disallowing it removes you from Gemini training and leaves Google Search untouched. Disallowing Googlebot is what would remove you from Search, and the two are easy to confuse. The same split exists at OpenAI, where GPTBot trains and OAI-SearchBot indexes for ChatGPT search.
Should I block AI search crawlers too?: Usually not. AI search crawlers index your pages so answers can cite and link to you, which sends readers back. Training crawlers take the content with no return path. Blocking both is a real choice, but it is a different one from opting out of training, and this tool warns when your selection crosses that line.
Does robots.txt actually stop anyone?: It stops the well-behaved. robots.txt is a request that operators choose to honour, not an access control, and a scraper that ignores it faces no technical barrier. If you need enforcement rather than a request, that belongs at your CDN or firewall, blocking by user agent and IP.
Where do these files go?: Both at the root: yoursite.com/robots.txt and yoursite.com/llms.txt, served as plain text. On most frameworks that means dropping them in the public or static folder.

All labs

Pairs with

The catalog behind this tool: real, published entries with verified pricing and licensing, not a list of affiliate links.

AI modelsThe models these crawlers feed.Meta Tag GeneratorThe tags on each page.InfrastructureWhere real blocking happens.