llms.txt
Whether it exists and is well formed
That it starts with a # title and has a summary and links to the key pages. We flag what’s missing and show you the file. We also check whether there’s an llms-full.txt.
Free tool · AI
llms.txt checker and AI crawler test
Is your website ready for AI? This llms.txt checker tells you whether a site has an llms.txt and which assistants can read it according to its robots.txt: the 17 crawlers from ChatGPT, Claude, Gemini, Perplexity and seven other services, with the rule that decides it.
Free · No sign-up · We don’t store the result
01
How to use the llms.txt checker
- 01
Paste the domain
Type the site’s address (the domain is enough, for example
yoursite.com). We read its root files:llms.txt,robots.txtand the sitemap. - 02
Look at the llms.txt
If it exists, you see its title, its summary, how many sections and links it has and any formatting warnings. You can also open the file as is.
- 03
Review the crawlers
A table with 17 AI crawlers: whether they can get in, whether they’re blocked and which robots.txt rule decides it.
02
What the llms.txt validator checks
robots.txt
17 AI crawlers
GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Meta-ExternalAgent and the rest: which ones can get in, which can’t and which rule decides it.
Sitemap
The site map
Whether robots.txt declares a sitemap and how many URLs it has. It’s what search engines, and many assistants, use to discover your pages.
03
AI crawlers: training isn’t searching
Training
GPTBot, ClaudeBot, CCBot…
They collect text to train models. Blocking them doesn’t remove you from the assistants’ search results, because search uses other crawlers.
Search
OAI-SearchBot, Claude-SearchBot, PerplexityBot
They index pages so the assistant can cite them in its answers. If you block them, you’re less likely to be recommended.
User visits
ChatGPT-User, Claude-User, Perplexity-User
They come in when someone asks the assistant to read your page. Blocking them stops it from reading a link it’s given.
04
What llms.txt is and how to create one
llms.txt is a Markdown text file that goes in the root of your website (yoursite.com/llms.txt). It sums up what the site is and links the pages that explain it best, so an AI assistant doesn’t have to guess among menus, banners and code. Jeremy Howard proposed it in September 2024 at llmstxt.org. It isn’t an official standard: some assistants read it and others don’t.
An example:
# Example Studio
> Web design studio in Barcelona. We build custom websites for small businesses and online stores.
## Services
- [Web design](https://example.com/web-design/): custom websites at a fixed price
- [Online stores](https://example.com/online-stores/): WooCommerce and Shopify
- [Pricing](https://example.com/pricing/)
## Optional
- [Blog](https://example.com/blog/)- The first line,
# Name, is the only required part. - A line starting with
>holds the summary: what you do and for whom, in one or two sentences. ##sections hold lists of links- [Title](url): note. The note is optional.- The
## Optionalsection groups secondary content: an assistant short on space can skip it.
Some sites also publish an llms-full.txt with the full text of their pages in a single file.
llms.txt and robots.txt don’t do the same job
robots.txt says who can get in and where; llms.txt doesn’t block or allow anything, it only guides. If an AI crawler is blocked in robots.txt, a good llms.txt won’t open the door for it. That’s why this tool checks both at once.
05
Why check llms.txt and robots.txt together
Many validators only check the llms.txt format. That’s of little use if robots.txt blocks the crawlers that were going to read it, and it happens more often than you’d think: a Disallow: / meant for something else, or a rule copied from another site, shuts them all out. If you want to block GPTBot in robots.txt on purpose, this table also shows you whether the rule works.
Here you see, crawler by crawler, which rule decides: a group with its name (User-agent: GPTBot) or the general rule (User-agent: *). And if the site has no robots.txt, we tell you: then every crawler can get in.
Check llms.txt, robots.txt and the rest of the SEO of every site you visit, without copying addresses.
Join the waitlist06
What the llms.txt checker can’t know
Whether AI reads it
robots.txt is a request
Reputable crawlers respect robots.txt, but it’s a courtesy rule: one that ignores it gets in anyway. And there’s no way to tell from the outside which assistant has read your llms.txt.
Server-side blocks
Firewalls and CDNs
A firewall or a CDN can block bots without touching robots.txt. Those blocks don’t show up in the files: you see them in your hosting or CDN dashboard.
New crawlers
We check 17 names
The ones from the most widely used assistants and models. New crawlers appear every few months; the list is updated along with the extension.
07
More free tools
WordPress
WordPress theme and plugin detector
Which theme, plugins and page builder a site uses, with their version and the clue that gives them away.
Analyze a website →SEO
SERP simulator: Google and social preview
How a page looks on Google and when shared on Facebook, X, LinkedIn and WhatsApp, with the title measured in pixels.
See the preview →Chrome extension
SEO inspector for Chrome: robots.txt, sitemap, llms.txt and more
The same, and much more, on the page you have open.
See the extension →Smart Design Inspector
This and much more, on any website
The Chrome extension does it on the page you have open, with no copying and pasting: tech stack, SEO, fonts, colors and CSS, and it sees the page the way your browser renders it (here we only read the HTML the server sends).
What is the llms.txt file?
A text file in the root of a website (yoursite.com/llms.txt) that sums up what the site is and links its most important pages, in Markdown, so AI assistants understand it better. It’s a 2024 proposal, not an official standard.
Is llms.txt required?
No. Neither Google nor the assistants require it, and it doesn’t affect your Google rankings. It takes little effort and lets you set which pages you want AI to cite.
How do I know if a site blocks ChatGPT or Claude?
Paste its address above. The tool reads its robots.txt and tells you, crawler by crawler, whether it can get in, whether it’s blocked or whether only some paths are closed, and which rule decides it.
How do I block GPTBot in robots.txt?
Add two lines to your robots.txt: “User-agent: GPTBot” and “Disallow: /”. That only stops OpenAI’s training; ChatGPT search uses OAI-SearchBot and user-requested visits use ChatGPT-User, which you block separately.
Does Google-Extended affect my Google rankings?
No. Google-Extended only decides whether Google can use your content for Gemini and for training. Google Search crawling is done by Googlebot, which is a different crawler.
Is it free?
Yes, free and with no sign-up. It’s a sample of the Smart Design Inspector extension, which checks this on the website you have open along with the rest of its SEO.
Do you store the websites I check?
No. The files are read on the spot to give you the result, and neither the address nor the result is stored.
Waitlist
Be one of the first to try it
Smart Design Inspector is in development. Leave your email and we’ll let you know the day it goes live on the Chrome Web Store, with the 7 Pro features free in early access.
One email on launch day and very few more · Unsubscribe in one click