- Home
- Skills
- Marketing & SEO
- llms.txt and AI Crawler Check
More screenshots
Works with the AI Agents you already use
llms.txt and AI Crawler Check
Check llms.txt and the AI crawler rules in robots.txt before you publish: missing H1, localhost or staging links, pages robots.txt blocks for every AI crawler, several bots on one User-agent line, misspelled bot names.
$5
llms.txt and AI Crawler Check
Example session with this skill installed
Check this project with llms-txt-ai-crawler-check.
- Read your context and instructions
- Compiled the llms txt
$ python3 scripts/check_llms_txt.py examples/risky
BLOCK docs-portal/static/llms.txt:1 L01 LLMS_TXT_WITHOUT_H1
## API v3 reference
Why: llms.txt has no H1 title (# Name) before its first
section. The llms.txt format makes the H1 with the site or project name its only required part, and tools that read the format start from it to find the summary and the link sections. A file that starts with a section, plain text or an HTML page is not an llms.txt file.
Safer: Start the file with one H1 line naming the site or project (# Acme Docs), then an optional > summary line, then
sections of - name: notes links. If the file is an HTML page, make the server return the plain Markdown file instead.
BLOCK shop/public/llms.txt:9 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
BLOCK shop/public/llms.txt:10 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
BLOCK shop/public/llms.txt:11 L02 LOCAL_OR_STAGING_LINK
BLOCK shop/public/llms.txt:12 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
BLOCK shop/public/llms.txt:16 L02 LOCAL_OR_STAGING_LINK
BLOCK shop/public/llms.txt:22 L05 SECRET_IN_LLMS_LINK
BLOCK shop/public/robots.txt:8 L04 USER_AGENT_LINE_NOT_ONE_NAME
REVIEW docs-portal/static/llms.txt L18 LARGE_LLMS_FILE
REVIEW shop/public/llms.txt:12 L16 DUPLICATE_LINK
REVIEW shop/public/llms.txt:17 L17 HTTP_LINK
REVIEW shop/public/llms.txt:18 L15 FILE_LIST_ITEM_INCOMPLETE
REVIEW shop/public/robots.txt:11 L12 AI_AGENT_NAME_NOT_MATCHED
REVIEW shop/public/robots.txt:14 L11 AI_SEARCH_AGENT_BLOCKED
REVIEW shop/public/robots.txt:17 L13 AI_GROUP_DROPS_DEFAULT_RULES
REVIEW shop/public/robots.txt:17 L14 USER_FETCHER_MAY_IGNORE_ROBOTS
(Each finding also has Why and Safer lines, as in the first one.)
Verdict: BLOCK. Checked 3 files, 3 with llms.txt or robots.txt content.
This is a static check of file text. NOT FLAGGED is not a guarantee.
Connects securely to your tools. The creator never sees your data.
About this skill
User-agent: GPTBot, ClaudeBot looks like it blocks two AI crawlers, but Google's parser reads only the first name and stricter parsers match neither. An llms.txt that lists your /docs/.md pages while robots.txt says Disallow: /.md$ to everyone advertises pages that AI tools are told to skip, and a group that only says User-agent: ClaudeBot plus Crawl-delay: 1 quietly opens every path your * group closes. This skill finds those and the other llms.txt and AI crawler mistakes listed below before you publish, and tells you how to fix each one.
Your agent runs a scanner on every llms.txt, llms-full.txt and robots.txt in your project's source folders (public/, static/, .well-known/ and the like). Build output folders are skipped in a project scan, so for an llms.txt that a Docusaurus, VitePress or Astro plugin generates, your agent builds the site and scans the output folder too. It reads llms.txt the way the llms.txt format describes it (H1 title, summary, sections of name: notes links), skipping code blocks and front matter, and reads robots.txt the way RFC 9309 describes it: one group per crawler, the longest matching rule wins, the * group only for crawlers no group names. When llms.txt and robots.txt sit in the same web folder (or Next.js app/robots.txt next to public/), it checks each llms.txt link against robots.txt for the AI crawler tokens of OpenAI, Anthropic, Google, Perplexity, Common Crawl, Apple, ByteDance and Meta. It never fetches your site. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach, and each rule links to the llms.txt format, RFC 9309 or the crawler operator's page behind it. Secrets are never printed.
What it checks
BLOCK
- L01: llms.txt without its H1 title (it starts with a section, plain text or an HTML page, or is empty)
- L02: file-list links to localhost, private IP addresses, .local/.test/.internal names, staging hosts, Netlify deploy previews, Vercel branch URLs, ngrok or Cloudflare tunnels, or placeholder domains
- L03: llms.txt links (or llms.txt itself) that robots.txt disallows for every known AI crawler and fetcher
- L04: several AI crawler names on one User-agent line, or a full browser user-agent string
- L05: links in llms.txt with a password, API key, token, JWT or signature in the URL, and real-looking passwords in database URLs in code examples
REVIEW
- L11: AI search and answer agents (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, meta-webindexer) blocked from the whole site while llms.txt is published
- L12: misspelled, versioned or outdated AI crawler names (GPT-Bot, Google-Extend, GPTBot/1.1, ChatGPT, anthropic-ai without ClaudeBot)
- L13: AI crawler groups that do not repeat the * group's Disallow rules, so those paths are open to them
- L14: blocks on ChatGPT-User, Perplexity-User or meta-externalfetcher, which their operators say may not follow robots.txt
- L15: list items without a link in an llms.txt section of links, or many links and none with a description
- L16: the same URL listed twice in llms.txt
- L17: http:// links in llms.txt
- L18: llms.txt over 100,000 characters or llms-full.txt over 1 MB
What's included
- The scanner (Python 3.9+, standard library only)
- A rules guide with the reason, safer approach and official source link for each rule
- A risky and a safe example project (a shop and a docs portal), and the real report the scanner produces on the risky one
Limits It reads the files as written. Build output folders (build/, dist/, .next/ and similar) are skipped when it scans a project; scan the output folder directly to check a generated llms.txt. robots.txt generated at build time (such as Next.js app/robots.ts), CDN or host settings and X-Robots-Tag headers are not read, and links are not fetched, so broken pages are not found. Whether a crawler obeys robots.txt is up to its operator, and new AI crawlers appear often. Classic search-engine rules in robots.txt are out of scope. NOT FLAGGED is not a guarantee.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Marketing & SEO.
Static check of llms.txt, llms-full.txt and robots.txt text. Robots rules made at build time or by a CDN, the pages behind the links, and real crawler behavior are not checked. NOT FLAGGED is not a guarantee.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 4 days ago
Permissions
Allowed Hosts
Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls. The example project includes a clearly fake key (NOTREAL-EXAMPLE-KEY) in one llms.txt link for the scanner to find, and fake example.com hosts. Nothing in it is run or fetched.