Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentsShip llms.txt + Unlock AI Crawlers: Access Fix Pack
llms.txt generator + AI crawler access fix: writes a starter llms.txt, audits robots.txt for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ships a fix checklist so AI search can read your site.
$7
Ship llms.txt + Unlock AI Crawlers: Access Fix Pack
Example session with this skill installed
Run an AI access audit on my robots.txt file and generate a starter llms.txt for my site "DevDocs Portal" using the URLs in sites.txt.
- Read your context and instructions
- Compiled the ship llms txt
I have completed the AI crawler access audit and generated the requested starter files. The audit identified two critical blocks affecting AI visibility.
AI Access Audit Report
| User-Agent | Status | Rule Source |
|---|---|---|
| GPTBot | FAIL | Disallow: / |
| ClaudeBot | PASS | Allowed by default |
| PerplexityBot | WARN | Partial block on /api/ |
| Google-Extended | PASS | Empty Disallow |
| CCBot | FAIL | Disallow: / |
Score: FAIL
The site is currently invisible to major LLM training crawlers due to a blanket Disallow rule in robots.txt.
starter llms.txt
# DevDocs Portal
> TODO: Add a brief 2-3 sentence summary of the site's purpose here.
## Key Pages
- [Home](https://example.com/): TODO: Page summary.
- [API Reference](https://example.com/api): TODO: Page summary.
- [Guides](https://example.com/guides): TODO: Page summary.
Next steps
- Update
robots.txtto includeUser-agent: GPTBotfollowed byAllow: /to permit OpenAI indexing. - Replace the
TODOplaceholders in the generatedllms.txtwith descriptive site metadata. - Upload the final
llms.txtto your site root and re-run the audit with the--liveflag.
Connects securely to your tools. The creator never sees your data.
About this skill
llms.txt generator + AI crawler access fix: writes a starter llms.txt, audits robots.txt for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ships a fix checklist so AI search can read your site.
Sample output (offline dry-run on the included Harborline robots + URL list)
bots=9 | blocked=GPTBot,Bytespider,CCBot | llms=ok | score=FAIL
| Bot | Status | Why it matters |
|---|---|---|
| GPTBot | blocked (Disallow: /) | Critical — OpenAI training/search crawler fully locked out |
| Bytespider | blocked | TikTok crawler fully locked out |
| CCBot | blocked | Common Crawl fully locked out |
| ClaudeBot | partial (/private/) | Review if intentional |
| PerplexityBot | partial (/drafts/, /internal/) | Review if intentional |
| Google-Extended | allowed | Gemini / Google-Extended training signal open |
| OAI-SearchBot, ChatGPT-User, Applebot-Extended | allowed (via *) | No specific block |
Starter llms.txt excerpt written by the pack:
# Harborline Docs
> Fictional docs site used in the Cygnix Labs dry-run.
## Key pages
- [Getting Started](https://docs.harborline.example/getting-started): TODO — one-line summary…
- [Overview](https://docs.harborline.example/api/overview): TODO — one-line summary…
Differentiation (read this first) Unlike Cygnix AEO Audit / aeo-readiness-checker Apify actor (which crawls pages and scores HTML/schema for AI visibility), this pack is the operational fix kit: it WRITES a starter llms.txt, audits robots rules for AI bots, and produces an agent-ready fix checklist.
The problem
Security plugins and default CMS robots often blanket-Disallow GPTBot, ClaudeBot, or CCBot. Sites also ship with no /llms.txt. AEO content work is wasted if crawlers never get in.
What it does
- Generates a starter
/llms.txtfrom your local URL/path list - Parses pasted or local
robots.txtfor GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, CCBot - Scores AI readiness: PASS / WARN / FAIL (missing llms.txt, blanket AI Disallow, sitemap hints, partial path rules)
- Offline-first (stdlib Python); optional
--livefetches only/robots.txtand/llms.txtfor origins you listed - Markdown report + prioritised fix checklist for Claude Code / Cursor / Codex
Why this beats prompting it yourself Same bot list and robots conventions every run. Offline works unplugged. Never invents live Allow lines. Writes the file your agent would otherwise hand-author inconsistently.
Use cases
- Pre-AEO access gate before content rewrites
- CMS / WAF audits after a plugin update blocks AI bots
- Agency deliverable: "AI crawler access + llms.txt starter" in one pass
Example session
"Audit this robots.txt for AI bots and write a starter llms.txt from urls.txt." → bots=9 | blocked=GPTBot,Bytespider,CCBot | llms=ok | score=FAIL → Fix checklist: remove GPTBot Disallow: / ; publish llms.txt with real page blurbs → Partial rules on ClaudeBot / PerplexityBot flagged for human review
Trust Works in Claude Code, Cursor and Codex · tested on the included Harborline sample · v1.0.0 · stdlib Python only · questions answered within 24h · Cygnix Labs
Pairs with: AEO Audit, SEO Content Brief Lite, Content Refresh Audit, LLM Link Verifier.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scanned
Verified clean 3 days ago
- Passed all security checks, Safe to install
Requires
Llmstxt
Yoursite