Works with the AI Agents you already use
llms.txt + AI Crawler Access Skill Pack
Audit whether AI crawlers can reach your key pages and draft a practical llms.txt — without cloaking, inventing URLs, or promising ranking gains.
$29
llms.txt + AI Crawler Access Skill Pack
Example session with this skill installed
Audit AI crawler access for https://example.com focusing on /guides/* and /docs/*. Draft an llms.txt that only lists URLs that exist. Recommend robots changes; don't apply them.
- Read your context and instructions
- Compiled the llms txt
EXAMPLE — fictional site, illustrative output only.
EXAMPLE audit: /guides/ allowed; /docs/ disallowed to unknown bots via broad Disallow: /docs — may block AI crawlers unintentionally.
EXAMPLE llms.txt draft: # EXAMPLE title + 5 real guide URLs + contact.
EXAMPLE robots note: Narrow the /docs rule; document GPTBot/ClaudeBot intent in the matrix. Publishing is a human step.
Connects securely to your tools. The creator never sees your data.
About this skill
llms.txt + AI Crawler Access is a Claude / Cursor skill for the technical access layer of GEO/AEO: robots policy, AI bot allow/deny decisions, and a sane llms.txt draft.
What it does
- Audit crawler access to priority URLs (fetch/robots perspective).
- Fill a bot-policy decision matrix (what you want GPTBot, ClaudeBot, etc. to do — recommendations only).
- Draft llms.txt pointing at real canonical pages you control.
- Validate checklist before you (or your host) publish.
What's in the zip
SKILL.md— crawler audit + llms.txt pipeline- Crawler access audit template
- llms.txt draft template
- Bot policy decision matrix
- robots fetch + llms.txt validate checklists
- EXAMPLE walkthrough
- README + cover art
What you bring
Domain, list of key URLs, and access to edit robots.txt / root files (or a ticket to whoever can). The skill drafts; it does not SSH into your server.
For / not for
For: SEOs and founders aligning technical access with AI-visibility goals.
Not for: Bypassing paywalls, cloaking, or ranking guarantees.
Companion skills on Agensi (HubDev)
Pairs with Citation Hub, Schema Markup, and Brand Citation Audit. Included in the HubDev GEO/AEO Skill Stack.
Sample output (real public sites, abridged)
Sites: stripe.com and anthropic.com (first audited 2026-11-06, re-checked 2026-11-13; neither is a HubDev client; files may change after these dates)
| Check | stripe.com | anthropic.com |
|---|---|---|
| robots.txt | 200, fetched | 200, fetched |
| Named AI crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) | None; falls back to the User-agent: * group | None; "User-Agent: * / Allow: /" |
| llms.txt | Present (200); starts with a one-line summary, then sections such as Payments | Not found (404) |
What the skill concludes: neither site blocks these AI crawlers by name, so access depends on the wildcard rules. Stripe already publishes an llms.txt, so the job is a review. Anthropic has none, so the skill would draft one from the site's main pages for the owner to approve. Every draft is clearly labeled as an example until the owner publishes it.
The audit only reports what the files say. It does not promise crawling, indexing, or citations.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Cannot bypass paywalls or cloaking. Requires manual verification of page existence to avoid 404s. llms.txt is emerging guidance and not a ranking guarantee.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 10 days ago
Permissions
No special permissions declared or detected