- Home
- Skills
- Agents & Orchestration
- rsl pay per crawl setup
Works with the AI Agents you already use
rsl pay per crawl setup
Generates machine-readable AI licensing terms and robots.txt configs for pay-per-crawl and bot management.
$16
rsl pay per crawl setup
Example session with this skill installed
I want to set up pay-per-crawl for my tech blog at dev-docs.com. I own all the articles in /blog but not the user comments in /forum. I want a training license for AI labs.
- Read your context and instructions
- Compiled the rsl pay per
Generated RSL config for dev-docs.com:
- robots.txt: License: https://dev-docs.com/license.xml
- license.xml: <content url="/blog*"> <permits>ai-train</permits> <payment type="crawl" rate="0.01" currency="USD" /> </content>
Note: The /forum path is excluded to protect non-owned content.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Publishers lose control over their data when AI crawlers scrape content without permission or compensation. Standard robots.txt files are often ignored, and declaring machine-readable commercial terms for AI training is technically complex.
What it does
- Generates site-specific
license.xmlfiles using the Really Simple Licensing (RSL) schema. - Produces
robots.txtdirectives for pay-per-crawl, pay-per-inference, and attribution models. - Configures path-specific crawler gating to protect user-generated or syndicated content.
- Maps Cloudflare Pay-Per-Crawl headers and HTTP 402 signal requirements for supported infrastructures.
Frameworks & tools
RSL (Really Simple Licensing) 1.0, Cloudflare Pay-Per-Crawl, robots.txt, XML.
Why this beats prompting it yourself
Generic prompts often produce hallucinated XML schemas or outdated bot lists. This skill uses a verified RSL snapshot to ensure syntax accuracy and forces a "truth-first" approach regarding the limits of crawler enforcement.
Use cases
- Declare commercial training rates for LLM datasets.
- Block specific AI agents while allowing search engine indexing.
- Set up pay-per-crawl licensing for high-value technical documentation.
- Implement RSL Collective enrollment terms for domain-wide content.
Known limitations
Cannot guarantee payment from AI labs as RSL compliance is voluntary. Cloudflare Pay-Per-Crawl requires separate Cloudflare-level configuration to function.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills to Reverse Engineer and Scrape Websites.
Compliance is voluntary; AI crawlers may ignore directives. Cloudflare Pay-Per-Crawl requires an active Cloudflare account and manual opt-in. Cannot verify content legal ownership.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 months ago
Permissions
No special permissions declared or detected