Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    rsl pay per crawl setup

    1

    Generates machine-readable AI licensing terms and robots.txt configs for pay-per-crawl and bot management.

    $16

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    rsl pay per crawl setup

    rsl pay per crawl setup

    Example session with this skill installed

    I want to set up pay-per-crawl for my tech blog at dev-docs.com. I own all the articles in /blog but not the user comments in /forum. I want a training license for AI labs.

    • Read your context and instructions
    • Compiled the rsl pay per

    Generated RSL config for dev-docs.com:

    1. robots.txt: License: https://dev-docs.com/license.xml
    2. license.xml: <content url="/blog*"> <permits>ai-train</permits> <payment type="crawl" rate="0.01" currency="USD" /> </content>
      Note: The /forum path is excluded to protect non-owned content.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Publishers lose control over their data when AI crawlers scrape content without permission or compensation. Standard robots.txt files are often ignored, and declaring machine-readable commercial terms for AI training is technically complex.

    What it does

    • Generates site-specific license.xml files using the Really Simple Licensing (RSL) schema.
    • Produces robots.txt directives for pay-per-crawl, pay-per-inference, and attribution models.
    • Configures path-specific crawler gating to protect user-generated or syndicated content.
    • Maps Cloudflare Pay-Per-Crawl headers and HTTP 402 signal requirements for supported infrastructures.

    Frameworks & tools

    RSL (Really Simple Licensing) 1.0, Cloudflare Pay-Per-Crawl, robots.txt, XML.

    Why this beats prompting it yourself

    Generic prompts often produce hallucinated XML schemas or outdated bot lists. This skill uses a verified RSL snapshot to ensure syntax accuracy and forces a "truth-first" approach regarding the limits of crawler enforcement.

    Use cases

    • Declare commercial training rates for LLM datasets.
    • Block specific AI agents while allowing search engine indexing.
    • Set up pay-per-crawl licensing for high-value technical documentation.
    • Implement RSL Collective enrollment terms for domain-wide content.

    Known limitations

    Cannot guarantee payment from AI labs as RSL compliance is voluntary. Cloudflare Pay-Per-Crawl requires separate Cloudflare-level configuration to function.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills to Reverse Engineer and Scrape Websites.

    Compliance is voluntary; AI crawlers may ignore directives. Cloudflare Pay-Per-Crawl requires an active Cloudflare account and manual opt-in. Cannot verify content legal ownership.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 months ago

    Permissions

    No special permissions declared or detected

    Listed3 months ago

    What's inside

    Frequently Asked Questions

    This skill automates the creation of machine-readable license.xml files and optimized robots.txt directives, allowing you to clearly state your commercial terms and payment requirements for AI data scraping.

    Yes, the generated configurations are designed to be granular, allowing you to explicitly block AI model training bots while preserving access for traditional SEO crawlers like Googlebot or Bingbot.

    The setup provides the necessary XML and header documentation to integrate with Cloudflare Pay-Per-Crawl and RSL Collective, though you will still need an active Cloudflare account to enforce the technical gating.

    You receive a custom-generated RSL license file (XML), a structured robots.txt configuration, and technical mapping for HTTP 402 signals tailored to your specific domain and content paths.

    While RSL is a voluntary standard, using this skill ensures your terms are signaled in the industry-standard machine-readable format that major AI labs use to identify commercial licensing requirements.

    New in Agents & Orchestration