More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Robots and Sitemap Check

    1

    Your agent runs a scanner on robots.txt and every sitemap XML file in the project or build output.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Robots and Sitemap Check

    Robots and Sitemap Check

    Example session with this skill installed

    Check this project with robots-sitemap-check.

    • Read your context and instructions
    • Compiled the robots and sitemap

    $ python3 scripts/check_robots.py examples/risky
    BLOCK robots.txt:4 R01 SITE_BLOCKED
    Disallow: /
    Why: robots.txt tells all crawlers (or Googlebot) not to crawl anything (Disallow: /). This is often left over from a staging site, and it drops the whole site out of search.
    Safer: Remove Disallow: / for that group, or replace it with the specific private paths (for example Disallow: /admin/).

    BLOCK sitemap-blog.xml R03 SITEMAP_NOT_VALID
    BLOCK sitemap.xml:5 R02 DEV_URL_IN_SITEMAP
    REVIEW docs/robots.txt R11 NO_SITEMAP_LINE
    REVIEW docs/sitemap.xml:4 R18 SITEMAP_URL_BLOCKED
    REVIEW robots.txt:2 R15 RULE_BEFORE_USER_AGENT
    REVIEW robots.txt:5 R13 ASSETS_BLOCKED
    REVIEW robots.txt:6 R14 UNKNOWN_DIRECTIVE
    REVIEW robots.txt:7 R12 SITEMAP_LINE_NOT_ABSOLUTE
    REVIEW sitemap.xml:3 R18 SITEMAP_URL_BLOCKED
    REVIEW sitemap.xml:4 R16 MIXED_SCHEME_OR_HOST
    REVIEW sitemap.xml:4 R17 BAD_LASTMOD
    REVIEW sitemap.xml:4 R18 SITEMAP_URL_BLOCKED
    REVIEW sitemap.xml:5 R16 MIXED_SCHEME_OR_HOST
    REVIEW sitemap.xml:5 R18 SITEMAP_URL_BLOCKED

    (Each finding also has Why and Safer lines, as in the first one.)
    Verdict: BLOCK. Checked 5 files, 5 with robots.txt or sitemap content.
    This is a static check of file text. NOT FLAGGED is not a guarantee.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A Disallow: / left over from staging quietly removes a whole site from search. This skill finds that and ten other robots.txt and sitemap problems, and tells you how to fix each one.

    Your agent runs a scanner on robots.txt and every sitemap XML file in the project or build output. It also checks each sitemap URL against robots.txt, using Google's longest-match rule. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach.

    What it checks

    BLOCK

    • R01: Disallow: / (or /*) for all crawlers or Googlebot
    • R02: localhost, private IP or staging URLs in the sitemap
    • R03: a sitemap that is not valid sitemap XML

    REVIEW

    • R11: no Sitemap line in robots.txt
    • R12: a relative Sitemap line
    • R13: CSS, JS or asset folders blocked
    • R14: misspelled or unsupported rules such as Noindex
    • R15: rules before any User-agent line
    • R16: mixed http/https or hosts in a sitemap
    • R17: bad or future lastmod dates
    • R18: sitemap URLs that robots.txt blocks

    What's included

    • The scanner (Python 3.9+, standard library only)
    • A rules guide with the reason and safer approach for each rule
    • Risky and safe example files, and the real report the scanner produces on the risky ones

    Limits It reads files as written and does not fetch the live site, unzip .xml.gz sitemaps, or follow a sitemap index to its child sitemaps. NOT FLAGGED is not a guarantee of rankings.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Marketing & SEO.

    Reads local files only; does not fetch the live site, read .xml.gz files, or follow sitemap index links. Sitemaps over 1 MB are reported as not checked. NOT FLAGGED is not a guarantee.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 5 days ago

    Software
    python3 (3.9 or newer)

    Permissions

    Terminal / Shell
    Read Files

    Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls.

    Listed8 days ago
    Updated4 days ago

    What's inside

    Frequently Asked Questions

    No. The script reads local files only and makes no network calls.

    No. It only reports. Your agent changes files only if you ask it to.

    Any agent that supports SKILL.md skills and can run a terminal command, such as Claude Code. It needs Python 3.9 or newer and nothing else.

    No. It reads robots.txt and sitemap files from your project or build output and makes no network calls.

    BLOCK: fix it before you ship. REVIEW: check it before release. NOT FLAGGED: no rule matched. It is a static check, so NOT FLAGGED is not a guarantee.

    The scanner, a rules guide with the reason and safer approach for every rule, a risky and a safe example, and the real report the scanner prints on the risky example.

    New in Marketing & SEO