- Home
- Skills
- Marketing & SEO
- Robots and Sitemap Check
More screenshots
Works with the AI Agents you already use
Robots and Sitemap Check
Your agent runs a scanner on robots.txt and every sitemap XML file in the project or build output.
$5
Robots and Sitemap Check
Example session with this skill installed
Check this project with robots-sitemap-check.
- Read your context and instructions
- Compiled the robots and sitemap
$ python3 scripts/check_robots.py examples/risky
BLOCK robots.txt:4 R01 SITE_BLOCKED
Disallow: /
Why: robots.txt tells all crawlers (or Googlebot) not to crawl anything (Disallow: /). This is often left over from a staging site, and it drops the whole site out of search.
Safer: Remove Disallow: / for that group, or replace it with the specific private paths (for example Disallow: /admin/).
BLOCK sitemap-blog.xml R03 SITEMAP_NOT_VALID
BLOCK sitemap.xml:5 R02 DEV_URL_IN_SITEMAP
REVIEW docs/robots.txt R11 NO_SITEMAP_LINE
REVIEW docs/sitemap.xml:4 R18 SITEMAP_URL_BLOCKED
REVIEW robots.txt:2 R15 RULE_BEFORE_USER_AGENT
REVIEW robots.txt:5 R13 ASSETS_BLOCKED
REVIEW robots.txt:6 R14 UNKNOWN_DIRECTIVE
REVIEW robots.txt:7 R12 SITEMAP_LINE_NOT_ABSOLUTE
REVIEW sitemap.xml:3 R18 SITEMAP_URL_BLOCKED
REVIEW sitemap.xml:4 R16 MIXED_SCHEME_OR_HOST
REVIEW sitemap.xml:4 R17 BAD_LASTMOD
REVIEW sitemap.xml:4 R18 SITEMAP_URL_BLOCKED
REVIEW sitemap.xml:5 R16 MIXED_SCHEME_OR_HOST
REVIEW sitemap.xml:5 R18 SITEMAP_URL_BLOCKED
(Each finding also has Why and Safer lines, as in the first one.)
Verdict: BLOCK. Checked 5 files, 5 with robots.txt or sitemap content.
This is a static check of file text. NOT FLAGGED is not a guarantee.
Connects securely to your tools. The creator never sees your data.
About this skill
A Disallow: / left over from staging quietly removes a whole site from search. This skill finds that and ten other robots.txt and sitemap problems, and tells you how to fix each one.
Your agent runs a scanner on robots.txt and every sitemap XML file in the project or build output. It also checks each sitemap URL against robots.txt, using Google's longest-match rule. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach.
What it checks
BLOCK
- R01: Disallow: / (or /*) for all crawlers or Googlebot
- R02: localhost, private IP or staging URLs in the sitemap
- R03: a sitemap that is not valid sitemap XML
REVIEW
- R11: no Sitemap line in robots.txt
- R12: a relative Sitemap line
- R13: CSS, JS or asset folders blocked
- R14: misspelled or unsupported rules such as Noindex
- R15: rules before any User-agent line
- R16: mixed http/https or hosts in a sitemap
- R17: bad or future lastmod dates
- R18: sitemap URLs that robots.txt blocks
What's included
- The scanner (Python 3.9+, standard library only)
- A rules guide with the reason and safer approach for each rule
- Risky and safe example files, and the real report the scanner produces on the risky ones
Limits It reads files as written and does not fetch the live site, unzip .xml.gz sitemaps, or follow a sitemap index to its child sitemaps. NOT FLAGGED is not a guarantee of rankings.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Marketing & SEO.
Reads local files only; does not fetch the live site, read .xml.gz files, or follow sitemap index links. Sitemaps over 1 MB are reported as not checked. NOT FLAGGED is not a guarantee.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 5 days ago
- Software
python3 (3.9 or newer)
Permissions
Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls.