More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Crawl Budget & Indexation Diagnostician

    2

    Forensic audit engine diagnosing Googlebot crawl budget starvation, GSC hold loops, queue saturation, and indexation deadlocks across any web stack.

    $39

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Crawl Budget & Indexation Diagnostician

    Crawl Budget & Indexation Diagnostician

    Example session with this skill installed

    Analyze our production XML sitemap index (2,450 URLs) and robots.txt to diagnose why newly published engineering posts are lingering in 'Discovered - currently not indexed'. Calculate TCR, flag robots hold loops, and provide a 3-tier remediation plan.

    • Read your context and instructions
    • Compiled the crawl budget

    1. Executive Crawl Health & Saturation Metrics

    • Crawl Health Status: CRITICAL_TAXONOMY_BLOAT (Queue Saturation Risk)
    • Total Evaluated URLs: 2,450 (Canonical Content: 1,200 • Taxonomy Archives: 1,250)
    • Taxonomy-to-Content Ratio (TCR): 1.04 (Editorial Standard: $\le 0.15$ • Critical Bloat: $> 0.50$)
    • Queue Saturation Index (QSI): 0.51 (51% of sitemap scheduling capacity consumed by low-value archives)
    • Robots-Sitemap Hold Loops (HOLD_QUEUE_LOOP): 18 collisions detected

    2. Robots-Sitemap Scheduling Deadlocks Detected

    Googlebot is scheduled to crawl these URLs via XML sitemaps, but blocked from fetching by robots.txt Disallow rules, freezing scheduler execution:

    Submitted Sitemap URLColliding Disallow RuleDeadlock Status
    https://mybrand.com/tag/finops/Disallow: /tag/HOLD_QUEUE_LOOP
    https://mybrand.com/tag/cloud-infrastructure/Disallow: /tag/HOLD_QUEUE_LOOP
    https://mybrand.com/tag/kubernetes-costs/Disallow: /tag/HOLD_QUEUE_LOOP
    https://mybrand.com/page/2/Disallow: /page/HOLD_QUEUE_LOOP

    3. Prioritized 3-Tier Scheduling Recovery Plan

    Immediate Tier 3 Queue Clearance: Unsubmit post_tag-sitemap.xml and author-sitemap.xml from sitemap_index.xml; inject <meta name="robots" content="noindex, follow"> across tag archives.
    2.

    Deadlock Elimination: Purge the 18 disallowed taxonomy URLs from active sitemaps to unfreeze Googlebot's scheduler queue.
    3.

    Static Reciprocal In-Content Anchors: Inject upward and lateral internal links for stalled canonicals to bypass dynamic homepage widget decay.
    4.

    Tier 1 Expedited Inspection: Use single-URL inspection requests for top priority cluster pillars, achieving indexation in 24–48 hours without failing batch validations.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Forensic Googlebot Scheduling & Crawl Budget Diagnostic Suite

    When canonical pages linger in Google Search Console's "Discovered - currently not indexed" or "Crawled - currently not indexed" states for weeks or months, the root cause is rarely an editorial content deficit. Instead, Googlebot's scheduling pipeline is throttled by sitemap queue saturation, robots.txt hold loops, unwarmed server TTFB spikes, or transient link decay.

    This skill provides an empirical, stack-agnostic diagnostic workflow to pinpoint and clear indexation logjams across Next.js, Astro, WordPress, Shopify, Remix, or headless documentation stacks.

    Key Architectural Capabilities

    1. 3-Tier Scheduling Queue Modeling: Maps URLs across Tier 1 (Manual Inspection Expedited), Tier 2 (Link Graph Quality Evaluation), and Tier 3 (Universal XML Sitemap Discovery).
    2. 4-State Discovery Lifecycle Tracking: Categorizes every candidate URL into Submitted & Indexed, Discovered Hold Queue, Quality Filtered, or Evicted/Stale.
    3. Taxonomy-to-Content Ratio (TCR): Quantifies structural archive bloat and flags critical taxonomy thresholds across Editorial, eCommerce, and Directory verticals.
    4. Queue Saturation Index (QSI): Measures the percentage of sitemap capacity consumed by low-value tag and format archives.
    5. Robots-Sitemap Hold Loop Detection (HOLD_QUEUE_LOOP): Pinpoints URLs submitted in XML sitemaps that collide with Disallow: directives in robots.txt.
    6. Single-URL Validation Bypass Protocol: Resolves false-positive GSC validation deadlocks without triggering failing batch re-crawls.
    7. Reciprocal Link Mesh Verification: Eliminates transient widget link decay by enforcing static triangular pillar-spoke anchors.

    Standalone Engine Included

    Includes scripts/crawl_budget_diagnostician.py (< 20 KB, pure Python 3.9+ standard library, zero third-party dependencies, zero API keys required).

    Open docs

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 8 days ago

    Requires

    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Network Access

    File Scopes

    XML
    sitemaps
    robots.txt
    and
    URL
    audit
    manifests

    Runs pure Python standard library diagnostic scripts locally. Zero telemetry or external credentials required.

    Listed8 days ago

    What's inside

    Frequently Asked Questions

    Googlebot discovers URLs via sitemaps or links, but places them in a scheduling hold queue if the site's daily crawl budget is saturated by thin taxonomy archives, if TTFB exceeds 800ms, or if pages lack static internal links.

    No. The bundled diagnostic engine operates on exported sitemaps, robots.txt files, or URL lists using pure Python standard library. It also works seamlessly alongside GSC MCP tools when available.

    TCR is the ratio of submitted taxonomy URLs (tags, categories, author pages) to canonical content URLs. For editorial sites, a TCR above 0.50 signals critical crawl budget dilution that starves canonical content.

    A hold loop occurs when a URL is submitted in an XML sitemap (telling Googlebot to crawl it) but also matches a Disallow directive in robots.txt (blocking Googlebot from fetching it). It freezes Googlebot's scheduler.

    Yes. The diagnostic metrics, formulas, and CLI tools are 100% stack-agnostic and work across Next.js, Astro, Shopify, Remix, Hugo, or custom enterprise web applications.