BP

    Blocked Page Recovery

    1

    Recover content from 403/429-blocked, paywalled and Cloudflare-walled pages via third-party archives and API endpoints instead of retry-looping.

    Free

    4 uses5.0 (1 review)

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    About this skill

    Blocked-Page Recovery

    When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall, or a bot-detection interstitial — don't give up and don't loop on the same URL. Third-party services often hold a copy of the page. Work down this ladder, cheapest first.

    The ladder

    1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
    2. archive.today    — domain rotation: archive.ph -> .md -> .li -> .is
    3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
    4. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
    5. Real browser     — browser automation as the last, most expensive resort
    

    Run it in one shot with the bundled script:

    python3 scripts/recover_page.py "https://example.com/blocked-article" --json
    

    The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance.

    Provenance discipline (non-negotiable)

    Every recovered copy carries a provenance you MUST preserve when citing:

    RouteProvenanceHow to cite
    Wayback / archive.todaysnapshotCite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale.
    Jina ReaderliveServer-side re-render of the live page; cite normally.
    Live fetch / browserliveCite normally.

    If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer — say so explicitly and note its age.

    Manual routes

    1. Wayback Machine (best provenance, try first)

    # Discovery: returns closest snapshot URL + timestamp as JSON
    curl -sL "https://archive.org/wayback/available?url={URL}"
    # Then fetch archived_snapshots.closest.url
    

    For enumerating many snapshots (or recovering deleted pages), the CDX index:

    curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"
    

    CDX intermittently returns 503 under load — if it does, fall back to the available API; don't retry-hammer it.

    Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).

    2. archive.today (paywalls, deleted content)

    User-submitted archives — often has paywalled news articles Wayback lacks. Rate-limits aggressively (429) and rotates domains, so iterate:

    for d in archive.ph archive.md archive.li archive.is; do
      curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
        -w "%{http_code}" && break
    done
    

    Validate the body, not the status code — a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.

    3. Jina Reader (requires JINA_API_KEY)

    r.jina.ai re-renders the live page in a real browser server-side and returns markdown. Anonymous access is dead (401 -> Turnstile); a key is required:

    curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"
    

    Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.

    4. API-first pivot

    WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:

    • /api/..., /graphql, or .json variants of the page URL
    • An RSS/Atom feed (/feed, /rss, <link rel="alternate"> in any copy you did recover)
    • A sitemap (/sitemap.xml) revealing canonical URLs that may not be gated

    Fake successes — routes that LIE

    These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:

    • Google Cache is dead (since mid-2024). webcache.googleusercontent.com returns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it.
    • AMP caches (*.cdn.ampproject.org) mostly return a ~300-byte <title>Redirecting</title> meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop.
    • Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.

    Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").

    Proxy relays: don't

    Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on — provenance is unverifiable. Prefer archives, which at least timestamp their copies.

    Requirements

    Python 3.8+, requests (falls back to urllib). No API keys required (Jina route is skipped automatically without JINA_API_KEY).

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse All Claude Skills.

    Cannot bypass strongly authenticated pages or non-archived content. Success depends on third-party archives having the page.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    5.0
    1 review
    5
    1
    4
    0
    3
    0
    2
    0
    1
    0

    This skill has been used 4 times.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Accounts
    Archive

    Permissions

    Allowed Hosts

    archive.org
    web.archive.org
    archive.ph
    archive.md
    archive.li
    archive.is

    Skill contacts only these hosts and sends only the target URL. No third-party re-render service (r.jina.ai removed in v1.1.0).

    Listed1 month ago
    Updated1 month ago

    What's inside

    Frequently Asked Questions

    This skill allows your AI agent to bypass Cloudflare barriers, 403/429 errors, and paywalls by intelligently routing requests through a 'ladder' of third-party archives and alternative API endpoints instead of repeatedly failing on the live URL.

    The skill prioritizes public archives like the Wayback Machine and archive.today first to minimize costs, only falling back to server-side rendering (via Jina Reader) or browser automation if cheaper methods fail.

    Every piece of recovered data includes a 'provenance' tag that tells your agent if the content is a live render or a historical snapshot, ensuring the agent doesn't present stale data as current news.

    No, the skill is designed to run out-of-the-box with standard Python libraries; it automatically skips the Jina Reader route if no API key is detected, remaining functional via the other available routes.

    The skill includes specialized detection logic to identify and reject 'fake successes,' such as Google Cache redirects, AMP stubs, and Cloudflare interstitial pages that masquerade as valid HTTP 200 responses.

    You receive the core recovery logic, a ready-to-use Python implementation script, and a standardized 'provenance' framework to ensure your agent cites its sources accurately.

    Free