Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Scrape Protected Websites

    1

    Fetch pages that block normal requests (403, captcha, JavaScript-only) as HTML, clean Markdown or AI-parsed JSON.

    Free

    Security scanned
    Scrape Protected Websites

    Scrape Protected Websites

    Example session with this skill installed

    Fetch the product price and availability for https://example.com/item-123 as structured JSON using the US proxy.

    • Read your context and instructions
    • Compiled the scrape protected websites

    I have successfully fetched the structured data for the requested URL using a US-based proxy. The ScrapeUnblocker API bypassed the site's protection and extracted the following fields.

    Extracted Product Data

    FieldValue
    URLhttps://example.com/item-123
    Page Typeproduct_listing
    TitleWireless Noise Cancelling Headphones - Model X
    Price$299.99
    CurrencyUSD
    AvailabilityIn Stock
    Rating4.8/5
    Review Count1,240
    Sourcescrape-unblocker-ai
    Data Extractedtrue

    Next steps

    1. Map these JSON fields to your internal database schema.
    2. If you need the full product description for an LLM, run the command again with markdown=true.
    3. Check the X-Origin-Status header if you encounter 404 errors on specific SKU variations.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Standard HTTP clients fail when faced with Cloudflare, 403 Forbidden errors, or JavaScript-heavy SPAs. This skill routes requests through a headless browser environment to bypass bot detection and retrieve clean data.

    What it does

    • Bypasses bot protection by using real browser rendering to circumvent 403, 429, and captcha blocks.
    • Extracts structured JSON using AI-parsed fields for products, articles, and listings without manual selectors.
    • Converts to Markdown to provide LLM-friendly content, stripping boilerplate for RAG or summarization.
    • Handles Geo-restrictions via residential proxies, allowing access to localized content using ISO country codes.
    • Waits for dynamic content by observing CSS selectors or XPaths before returning the page source.

    How it works

    1. Configure access by setting the SCRAPEUNBLOCKER_KEY environment variable with your API credentials.
    2. Execute requests via the bundled Python helper script or direct curl commands to the API endpoint.
    3. Specify output format by setting parameters for raw HTML, Markdown, or parsed JSON data.
    4. Handle responses by checking for the X-Origin-Status header and verifying content extraction status.

    Frameworks & tools

    The skill utilizes the ScrapeUnblocker API and a Python helper script. It supports integration via official scrapeunblocker SDKs for Python and Node.js, requiring only the standard library for the base implementation.

    Why this beats prompting it yourself

    Most LLMs cannot browse the live web effectively when faced with sophisticated anti-bot headers or JavaScript requirements. This skill provides the specialized infrastructure needed to fetch the raw data that prompts alone cannot reach.

    Use cases

    • Competitor monitoring to track prices and availability on protected retail sites.
    • RAG pipeline ingestion to fetch clean, distraction-free Markdown from news articles.
    • Market research to gather structured data from localized search results across different countries.
    • Data archival to capture fully rendered HTML from sites that rely heavily on React or Vue.

    Known limitations

    Requires a ScrapeUnblocker API key. Does not support bypass of logins or paywalls. Parameters like markdown and parsed_data cannot be combined in a single request.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Workflow Automation.

    Requires ScrapeUnblocker API key. No login/paywall bypass. Specific formats (Markdown/JSON) cannot be combined in one call. Max timeout is 60s.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 13 days ago

    • Free with an Agensi account

    Accounts
    Scrapeunblocker

    Permissions

    Terminal / Shell
    Network Access
    Environment Variables

    Allowed Hosts

    api.scrapeunblocker.com
    example.com
    app.scrapeunblocker.com
    Listed13 days ago

    What's inside

    Frequently Asked Questions

    Standard HTTP requests are often blocked by Cloudflare or 403 errors; this skill uses a headless browser environment and residential proxies to mimic real human behavior and bypass these protections.

    The skill is designed as a modular tool compatible with any agent framework that supports Python or Node.js, including LangChain, AutoGPT, and custom OpenAI Swarm implementations.

    You will receive the core logic for the scraping integration, a Python helper script for rapid deployment, and documentation for configuring the required ScrapeUnblocker API credentials.

    Yes, you can use the 'wait_for' parameter to specify CSS selectors or XPaths, ensuring the skill only captures the data once the JavaScript elements have fully rendered.

    By default, the skill returns data in LLM-friendly Markdown to strip out boilerplate code, but it can also be configured to return structured JSON or raw HTML depending on your agent's needs.

    This skill does not support bypassing authentication screens or subscription-based paywalls; it is intended for accessing publicly available content that is protected by bot-detection layers.

    New in Workflow Automation

    Free