Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Web Scraper: Polite, Reliable Data Extraction

    2

    Web Scraper makes your agent extract data like a careful data engineer.

    Free

    12 uses5.0 (1 review)
    Web Scraper: Polite, Reliable Data Extraction

    Web Scraper: Polite, Reliable Data Extraction

    Example session with this skill installed

    Get all products (name, price, availability, URL) from this supplier catalog into a spreadsheet: https://example-supplier.com/catalog

    • Read your context and instructions
    • Compiled the web scraper

    ## Checks - No public API; `sitemap.xml` lists 412 product URLs, but category pages already contain all fields -> scrape 18 category pages instead of 412 product pages (23x fewer requests) - `robots.txt`: catalog allowed, `/cart` and `/account` disallowed, `Crawl-delay: 2` -> delay set to 2 s - Content is in the initial HTML (no JS needed); each card also has JSON-LD `Product` with price and availability -> used it as the primary source (more stable than CSS classes) ## Selector map | Field | Source | |---|---| | name | JSON-LD `name` (fallback `.card h3`) | | price | JSON-LD `offers.price` + `priceCurrency` | | availability | JSON-LD `offers.availability` -> in_stock / out_of_stock | | url | `.card a@href` (absolute) | | pagination | `a[rel=next]` | ## Run ```bash python scripts/polite_scrape.py --urls categories.txt --item ".card" \ --field name="h3" --field price=".price" --field url="a@href" \ --next "a[rel=next]@href" --delay 2 --out raw.csv ``` Then merged with the JSON-LD extraction and cleaned. ## Result: `products.xlsx` - **396 products** from 18 pages in 41 s (cached; re-runs are instant) - 16 duplicates removed (products listed in two categories) - Prices `1.234,50 €` -> `1234.50` + `currency=EUR`; availability normalised - Empty fields: name 0 %, price 1.5 % (6 "price on request" items, flagged), url 0 % - Spot-checked 3 random products against the live site: all match ## Notes - If the site redesigns, JSON-LD is the most likely to survive; the CSS fallback for name may break. - For weekly price tracking: run with a fresh cache every Monday and append `scraped_at` - want me to set that up?

    Connects securely to your tools. The creator never sees your data.

    About this skill

    ## Get the data without hammering the site (or getting garbage) Quick scraping scripts break on page 2, mangle accents and prices, ignore robots.txt and get blocked. Web Scraper makes your agent extract data like a careful data engineer. ## What it does - **Official sources first**: public APIs, RSS, sitemaps, exports, open data, and data already embedded in the page (JSON-LD, `__NEXT_DATA__`, `__NUXT__`, XHR endpoints). - **Legitimacy check**: robots.txt and terms; no login/paywall/CAPTCHA bypass, no block evasion, no bulk personal data - says so clearly when a request crosses the line. - **Inspect before coding**: record container, field selectors (stable hooks over generated classes), pagination pattern, static vs JS-rendered. - **Right tool**: included `polite_scrape.py` (requests + BeautifulSoup) for static pages, Playwright for JS pages, direct JSON APIs, `pandas.read_html` for tables, Scrapy for large crawls. - **Polite & robust**: real User-Agent, robots.txt + Crawl-delay, 1 request at a time with delay, backoff on 429/5xx with Retry-After, page cache, retries, resume. - **Clean & validated output**: locale-aware numbers, ISO dates, absolute URLs, dedupe, `_source` URL, empty-field report and spot checks against the live page. - **Re-runnable**: script + config, schedule notes and which selectors are fragile. For price tracking, catalogs, listings, directories, research datasets and HTML tables to spreadsheets. ## Includes - SKILL.md workflow - scripts/polite_scrape.py - tested scraper: robots.txt, rate limit, retries, cache, pagination, CSV/JSON, encoding fix - references/selectors.md - hidden structured data, selector tips, pagination patterns, Playwright snippet, cleaning helpers Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI and any SKILL.md-compatible agent. Requires Python 3.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for APIs & Backend.

    Does not bypass CAPTCHAs, paywalls, or anti-bot shields. No support for scraping behind logins or collecting private personal data. Requires Python environment for script execution.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    5.0
    1 review
    5
    1
    4
    0
    3
    0
    2
    0
    1
    0

    This skill has been used 12 times.

    Trust & safety

    Security scan

    Security scan passed 9 days ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Browser
    Read Files
    Write Files
    Network Access
    Listed9 days ago
    Updated6 days ago

    What's inside

    Frequently Asked Questions

    This skill is designed to work with any AI agent capable of executing Python environments, including AutoGPT, BabyAGI, and custom LangChain agents. It integrates seamlessly via standard API calls or direct script execution depending on your agent's architecture.

    Unlike basic LLM-generated scripts, this skill includes built-in 'politeness' protocols like rate limiting and robots.txt compliance to prevent your IP from being flagged. It also uses Playwright to handle dynamic, JavaScript-heavy sites that standard scrapers often fail to load.

    The skill is pre-configured with the necessary libraries (BeautifulSoup, Playwright, Pandas) and a wrapper script; you simply need to provide the target URL and desired data schema. No complex environment configuration is required beyond standard Python package installation.

    You will receive the full functional skill package, including the extraction engine, data normalization modules, and a quality validation suite. This covers everything from initial site inspection to final CSV/JSON output generation.

    To ensure ethical use and high success rates, the skill does not support bypassing CAPTCHAs, logging into private accounts, or accessing paywalled content. It is optimized for extracting publicly available, structured data.

    Yes, the skill includes a data normalization layer that automatically converts inconsistent web data into standardized formats, such as ISO 8601 for dates and floats for currency. It also performs a reliability check to flag empty fields or potential errors before delivery.

    New in APIs & Backend

    Free