- Home
- Skills
- APIs & Backend
- Web Scraper: Polite, Reliable Data Extraction
Works with the AI Agents you already use
Web Scraper: Polite, Reliable Data Extraction
Web Scraper makes your agent extract data like a careful data engineer.
Free
Web Scraper: Polite, Reliable Data Extraction
Example session with this skill installed
Get all products (name, price, availability, URL) from this supplier catalog into a spreadsheet: https://example-supplier.com/catalog
- Read your context and instructions
- Compiled the web scraper
## Checks - No public API; `sitemap.xml` lists 412 product URLs, but category pages already contain all fields -> scrape 18 category pages instead of 412 product pages (23x fewer requests) - `robots.txt`: catalog allowed, `/cart` and `/account` disallowed, `Crawl-delay: 2` -> delay set to 2 s - Content is in the initial HTML (no JS needed); each card also has JSON-LD `Product` with price and availability -> used it as the primary source (more stable than CSS classes) ## Selector map | Field | Source | |---|---| | name | JSON-LD `name` (fallback `.card h3`) | | price | JSON-LD `offers.price` + `priceCurrency` | | availability | JSON-LD `offers.availability` -> in_stock / out_of_stock | | url | `.card a@href` (absolute) | | pagination | `a[rel=next]` | ## Run ```bash python scripts/polite_scrape.py --urls categories.txt --item ".card" \ --field name="h3" --field price=".price" --field url="a@href" \ --next "a[rel=next]@href" --delay 2 --out raw.csv ``` Then merged with the JSON-LD extraction and cleaned. ## Result: `products.xlsx` - **396 products** from 18 pages in 41 s (cached; re-runs are instant) - 16 duplicates removed (products listed in two categories) - Prices `1.234,50 €` -> `1234.50` + `currency=EUR`; availability normalised - Empty fields: name 0 %, price 1.5 % (6 "price on request" items, flagged), url 0 % - Spot-checked 3 random products against the live site: all match ## Notes - If the site redesigns, JSON-LD is the most likely to survive; the CSS fallback for name may break. - For weekly price tracking: run with a fresh cache every Monday and append `scraped_at` - want me to set that up?
Connects securely to your tools. The creator never sees your data.
About this skill
## Get the data without hammering the site (or getting garbage) Quick scraping scripts break on page 2, mangle accents and prices, ignore robots.txt and get blocked. Web Scraper makes your agent extract data like a careful data engineer. ## What it does - **Official sources first**: public APIs, RSS, sitemaps, exports, open data, and data already embedded in the page (JSON-LD, `__NEXT_DATA__`, `__NUXT__`, XHR endpoints). - **Legitimacy check**: robots.txt and terms; no login/paywall/CAPTCHA bypass, no block evasion, no bulk personal data - says so clearly when a request crosses the line. - **Inspect before coding**: record container, field selectors (stable hooks over generated classes), pagination pattern, static vs JS-rendered. - **Right tool**: included `polite_scrape.py` (requests + BeautifulSoup) for static pages, Playwright for JS pages, direct JSON APIs, `pandas.read_html` for tables, Scrapy for large crawls. - **Polite & robust**: real User-Agent, robots.txt + Crawl-delay, 1 request at a time with delay, backoff on 429/5xx with Retry-After, page cache, retries, resume. - **Clean & validated output**: locale-aware numbers, ISO dates, absolute URLs, dedupe, `_source` URL, empty-field report and spot checks against the live page. - **Re-runnable**: script + config, schedule notes and which selectors are fragile. For price tracking, catalogs, listings, directories, research datasets and HTML tables to spreadsheets. ## Includes - SKILL.md workflow - scripts/polite_scrape.py - tested scraper: robots.txt, rate limit, retries, cache, pagination, CSV/JSON, encoding fix - references/selectors.md - hidden structured data, selector tips, pagination patterns, Playwright snippet, cleaning helpers Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI and any SKILL.md-compatible agent. Requires Python 3.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for APIs & Backend.
Does not bypass CAPTCHAs, paywalls, or anti-bot shields. No support for scraping behind logins or collecting private personal data. Requires Python environment for script execution.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
This skill has been used 12 times.
Trust & safety
Security scan
Security scan passed 9 days ago
- Free with an Agensi account