- Home
- Skills
- Testing & Debugging
- Stealth Web Scraper
Works with the AI Agents you already use
Stealth Web Scraper
Scrape pages that block plain requests: impersonate real browsers with curl_cffi, keep a session cookie jar, rotate to Playwright when JavaScript rendering or a
Free
Stealth Web Scraper
Example session with this skill installed
Scrape the product names and prices from https://example-store.com/deals which uses a heavy anti-bot wall and loads content via JavaScript.
- Read your context and instructions
- Compiled the stealth web scraper
Starting Stage 3 escalation with Playwright. Waiting for '.product-grid' selector...
Data extracted
- "Pro Wireless Mouse" | $49.99
- "Mechanical Keyboard" | $89.00
- "USB-C Hub" | $25.50
Crawl complete. 3 items saved to page.html.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Standard scraping requests often trigger 403 Forbidden errors, CAPTCHA walls, or return empty HTML because the target site detects non-browser TLS fingerprints or requires JavaScript execution.
What it does
- Escalates through a three-stage "scraping ladder" from simple headers to full browser emulation to minimize overhead.
- Impersonates real Chrome TLS fingerprints and header ordering to bypass sophisticated bot detection systems.
- Maintains persistent session cookie jars across requests to handle multi-step navigation or authentication.
- Renders JavaScript-heavy pages and waits for specific DOM elements using headless Chromium.
- Extracts structured data from raw HTML using CSS selectors and XPath.
Frameworks & tools
Python, curl_cffi, Playwright, Chromium, and Parsel.
Why this beats prompting it yourself
Writing reliable scraping logic requires deep knowledge of TLS fingerprinting and browser emulation. This skill implements a proven escalation strategy that avoids over-using heavy resources like Playwright when lightweight impersonation is sufficient.
Use cases
- Extracting product listings from e-commerce sites with anti-bot protection.
- Gathering data from single-page applications (SPAs) that require script execution.
- Collecting research data from sites that block standard Python Requests or cURL.
- Scraping paginated content that requires session and cookie persistence.
Known limitations
Not designed for bypassing logins, paywalls via credential abuse, or high-volume abusive scraping. Requires Python environment with Playwright binaries installed.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills to Reverse Engineer and Scrape Websites.
Requires local installation of curl_cffi and Playwright. Not intended for bypassing paywalls or credential-based authentication.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
4 uses
Times used to date
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 3 months ago
- Free with an Agensi account