More screenshots
Works with the AI Agents you already use
Playwright Scraper: Extract Pages That Fight Back
Extract content from sites that need a real browser — the escalation ladder from plain fetch to stealth, waiting that works, and scraping ethics that keep you out of trouble.
$5
Playwright Scraper: Extract Pages That Fight Back
Example session with this skill installed
A React job board with infinite scroll returns empty HTML to plain fetch. Scrape all listings — title, company, location, link — and make it survive the next redesign.
- Read your context and instructions
- Compiled the playwright scraper
Level 1 first: plain HTTP fetch returned the shell — empty content, 200 OK — which is the classic JavaScript-rendered signature. Level 2 sufficed: headless Chromium, no tricks, because the board has no anti-bot. Never start at level 3. Script: launched headless Chromium (headed mode kept one flag away for debugging), set viewport, user agent, locale, and timezone to match a plausible visitor. Navigated and waited for the content selector [data-testid='job-card'] — never a fixed sleep, never the page load event, which SPAs fire long before data arrives. Paginated by infinite scroll: scroll to bottom, wait 1.5s, count cards, repeat until the count stops growing. Two consecutive scrolls adding nothing means done. Extracted title, company, location, and link per card via $$eval, preferring semantic selectors over brittle CSS paths, and pulled the embedded JSON where the page offered it — cleaner than DOM scraping. Persisted raw HTML per page plus a jobs.json. 340 listings in about 4 minutes, browser closed in a finally block. When the site redesigns next quarter, the saved HTML shows exactly which selectors broke — the extractor gets fixed without a re-scrape. The ladder is the discipline: the lightest tool that works, proven by trying, not by reaching for the heaviest tool first.
Connects securely to your tools. The creator never sees your data.
About this skill
Some pages only exist after JavaScript runs; some actively fight automation. This skill is the escalation ladder: start with the lightest tool that works and only bring out the browser — and the stealth — when the page demands it. It covers Playwright setup, the script anatomy (launch, context, navigate and wait, extract, paginate, persist, close), waiting strategies that fix flaky scrapers, stealth honestly (what anti-bot systems actually look for and where the line is), and a failure playbook that maps symptoms to causes to fixes.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse All Claude Skills.
Bypassing a CAPTCHA or terms of service to scrape at scale can be unlawful depending on jurisdiction and target — the skill teaches the line, it does not cross it.
Stealth is countermeasure work: expect to iterate, and know that a residential or mobile IP is often the actual fix, not fancier JavaScript.
Specialized sources (YouTube transcripts, Reddit threads) are better served by purpose-built scrapers than a generic browser script.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
- Software
- Playwright
Node.js