Nex Scraper Politeness Pattern

What it does

The Nex Scraper Politeness Pattern provides a production-ready, asynchronous Python fetch layer designed for ethical and sustainable web scraping. It moves beyond simple HTTP requests by implementing a sophisticated coordination layer that prevents your agent from being flagged as a "bad bot." The skill generates a complete package including a PoliteScraper engine and a specialized email_extractor.

Why use this skill

Standard scraping scripts often fail because they lack the "politeness" heuristics that real-world websites expect. Instead of just hammering a URL, this skill implements a global concurrency cap paired with a per-domain throttle. This means you can scrape 100 different websites simultaneously, but your agent will never hit any single site more than once every few seconds. It ensures your bot identifies itself honestly, respects robots.txt, and handles transient network errors with exponential backoff.

Supported tools and features

• Async Concurrency: Managed via httpx and semaphores for high-performance non-blocking I/O.
• Per-Domain Throttling: Unique locking mechanism to ensure specific hosts aren't overwhelmed.
• Honest Identification: Automatically configures a transparent User-Agent with contact details.
• Robots.txt Awareness: Integrated checking and caching of host permissions.
• Defensible Lead Gen: A regex-based extractor that targets role-based emails (info@, contact@) while automatically filtering out sensitive personal addresses.

The Result

You receive a modular Python package (polite_scraper.py, email_extract.py, and requirements.txt). The code is cleanly structured as an async context manager, providing a fetch_many() method that returns structured results without crashing on individual failures.

Built by Nex AI. More skills and info at nex-ai.be and slopsome.com.