Nex Scraper Politeness Pattern

What it does

• Generates polite_scraper.py, a PoliteScraper async context manager with fetch() and fetch_many() built on httpx.
• Enforces a global concurrency cap (semaphore) plus a per-domain delay, so many domains run in parallel while any single host is touched gently.
• Checks robots.txt before fetching (cached per domain) and sends an honest identifying User-Agent with a contact URL, never a spoofed browser string.
• Retries transient failures (timeout, 5xx, transport errors) with exponential backoff; permanent failures return a clean FetchResult instead of crashing the batch.
• Ships email_extract.py, a generic-mailbox extractor that keeps only role inboxes (info, contact, hello) and skips personal addresses, plus requirements.txt.

Why use this

The delay and concurrency cap are what keep you off blocklists, and the role-inbox gate is what keeps B2B contact-gathering defensible. This is the actual fetch layer a Belgian AI agency uses to enrich prospects without behaving like a bad bot, generalised into a reusable module: you bring the parsing, it makes the fetching well-behaved.

See it in action

Prompt:

I have a few hundred business URLs and I want to fetch them without getting blocked, and pull only the general contact email from each.

Result:

from polite_scraper import PoliteScraper
from email_extract import extract_generic_emails

async with PoliteScraper(
    max_concurrency=2,
    per_domain_delay=3.0,
    user_agent=MyBot/1.0 (+https://example.com/bot),
    timeout=15.0,
    max_retries=3,
    respect_robots=True,
) as scraper:
    results = await scraper.fetch_many(urls)

for r in results:
    if r.ok:
        emails = extract_generic_emails(r.text)
        # keeps info@, contact@, hello@; drops jan.peeters@

Use cases

• Enriching a prospect list by fetching a few hundred business websites overnight without tripping rate limits.
• Adding timeouts, retries, and backoff to an existing crawler that dies on one bad URL.
• Gathering only published role inboxes for B2B outreach, skipping personal addresses.
• Making an internal batch fetcher a visible, well-behaved robots.txt citizen.

Known limitations

• Fetch layer only: parsing beyond the email extractor is your concern.
• Fetches raw HTML; JS-rendered sites need a headless browser tool instead.
• Deliberately slow by design; it is built to avoid high-speed bulk crawling, not to enable it.

FAQ

What exactly is in the download?

SKILL.md, three assets (polite_scraper.py, email_extract.py, requirements.txt), and references/politeness-rules.md explaining why each control matters and how to tune it.

Which agents does it work with?

Claude Code, Claude Cowork, Cursor, Codex CLI, and any agent that reads the SKILL.md format.

Do I need anything else?

Python with httpx (the only dependency). No API keys, no proxy service.

Why does the email extractor skip personal addresses?

A published business inbox like info@ is defensible for B2B outreach; harvesting a named person's address is what gets you blocked and is hard to justify under GDPR-oriented practice.