Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    onlinewebdirectory

    1

    Predict bot blocks, identify required fetch strategies, and discover machine-readable interfaces via a shared registry.

    Free

    Security scanned
    onlinewebdirectory

    onlinewebdirectory

    Example session with this skill installed

    I need to pull the product categories from g2.com. Before you fetch anything, check how g2.com treats agents and pick the right way in.

    • Read your context and instructions
    • Compiled the onlinewebdirectory

    g2.com: checked before fetching

    Short answer: don't scrape the pages. g2.com puts a Cloudflare bot challenge in front of plain HTTP clients, but it publishes an llms.txt, so I read that instead.

    What the directory says about g2.com

    FactValueSource
    AccessBot challengedRegistry check, 25 Oct 2026
    Bot wallCloudflareRegistry check
    Robots rulesRestricts some pathsRegistry check
    Machine-readable alternativellms.txt at /llms.txtRegistry check

    Fetch strategy returned: use_machine_interface, prefer /llms.txt, expect a Cloudflare challenge on normal pages.

    What I did

    1. Skipped the HTML pages, which would have hit the Cloudflare challenge and wasted a headless browser run.
    2. Fetched https://g2.com/llms.txt with a plain HTTP request.
    3. Extracted the category sections from it.

    After fetching

    I reported what I saw back to the directory (llms.txt present and readable, signed with my key), so the next agent that checks g2.com gets a better-verified answer.

    Next steps

    • If you need detail that llms.txt doesn't cover, I can use a real browser for those specific pages only.
    • Want me to check the other sites on your list the same way first?

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Scraping the web with agents often leads to wasted resources, blocked requests, and unnecessary DOM parsing. Websites change their bot policies without notice, leaving your scrapers to fail silently or hit expensive paywalls.

    This skill provides a programmatic interface to a shared registry of how websites treat agents. It allows your agents to check a site's access posture before attempting a fetch, ensuring they choose the right client—whether that is a plain HTTP request, a headless browser, or a machine-readable interface like llms.txt. By contributing observations back to the registry, your agents help maintain a crowd-verified map of the web's bot-friendliness.

    What it does

    • Pre-fetch lookup checks if a domain blocks agents or requires JavaScript before you spend compute.
    • Client selection identifies the specific bot wall (Cloudflare, DataDome, etc.) to help choose the correct impersonation strategy.
    • Interface discovery locates llms.txt, RSS feeds, or sitemaps so you can skip HTML scraping entirely.
    • Crowd verification tracks whether observations are verified by multiple agents or are single-report hints.
    • Observation reporting lets agents sign and submit facts about rate limits, challenges, and robots.txt posture.

    How it works

    1. Lookup origin: Call lookup for one or more domains to retrieve the current strategy.fetch and access_posture.
    2. Evaluate facts: Check strategy.basis to determine if the data is crowd-verified or a pending registry check.
    3. Execute fetch: Use the recommended client (e.g., headless_chrome or plain_http) based on the lookup results.
    4. Submit report: Record what actually happened during the fetch to update the registry and earn points.

    Frameworks & tools

    Built with Node.js. It supports integration via a CLI wrapper (owd.mjs), a remote MCP server, or a direct REST API.

    Why this beats prompting it yourself

    LLMs have outdated knowledge of website structures and bot defenses. This skill provides real-time, crowd-sourced intelligence that prevents agents from repeatedly hitting known blocks or missing hidden machine-readable endpoints.

    Use cases

    • Cost optimization by skipping domains known to have hard paywalls or login requirements.
    • Reliable scraping by identifying exactly which sites require a headless browser for JavaScript execution.
    • Data enrichment by automatically finding and utilizing llms.txt or sitemaps instead of noisy HTML.

    Known limitations

    Requires Node.js for the CLI version. Facts can go stale after 14 to 90 days, requiring fresh reports to maintain accuracy. Free lookups are limited to 100 per day per key.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires Node.js for CLI. Facts expire after 14 to 90 days. Daily lookup quota applies. Data reliability depends on recent crowd reports.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 17 days ago

    • Free with an Agensi account

    Accounts
    Onlinewebdirectory

    Permissions

    Terminal / Shell
    Network Access
    Write Files
    Read Files
    Environment Variables

    Allowed Hosts

    onlinewebdirectory.net

    File Scopes

    ~/.config/onlinewebdirectory/**
    Listed17 days ago

    What's inside

    Frequently Asked Questions

    This skill acts as an intelligence layer that checks a domain's bot defenses (like Cloudflare or DataDome) before your agent attempts a fetch, saving you from wasted API costs and blocked requests.

    It is compatible with any agent framework that supports MCP (Model Context Protocol), Node.js environments, or REST API calls, making it easy to integrate with LangChain, AutoGPT, or custom-built agents.

    The registry identifies 'low-noise' alternatives like llms.txt, RSS feeds, and sitemaps, allowing your agent to extract structured data without performing expensive and error-prone HTML parsing.

    While the registry is crowd-verified and constantly updated, data can be flagged as 'pending' or 'stale' after 14-90 days, prompting your agent to verify the current status and report back.

    You receive the Node.js CLI wrapper (`owd.mjs`), access to the remote MCP server for direct LLM integration, and a dedicated API key for performing up to 100 daily registry lookups.

    New in LLM Ops

    Free