Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    agentic traffic classifier

    1

    Classify web traffic into 13 categories to distinguish between AI agents, training crawlers, bots, and humans.

    Free

    Security scanned
    agentic traffic classifier

    agentic traffic classifier

    Example session with this skill installed

    Analyze the attached access.log from last Tuesday. I need to know how many AI agents are visiting compared to humans and if any scrapers are pretending to be Google.

    • Read your context and instructions
    • Compiled the agentic traffic classifier

    Analysis complete. I processed 45,200 sessions from your access.log. The traffic shows a significant presence of AI-assisted browsing and two persistent scrapers using forged user agents.

    Who visits example.com (2023-10-24 00:00 → 23:59)

    People 62.4% · AI agents 12.8% · AI crawlers 15.2% · classic bots 8.1% · invalid 1.5%

    AI companies

    OperatorAgents (sessions)Crawlers (sessions)Referred humans
    OpenAI2,4103,105450
    Anthropic1,1021,890112
    Perplexity2,2401,050890
    Google458952,105

    Classes

    ClassSessionsRequestsWhat to do
    ai-agent-user-fetch5,75218,200Allow - customer-facing
    ai-crawler-training6,89082,400Block via robots.txt
    fake-crawler67812,450Block at WAF (IP level)
    human-via-ai3,55724,100Tag in Analytics

    Inferred agents to review

    Session 0xf3a2: Identified as ai-agent-browser (Medium Confidence). Browser on AWS network, 45s irregular intervals between page loads, submitted POST /checkout.

    Session 0xbc41: Identified as ai-agent-browser (Low Confidence). Headless Chrome on DigitalOcean, but only 2 requests.

    Policy

    • robots.txt: Add User-agent: GPTBot Disallow: / and User-agent: CCBot Disallow: /.
    • WAF: Block IPs flagged in the fake-crawler class (see detailed log for IP list).
    • Analytics: Create a custom segment for human-via-ai using Referer matching perplexity.ai|chatgpt.com.

    Next steps

    1. Export the list of fake-crawler IPs to your firewall blocklist.
    2. Update robots.txt to prevent model training while maintaining AI search visibility.
    3. Enable Signature-Agent header logging in Cloudflare to capture authenticated agents.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Modern web logs are no longer just humans and Googlebot. AI agents, training crawlers, and LLM-driven browsers now represent a significant portion of traffic, often disguised as standard users or classic bots. This skill provides the forensics needed to distinguish between a helpful AI agent acting for a customer and a scraper harvesting data for a model.

    What it does

    • Traffic classification sorts every session into 13 specific classes including signed agents, user-fetch assistants, and training crawlers.
    • Agent browser inference identifies AI-driven headless browsers by analyzing hosting network data, asset loading patterns, and task-like pacing.
    • Crawler verification performs reverse and forward DNS lookups to expose fake crawlers and disguised bots.
    • Operator identification maps traffic to specific AI companies like OpenAI, Anthropic, and Perplexity.
    • Policy generation creates ready-to-use robots.txt, WAF rules, and analytics segments based on the traffic mix.

    How it works

    1. Provide the logs in access.log, CSV, or JSON format, ideally including headers like signature_agent and method.
    2. Execute the analysis using either the connected MCP endpoint for network-enriched data or the bundled Node.js script.
    3. Review the confidence scores for inferred agent sessions, checking evidence like ASN data and session timing.
    4. Generate the policy using the provided templates to treat agents, crawlers, and humans differently.

    Frameworks & tools

    Built for Node.js 20+ environments. Integrates with MCP (Model Context Protocol) via a dedicated endpoint or runs as a standalone script using iptoasn data for network intelligence.

    Why this beats prompting it yourself

    Generic prompts fail to distinguish between ChatGPT-User (an agent acting for a person) and GPTBot (a training crawler), leading to poor blocking decisions. This skill uses a codified logic engine and DNS verification to ensure every classification is backed by evidence rather than guesswork.

    Use cases

    • Auditing how much site traffic is actually AI agents vs. humans.
    • Creating a robots.txt that allows AI search but blocks training scrapers.
    • Identifying high-value customers arriving via AI assistant referrals.
    • Protecting site resources from disguised bots and fake crawlers.

    Known limitations

    Classification accuracy depends on log quality. Inferred agent detection requires the ASN table for network identification. Signature verification must be performed at the edge to be authoritative.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires Node 20+ for local script. DNS verification requires network access. Inference accuracy depends on presence of ASN data and session duration.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    • Free with an Agensi account

    Permissions

    Network Access
    Read Files
    Listed2 days ago

    What's inside

    Frequently Asked Questions

    The skill distinguishes between 13 categories, including training crawlers (e.g., GPTBot), user-initiated agents (e.g., ChatGPT-User), search assistants (e.g., Perplexity), and standard human traffic. This ensures you can block data scrapers while still allowing AI assistants that bring you customers.

    The tool is designed for Node.js 20+ environments and accepts logs in access.log, CSV, or JSON formats. It can be deployed as a standalone script for batch processing or integrated into your workflow via the Model Context Protocol (MCP) for real-time network intelligence.

    Yes, the skill uses a combination of forward and reverse DNS lookups and ASN (Autonomous System Number) data to expose bots that spoof their User-Agent headers to look like legitimate crawlers or humans.

    You receive the core classification engine, a dedicated MCP endpoint for network-enriched analysis, a Node.js execution script, and pre-built templates for generating robots.txt and WAF rules.

    While standard log analysis is highly accurate, the 'Agent Browser Inference' feature provides a confidence score based on hosting data and session pacing to help you make informed decisions on traffic that lacks a clear signature.

    The classifier is regularly updated to include the latest signatures and IP ranges from major AI operators like OpenAI, Anthropic, and Google, ensuring your traffic segments remain accurate as new agents emerge.

    New in LLM Ops

    Free