Works with the AI Agents you already use
agentic traffic classifier
Classify web traffic into 13 categories to distinguish between AI agents, training crawlers, bots, and humans.
Free
agentic traffic classifier
Example session with this skill installed
Analyze the attached access.log from last Tuesday. I need to know how many AI agents are visiting compared to humans and if any scrapers are pretending to be Google.
- Read your context and instructions
- Compiled the agentic traffic classifier
Analysis complete. I processed 45,200 sessions from your access.log. The traffic shows a significant presence of AI-assisted browsing and two persistent scrapers using forged user agents.
Who visits example.com (2023-10-24 00:00 → 23:59)
People 62.4% · AI agents 12.8% · AI crawlers 15.2% · classic bots 8.1% · invalid 1.5%
AI companies
| Operator | Agents (sessions) | Crawlers (sessions) | Referred humans |
|---|---|---|---|
| OpenAI | 2,410 | 3,105 | 450 |
| Anthropic | 1,102 | 1,890 | 112 |
| Perplexity | 2,240 | 1,050 | 890 |
| 45 | 895 | 2,105 |
Classes
| Class | Sessions | Requests | What to do |
|---|---|---|---|
| ai-agent-user-fetch | 5,752 | 18,200 | Allow - customer-facing |
| ai-crawler-training | 6,890 | 82,400 | Block via robots.txt |
| fake-crawler | 678 | 12,450 | Block at WAF (IP level) |
| human-via-ai | 3,557 | 24,100 | Tag in Analytics |
Inferred agents to review
Session 0xf3a2: Identified as ai-agent-browser (Medium Confidence). Browser on AWS network, 45s irregular intervals between page loads, submitted POST /checkout.
Session 0xbc41: Identified as ai-agent-browser (Low Confidence). Headless Chrome on DigitalOcean, but only 2 requests.
Policy
- robots.txt: Add
User-agent: GPTBot Disallow: /andUser-agent: CCBot Disallow: /. - WAF: Block IPs flagged in the
fake-crawlerclass (see detailed log for IP list). - Analytics: Create a custom segment for
human-via-aiusingReferermatchingperplexity.ai|chatgpt.com.
Next steps
- Export the list of
fake-crawlerIPs to your firewall blocklist. - Update robots.txt to prevent model training while maintaining AI search visibility.
- Enable
Signature-Agentheader logging in Cloudflare to capture authenticated agents.
Connects securely to your tools. The creator never sees your data.
About this skill
Modern web logs are no longer just humans and Googlebot. AI agents, training crawlers, and LLM-driven browsers now represent a significant portion of traffic, often disguised as standard users or classic bots. This skill provides the forensics needed to distinguish between a helpful AI agent acting for a customer and a scraper harvesting data for a model.
What it does
- Traffic classification sorts every session into 13 specific classes including signed agents, user-fetch assistants, and training crawlers.
- Agent browser inference identifies AI-driven headless browsers by analyzing hosting network data, asset loading patterns, and task-like pacing.
- Crawler verification performs reverse and forward DNS lookups to expose fake crawlers and disguised bots.
- Operator identification maps traffic to specific AI companies like OpenAI, Anthropic, and Perplexity.
- Policy generation creates ready-to-use robots.txt, WAF rules, and analytics segments based on the traffic mix.
How it works
- Provide the logs in access.log, CSV, or JSON format, ideally including headers like
signature_agentandmethod. - Execute the analysis using either the connected MCP endpoint for network-enriched data or the bundled Node.js script.
- Review the confidence scores for inferred agent sessions, checking evidence like ASN data and session timing.
- Generate the policy using the provided templates to treat agents, crawlers, and humans differently.
Frameworks & tools
Built for Node.js 20+ environments. Integrates with MCP (Model Context Protocol) via a dedicated endpoint or runs as a standalone script using iptoasn data for network intelligence.
Why this beats prompting it yourself
Generic prompts fail to distinguish between ChatGPT-User (an agent acting for a person) and GPTBot (a training crawler), leading to poor blocking decisions. This skill uses a codified logic engine and DNS verification to ensure every classification is backed by evidence rather than guesswork.
Use cases
- Auditing how much site traffic is actually AI agents vs. humans.
- Creating a robots.txt that allows AI search but blocks training scrapers.
- Identifying high-value customers arriving via AI assistant referrals.
- Protecting site resources from disguised bots and fake crawlers.
Known limitations
Classification accuracy depends on log quality. Inferred agent detection requires the ASN table for network identification. Signature verification must be performed at the edge to be authoritative.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires Node 20+ for local script. DNS verification requires network access. Inference accuracy depends on presence of ASN data and session duration.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 2 days ago
- Free with an Agensi account