scrapling technical seo crawler
by Roy Yuen
Professional technical SEO crawler and site auditor for deep architectural analysis and issue detection.
New: Skill bounties are live. Post a request, fund the bounty, and creators compete for 7 days to build it -> See open bounties
THE AGENSI STORE
6 skills found
by Roy Yuen
Professional technical SEO crawler and site auditor for deep architectural analysis and issue detection.
by Joker
Python project routing for Web API/crawler/automation/data/CLI. FastAPI/Django/Flask selection, async patterns, deployment strategies.
Check your robots.txt and sitemap.xml for the mistakes that quietly block crawlers. Flags a site-wide Disallow: /, malformed directives, a missing Sitemap: line, sitemap XML that is not well-formed, <loc> URLs missing a scheme, mixed http and https, and sitemaps over the 50,000-URL or 50MB limits. Reads local files or pasted text and never touches the live site.
Catch the Open Graph and Twitter Card problems that make a shared link preview blank or broken. Flags missing og:title, og:image, og:description, og:url, or twitter:card, a relative og:image URL (crawlers need an absolute one), an SVG og:image (no platform renders it), and image dimensions that miss the 1200x630 standard. Reads your HTML or pasted head markup; it does not fetch the page or refresh a platform's cache.
Generate a spec-compliant llms.txt (and optional llms-full.txt) for your site or repo so AI agents and crawlers can navigate it. Curates the pages that matter, writes the exact llmstxt.org structure — single H1, blockquote summary, and link sections in the precise format agents parse — then validates the format and tells you where to put it. The honest version: a low-cost, machine-readable surface for the agentic web, not an overhyped SEO trick.
Audit and optimize website visibility, reachability, and actionability for AI agents and LLM crawlers.