More screenshots
Works with the AI Agents you already use
SGLang: Structured LLM Serving with Prefix Caching
Serve LLMs with structured generation and a radix-tree prefix cache — JSON and tool calls that parse, agentic loops that run 5-10x faster, and tuning by workload.
$5
SGLang: Structured LLM Serving with Prefix Caching
Example session with this skill installed
Our agent calls tools 20 times per task with the same 2k-token system prompt, and every call pays full price. Make it fast, and make its tool calls parse every time.
- Read your context and instructions
- Compiled the sglang
Measured first: prefix hit rate on the agent workload was under 10 percent, because a timestamp and a request ID were embedded at the top of every prompt, so no two prompts ever shared a prefix. The prompt was restructured — system prompt, tool definitions, and examples first, variable user input last — and the hit rate jumped past 85 percent on the same workload. The 2k-token system prompt is now computed once per server, not once per tool call. Tool calling was moved to constrained JSON: the model emits the tool name and arguments against a tight schema, with optional parameters cut to the minimum so there is nothing to hallucinate. Output is still validated against the schema — constraints reduce errors, they do not eliminate them. Throughput on the agent loop rose roughly 6x on the same hardware, and tool-call parse failures fell from about one in twenty to zero in a hundred test calls. A warm-up request now runs before the server joins the load balancer, so the first real request is never the cold one. Memory is watched with an eviction policy set, because the cache grows with unique-prefix volume. The one honest limit is recorded: the cache is per-server and in memory — it does not survive restarts or share across replicas, so a deploy still pays one cold round. The tuning table for this workload is saved with the deployment notes, so the next engineer inherits the reasoning, not just the flags.
Connects securely to your tools. The creator never sees your data.
About this skill
Most serving stacks treat every request as independent text. SGLang's bet is different: real workloads repeat themselves — system prompts, few-shot examples, conversation history — and structured outputs like JSON and tool calls are the actual product. This skill covers the radix-tree prefix cache: designing prompts for prefix sharing, why agentic loops are the killer workload, and how multi-turn chat gets relatively cheaper as it grows. It covers structured generation as a feature — JSON mode, regex and grammar constraints, tool calling — with the discipline of constraining the shape, not the content. It also covers deployment: launch and sizing, continuous batching, speculative decoding, the OpenAI-compatible endpoint, a workload tuning guide, and the failure modes — cold start, cache thrash, over-constrained grammars, and memory growth — that show up in production.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
It covers SGLang specifically; the prefix-cache design assumes repeated prompt structure — wildly diverse prompts get zero cache benefit.
Cache is per-server and in memory; it does not survive restarts or share across replicas.
Speculative decoding needs a second draft model; the latency win is workload-dependent and must be measured.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
No special permissions declared or detected