More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    vLLM: High-Throughput LLM Serving

    1

    Serve open models in production with vLLM: paper-first VRAM capacity math, a four-rung deployment ladder, online vs. offline modes, quantization checks, and load-tested limits.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    vLLM: High-Throughput LLM Serving

    vLLM: High-Throughput LLM Serving

    Example session with this skill installed

    We need to serve a 32B open model for an internal document-QA tool. About 40 engineers, documents up to roughly 30k tokens, and we have two 48GB GPUs. Latency should feel responsive but this is internal, not customer-facing. Plan the deployment.

    • Read your context and instructions
    • Compiled the vllm

    Capacity math first. At FP16 the weights alone are 32B × 2 bytes = 64GB. Two 48GB GPUs give 96GB total, so with tensor parallelism across both cards the weights fit with about 32GB left for KV cache and overhead — workable, but tight for 30k-token contexts at real concurrency. The alternative worth pricing before you commit: an AWQ 4-bit checkpoint lands near 18GB, fits on one GPU, and leaves the second card free — at the cost of a quantization quality check you must actually run. Given the 30k context requirement, the plan takes the FP16 two-GPU route and keeps AWQ as the fallback if the load test shows KV pressure you cannot afford.

    Configuration, by the ladder: rung 1 sanity check for coherent output with the right chat template; then tensor-parallel size 2, GPU memory utilization 0.9, and max model length set to 32768 — your real document size, not the model's maximum. Verify both GPUs show utilization during a request; an idle second card means the parallelism flag did not take.

    Load test with real documents, including the 30k-token worst case, ramping concurrency in steps. Expect the knee earlier than intuition says — long contexts make KV cache, not weights, the binding constraint. Set the concurrency cap below the measured knee (expect somewhere in the 8–16 range, but the test decides, not this plan), and define over-cap behavior as short queueing, then a fast, clear busy error.

    Production rung: containerized, request timeout 300 seconds for the longest documents, metrics endpoint on, health check wired. Document the cap and the fallback: if KV pressure bites in week one, the pre-approved moves are capping context at 24k or switching to the verified AWQ checkpoint — decided now, not during an incident.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A model that runs in a notebook is not a model that serves. Production serving fails in predictable places: nobody did the VRAM math, a laptop configuration got deployed as a production one, the context window was set to the brochure maximum and ate the KV cache, or the first real traffic found the concurrency knee that a load test would have found first. This skill is a runbook that prevents those failures in order. It starts with an inputs/outputs contract and a stop rule — a single-user experiment does not need a server. Step one is capacity math on paper: weights at your precision, KV cache growth with context and concurrency, and a hard headroom rule, captured in a fill-in capacity worksheet. Step two climbs a four-rung deployment ladder — local sanity, configured single GPU, verified multi-GPU, then production with timeouts, metrics, health checks, and a container — with an explicit warning against skipping rungs. Step three chooses the serving mode with a comparison table: online API serving tuned for latency, or offline batch inference through the in-process class when you have a prompt list and a deadline instead of users. Step four covers quantization strategy (AWQ/GPTQ 4-bit, FP8) with the non-negotiable rule that a quantized checkpoint gets verified against the full-precision baseline on your own evals. Step five is load-test discipline: representative prompts including your longest real case, a gradual concurrency ramp, and the max-concurrency cap set below the measured knee. A complete worked example (an 8B model served to a team on one 24GB GPU), a pre-launch checklist, a five-dimension readiness rubric, an extended troubleshooting table, and edge cases from giant-prompt stalls to context creep close it out.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for DevOps & Cloud.

    • It plans and validates deployments; it does not provision servers or run the load test for you.
    • Capacity figures are planning estimates — your measured load test is the arbiter, and the skill says so.
    • It is vLLM-specific in its configuration guidance; the math and discipline transfer to other engines, the flags do not.
    • It covers single-service deployments, not multi-region routing, autoscaling policy, or model selection.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    Internal users are still concurrent users, and the failure the load test prevents — the server falling over at the exact moment the team relies on it — does not check whether the audience was internal. The skill's load test is deliberately small: representative prompts, a gradual ramp, one recorded knee.

    No — that is one of the named anti-patterns. Unused context capacity is reserved KV cache: it lowers the concurrency you can serve and raises your OOM risk under load. Set max model length to what your traffic actually needs, and revisit it when the prompt-length distribution changes.

    Whenever the workload is a list with a deadline rather than users with latency expectations — dataset labeling, evaluation runs, generating distillation data. The mode table in the skill makes the call, and offline removes the HTTP layer entirely, so throughput is not close between the two for that shape of work.

    New in DevOps & Cloud