Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    llm inference optimization

    1

    Systematic optimization for LLM latency, throughput, and memory usage across major serving frameworks.

    $5

    /mo

    Subscription · cancel anytime

    Secure checkout via Stripe

    • Always the latest version
    • Delivered live to your agent via MCP
    • Cancel anytime, access ends at period end
    Security scanned
    llm inference optimization

    llm inference optimization

    Example session with this skill installed

    My Llama-3-70B instance is hitting OOM on an A100 and the latency is too high for real-time chat. How do I optimize this for vLLM?

    • Read your context and instructions
    • Compiled the llm inference optimization

    I have analyzed your deployment constraints for Llama-3-70B on an A100. To resolve the Out-of-Memory (OOM) errors and reduce latency, you should implement a combination of 4-bit quantization and PagedAttention optimizations.

    Optimization Strategy for Llama-3-70B

    ComponentRecommendationImpact
    QuantizationAWQ 4-bit (W4A16)~50% VRAM reduction
    Serving FrameworkvLLM with PagedAttentionEliminates memory fragmentation
    KV CacheFP8 KV Cache ScalingReduces memory footprint of long contexts
    BatchingContinuous BatchingIncreases throughput under load
    AttentionFlashAttention-2Optimizes prefill and decode speed

    Recommended Deployment Command

    vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
      --quantization awq \
      --dtype half \
      --gpu-memory-utilization 0.95 \
      --max-model-len 4096 \
      --enable-prefix-caching \
      --kv-cache-dtype fp8
    

    Next steps

    1. Benchmark TTFT and TPOT using the vLLM benchmark suite to ensure the AWQ quantization meets your quality thresholds.
    2. Monitor GPU memory fragmentation via the vLLM metrics endpoint to tune gpu-memory-utilization.
    3. Enable speculative decoding with a Llama-3-8B draft model if TPOT remains above 50ms.
    $5/mo

    Subscription · always the latest version

    • Secure checkout via Stripe
    • Cancel anytime
    • Delivered live via MCP

    About this skill

    High latency and memory bottlenecks stall LLM deployments in production. This skill provides a systematic workflow to reduce Time to First Token (TTFT), increase tokens per second, and minimize GPU memory consumption.

    What it does

    • Performance profiling identifies bottlenecks in TTFT, TPOT, and memory usage before applying optimizations.
    • Quantization selection guides the implementation of FP8, AWQ, or GPTQ based on hardware constraints and quality requirements.
    • Serving stack optimization matches workloads to frameworks like vLLM, SGLang, or TensorRT-LLM for maximum throughput.
    • KV cache management implements PagedAttention and prefix caching to eliminate memory fragmentation.
    • Decoding acceleration uses speculative decoding and FlashAttention kernels to speed up token generation.

    How it works

    1. Analyze metrics using the profiling step to establish baselines for latency and GPU utilization.
    2. Select framework and quantization levels based on the specific model size and available hardware.
    3. Configure server parameters for batching, scheduling, and cache reuse to handle concurrent requests.

    Frameworks & tools

    Supports vLLM, SGLang, TensorRT-LLM, HuggingFace TGI, and llama.cpp. Compatible with NVIDIA Hopper architectures for FP8 and various quantization methods including AWQ and GPTQ.

    Why this beats prompting it yourself

    Generic prompts often miss hardware-specific optimizations like FlashDecoding or prefix-aware batching. This skill provides a structured decision tree that balances the trade-offs between model quality, speed, and cost without blind trial and error.

    Use cases

    • Reducing the serving cost of 70B+ parameter models on limited VRAM.
    • Optimizing agentic workflows that rely on heavy system prompt reuse.
    • Improving the responsiveness of real-time chat applications by lowering TTFT.
    • Scaling throughput for high-volume batch processing tasks.

    Known limitations

    Optimization results depend heavily on specific GPU architecture. Higher levels of quantization like W4A4 may result in noticeable model quality degradation.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Hardware dependent. Results vary by GPU architecture. Aggressive quantization (e.g., W4A4) may degrade output quality. Requires initial profiling to be effective.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 22 days ago

    Permissions

    Terminal / Shell
    Listed22 days ago

    What's inside

    Frequently Asked Questions

    This skill provides a systematic framework to resolve high latency (TTFT), low throughput, and GPU memory bottlenecks that typically stall large-scale LLM deployments.

    Yes, the skill is designed to work across all major serving frameworks including vLLM, SGLang, TensorRT-LLM, HuggingFace TGI, and llama.cpp.

    You will receive a structured optimization workflow, decision trees for quantization selection, and configuration guides for high-concurrency batching and KV cache management.

    While optimized for NVIDIA Hopper (H100/H200) for FP8 support, the workflows include strategies for AWQ and GPTQ quantization compatible with older Ampere and Turing architectures.

    The skill includes a 'Quality vs. Speed' matrix to help you choose the right quantization level (like FP8 vs. Int4) to ensure latency gains don't compromise your model's accuracy.

    New in LLM Ops