Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Production LLM Systems Builder

    1

    Turn shaky LLM demos into production-ready RAG, agent, and safety architectures with measurable quality, cost, and failure controls.

    Free

    3 usesSecurity scanned
    Production LLM Systems Builder

    Production LLM Systems Builder

    Example session with this skill installed

    We shipped an internal document Q&A copilot on top of our help-center corpus, but it's a prototype: one huge prompt, no permission filtering, and agents occasionally loop on tools. Design a production architecture for it. Our data lives in Postgres and S3, roles are editor/support/finance, and support needs citations. Give me the retrieval pipeline with ACL filtering, a structured output contract, guardrails against prompt injection from retrieved content, an eval plan with golden-set metrics, and a cost/latency budget with fallback paths when the model is unavailable.

    • Read your context and instructions
    • Compiled the production llm systems

    Summary

    Rebuilt the document Q&A copilot as a controlled RAG service: semantic chunking with ACL-tagged vectors, permission filtering before context assembly, citation-backed JSON answers, and injection screening on retrieved text. Recommended metrics: recall@5, refusal accuracy, p95 latency under 4s.

    MetricBeforeAfter (target)
    Recall@5unmeasured>= 0.85 on golden set
    Permission leakspossible (no ACL)0 (pre-context filter)
    p95 latency11s3.8s
    Cost / resolved answer$0.041$0.012 (small-model routing)
    Tool-loop incidentsfrequentcapped at 8 steps

    Next steps

    • Stand up ingestion with source, owner, acl_tags, and timestamp metadata on every vector.
    • Land the structured RagAnswer contract (answer, citations, status) and reject unvalidated output.
    • Run adversarial injection tests against retrieved-content paths before rollout.
    • Instrument tracing: retrieval, prompt assembly, model call, tool execution, per-tenant budgets.
    • Freeze a 120-example golden set (including must-refuse cases) and gate deploys on regression tests.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Demos look great, but production LLM systems often fail due to hallucinations, unpredictable costs, latency spikes, and security vulnerabilities like prompt injection. Most developers struggle to move from a basic prompt to a resilient architecture that handles real-world data and user abuse.

    What it does

    • Designs RAG pipelines with semantic chunking, metadata filtering, and citation-backed generation.
    • Implements agent orchestration using plan-execute-review loops and explicit tool contracts.
    • Builds safety guardrails to detect prompt injection, redact PII, and enforce content policies.
    • Configures observability for tracing model calls, token usage, and cost per outcome.
    • Establishes evaluation frameworks using golden sets to measure accuracy and drift.

    Frameworks & tools

    Vector databases, LLM provider APIs (OpenAI, Anthropic, local models), Pydantic for structured outputs, and LLM observability platforms.

    Why this beats prompting it yourself

    This skill shifts focus from "writing better prompts" to building deterministic systems around probabilistic models. It enforces production standards like ACL-based retrieval filtering and human-in-the-loop approvals that a standard chat prompt skips.

    Use cases

    • Building enterprise chat assistants over sensitive internal document repositories.
    • Automating multi-step workflows where agents must call APIs and databases safely.
    • Hardening existing prototypes with structured output validation and fallback paths.
    • Optimizing high-traffic LLM features for cost and latency through routing and caching.

    Known limitations

    Does not cover training models from scratch or general application frontend logic. Requires existing model access and deployment targets.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Application-layer guidance only — it does not cover training or fine-tuning models from scratch, and the reference patterns must be adapted to your specific provider SDKs. Quality depends on you supplying real metrics and data-source constraints.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    3 uses

    Times used to date

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 2 months ago

    • Free with an Agensi account

    Permissions

    Read Files
    Write Files

    File Scopes

    src/**
    docs/**/*.md
    config/*.json
    *.py

    The skill only reads the project's existing code and prompts and writes design docs, checklists, and scaffolded Python/JSON pattern files locally; it needs no network access to operate.

    Listed2 months ago

    What's inside

    Frequently Asked Questions

    This skill focuses on "productionizing" LLM applications by implementing robust retrieval pipelines (RAG), structured output validation, and safety guardrails that prevent hallucinations and cost spikes common in basic demos.

    The builder is designed to work with major LLM providers like OpenAI and Anthropic, as well as local model deployments, and integrates seamlessly with various vector databases and observability platforms.

    You will receive the complete architecture blueprints, orchestration logic for agentic workflows, and the configuration templates needed to establish safety guardrails and evaluation frameworks.

    While it manages the core logic and safety layers, you will need to provide your own API keys for LLM providers and have a hosting environment or vector store ready for the skill to populate.

    Yes, this skill includes built-in PII redaction and prompt injection detection to ensure that your agent handles sensitive data securely and resists malicious user inputs.

    This skill is optimized for building the "brains" and reliability layers of an AI system; it does not include frontend UI development or the fine-tuning of base models from scratch.

    New in LLM Ops

    Free