Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    anvil

    1

    Instrument Python/TS apps, track LLM costs, and enforce CI quality gates with Anvil observability.

    Free

    Security scanned
    anvil

    anvil

    Example session with this skill installed

    I need to instrument a TypeScript RAG pipeline and add a CI gate that fails if the hallucination score drops below 90 percent.

    • Read your context and instructions
    • Compiled the anvil

    I have updated your TypeScript pipeline to include multi-span waterfall tracing and generated a GitHub Actions workflow to gate your pull requests.

    Anvil Instrumentation and CI Config

    // src/lib/llm.ts
    import * as anvil from "@anvil/sdk";
    
    anvil.init({ apiKey: process.env.ANVIL_API_KEY! });
    
    export async function ragPipeline(question: string, userId: string) {
      return await anvil.trace("qa-pipeline", { input: question, userRef: userId }, async (t) => {
        const context = await anvil.span("vector-retrieval", { type: "retrieval" }, async () => {
          return await vectorDb.query(question);
        });
    
        const completion = await anvil.span("llm-generation", { type: "llm" }, async () => {
          return await openai.chat.completions.create({
            model: "gpt-4o",
            messages: [{ role: "user", content: `Context: ${context}\n\nQuestion: ${question}` }]
          });
        });
    
        t.output = completion.choices[0].message.content;
        return { answer: t.output, traceId: t.id };
      });
    }
    
    
    # .github/workflows/anvil-eval.yml
    name: LLM Quality Gate
    on: [pull_request]
    jobs:
      eval:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - name: Run Anvil Evals
            run: |
              npx @anvil/cli eval run \
                --dataset "golden-rag-set" \
                --grader hallucination \
                --grader-config '{"model": "claude-3-5-sonnet-20241022"}' \
                --min-score 0.90 \
                --max-regression 0.05
            env:
              ANVIL_API_KEY: ${{ secrets.ANVIL_API_KEY }}
    

    Next steps

    • Ensure ANVIL_API_KEY is added to your GitHub Repository Secrets.
    • Pass the traceId returned by ragPipeline to your frontend to enable anvil.feedback() calls.
    • Verify that the golden-rag-set dataset exists in your Anvil dashboard.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Production LLM features often lack the visibility needed to debug failures or prevent regressions. This skill provides the instrumentation and evaluation framework required to monitor, trace, and gate LLM applications using Anvil.

    What it does

    • Tracing instrumentation adds non-blocking execution tracking to Python and TypeScript LLM chains to capture latency, costs, and traces.
    • Feedback collection wires user-facing thumbs up/down signals directly to specific trace IDs for failure triaging.
    • Dataset construction builds evaluation sets from real-world production complaints captured in the Failure Inbox.
    • Automated grading configures deterministic checks like regex and JSON validation alongside LLM judges for accuracy and hallucination.
    • CI quality gating integrates with GitHub Actions to block pull requests that breach regression thresholds or minimum scores.

    How it works

    1. Initialize the SDK in your application using anvil.init() with your API key and self-hosted URL.
    2. Wrap your LLM functions with @anvil.observe or anvil.trace to begin asynchronous shipping of traces.
    3. Configure graders using the anvil eval run CLI or YAML configs to define your quality bar.
    4. Deploy the provided GitHub Actions template to enforce score thresholds on every pull request.

    Frameworks & tools

    Python (SDK), TypeScript/Node.js (SDK), LangChain integration, OpenAI, Anthropic, Google Gemini, and local models. Supports GitHub Actions for CI/CD workflows.

    Why this beats prompting it yourself

    Prompting an LLM to "be an evaluator" results in inconsistent scoring without statistical rigor. This skill implements specific instrumentation patterns, idempotent ingestion, and bootstrap confidence intervals to ensure your evaluation deltas are statistically significant before you fail a build.

    Use cases

    • Production monitoring to track token costs and latency across multi-step RAG pipelines.
    • Regression testing to ensure new prompts don't increase hallucination rates in gold datasets.
    • User feedback loops to automatically promote failed production traces into fine-tuning or evaluation sets.

    Known limitations

    Statistical reporting requires a minimum sample size of 30 reports to avoid inconclusive results. Requires a self-hosted or managed Anvil instance to receive traces.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Statistical reporting for evals requires n > 30 for conclusive results. Requires a running Anvil instance (self-hosted or managed) to ingest traces.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 20 days ago

    • Free with an Agensi account

    Software
    Docker (for local Anvil backend)
    Python 3.11+
    orNode.js 18+

    Permissions

    Terminal / Shell
    Read Files
    Network Access
    Environment Variables
    Listed20 days ago

    What's inside

    Frequently Asked Questions

    Anvil solves the 'black box' problem in LLM deployments by providing deep tracing for latency and costs, alongside a framework to catch hallucinations before they reach production via automated CI quality gates.

    Anvil supports both Python and TypeScript/Node.js environments, offering native integrations for LangChain and major providers like OpenAI, Anthropic, and Google Gemini.

    Setup involves initializing the SDK with `anvil.init()`, wrapping your core LLM functions with the `@anvil.observe` decorator, and configuring your quality thresholds via a YAML file or the CLI.

    Yes, the skill includes a GitHub Actions template that allows you to set minimum score thresholds, effectively blocking pull requests that cause performance or accuracy regressions.

    While Anvil can track single traces, the statistical regression testing and CI gating features require a minimum of 30 reports to ensure evaluation results are mathematically significant.

    You receive the full Python and TypeScript SDK instrumentation guides, the evaluation CLI tool, pre-configured LLM grading templates, and the GitHub Actions workflow for CI/CD integration.

    New in LLM Ops

    Free