Works with the AI Agents you already use
anvil
Instrument Python/TS apps, track LLM costs, and enforce CI quality gates with Anvil observability.
Free
anvil
Example session with this skill installed
I need to instrument a TypeScript RAG pipeline and add a CI gate that fails if the hallucination score drops below 90 percent.
- Read your context and instructions
- Compiled the anvil
I have updated your TypeScript pipeline to include multi-span waterfall tracing and generated a GitHub Actions workflow to gate your pull requests.
Anvil Instrumentation and CI Config
// src/lib/llm.ts
import * as anvil from "@anvil/sdk";
anvil.init({ apiKey: process.env.ANVIL_API_KEY! });
export async function ragPipeline(question: string, userId: string) {
return await anvil.trace("qa-pipeline", { input: question, userRef: userId }, async (t) => {
const context = await anvil.span("vector-retrieval", { type: "retrieval" }, async () => {
return await vectorDb.query(question);
});
const completion = await anvil.span("llm-generation", { type: "llm" }, async () => {
return await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: `Context: ${context}\n\nQuestion: ${question}` }]
});
});
t.output = completion.choices[0].message.content;
return { answer: t.output, traceId: t.id };
});
}
# .github/workflows/anvil-eval.yml
name: LLM Quality Gate
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Anvil Evals
run: |
npx @anvil/cli eval run \
--dataset "golden-rag-set" \
--grader hallucination \
--grader-config '{"model": "claude-3-5-sonnet-20241022"}' \
--min-score 0.90 \
--max-regression 0.05
env:
ANVIL_API_KEY: ${{ secrets.ANVIL_API_KEY }}
Next steps
- Ensure
ANVIL_API_KEYis added to your GitHub Repository Secrets. - Pass the
traceIdreturned byragPipelineto your frontend to enableanvil.feedback()calls. - Verify that the
golden-rag-setdataset exists in your Anvil dashboard.
Connects securely to your tools. The creator never sees your data.
About this skill
Production LLM features often lack the visibility needed to debug failures or prevent regressions. This skill provides the instrumentation and evaluation framework required to monitor, trace, and gate LLM applications using Anvil.
What it does
- Tracing instrumentation adds non-blocking execution tracking to Python and TypeScript LLM chains to capture latency, costs, and traces.
- Feedback collection wires user-facing thumbs up/down signals directly to specific trace IDs for failure triaging.
- Dataset construction builds evaluation sets from real-world production complaints captured in the Failure Inbox.
- Automated grading configures deterministic checks like regex and JSON validation alongside LLM judges for accuracy and hallucination.
- CI quality gating integrates with GitHub Actions to block pull requests that breach regression thresholds or minimum scores.
How it works
- Initialize the SDK in your application using
anvil.init()with your API key and self-hosted URL. - Wrap your LLM functions with
@anvil.observeoranvil.traceto begin asynchronous shipping of traces. - Configure graders using the
anvil eval runCLI or YAML configs to define your quality bar. - Deploy the provided GitHub Actions template to enforce score thresholds on every pull request.
Frameworks & tools
Python (SDK), TypeScript/Node.js (SDK), LangChain integration, OpenAI, Anthropic, Google Gemini, and local models. Supports GitHub Actions for CI/CD workflows.
Why this beats prompting it yourself
Prompting an LLM to "be an evaluator" results in inconsistent scoring without statistical rigor. This skill implements specific instrumentation patterns, idempotent ingestion, and bootstrap confidence intervals to ensure your evaluation deltas are statistically significant before you fail a build.
Use cases
- Production monitoring to track token costs and latency across multi-step RAG pipelines.
- Regression testing to ensure new prompts don't increase hallucination rates in gold datasets.
- User feedback loops to automatically promote failed production traces into fine-tuning or evaluation sets.
Known limitations
Statistical reporting requires a minimum sample size of 30 reports to avoid inconclusive results. Requires a self-hosted or managed Anvil instance to receive traces.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Statistical reporting for evals requires n > 30 for conclusive results. Requires a running Anvil instance (self-hosted or managed) to ingest traces.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 20 days ago
- Free with an Agensi account
- Software
Docker (for local Anvil backend)
Python 3.11+
orNode.js 18+