Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Assay — RAG Regression Gate for CI

    1

    Open-source CI gate for RAG quality, enforcing regression policies for groundedness, latency, and cost.

    Free

    Security scanned
    Assay — RAG Regression Gate for CI

    Assay — RAG Regression Gate for CI

    Example session with this skill installed

    Help me set up a quality gate for my RAG system. I have a production run I trust and I want to make sure my new chunking logic doesn't hurt groundedness.

    • Read your context and instructions
    • Compiled the assay

    I will help you establish a baseline from your trusted run and configure the regression gate.

    RAG Quality Gate Configuration

    First, I have pinned your trusted production run (Run 001) as the primary baseline. I have also generated a regression policy that will fail the build if groundedness drops by more than 5%.

    Regression Policy (policy.yaml)

    version: 1
thresholds:
  groundedness: 5.0
  answer_relevance: 5.0
  context_recall: 5.0
  p95_latency_ms: 20.0
  avg_cost_usd: 10.0

    Execution Commands

    Run these commands to evaluate your new chunking logic against the baseline:

    
# 1. Create a new run for the chunking experiment
assay run create --dataset 1 --name run-chunking-v2 --target http://localhost:9000/query

# 2. Execute the run with the LLM judge enabled
assay run execute 2 --verbose --judge

# 3. Evaluate the gate against the baseline
assay gate --baseline baseline-v1 --run 2 --policy policy.yaml

    Next steps

    1. Review the worst_failures list in the gate output to see which specific questions regressed.
    2. If the gate fails on latency but quality improved, decide if you need to adjust the p95_latency_ms threshold.
    3. Integrate the assay gate command into your GitHub Actions workflow for automated PR reviews.
    Regression Policy (policy.yaml

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Assay — RAG Regression Gate for CI

    Stop shipping RAG changes on vibes. This skill turns your AI agent into an operator for Assay, the open-source CI gate for RAG quality regressions, so every change to your retrieval stack comes with evidence: did quality get better, or did it get worse?

    What this skill does:

    1. Golden dataset — build or import the questions your system must keep answering well, including questions it should refuse.
    2. Evaluation run — Assay calls your endpoint and scores groundedness, answer relevance, answer completeness, context recall, citation accuracy, refusal correctness, latency, and cost.
    3. Baseline + gate — pin a known-good run, set a regression policy in YAML, and gate every later run: exit code 0 ships, 1 blocks, 2 means setup error.
    4. Readable verdicts — per-metric deltas, the reasons for a FAIL, and the worst failing questions, in text, JSON, or JUnit.
    5. CI integration — a working GitHub Actions pattern puts the gate on every pull request.
    6. Judge validation — before you trust LLM-as-judge scores, validate the judge against blind human labels with Cohen's kappa, an adversarial suite, and bias probes.

    Why a gate, not just metrics? Metric libraries tell you a score. They don't tell you whether this change should ship. Assay's whole design is the missing half: a pinned baseline, a policy you define, and a pass/fail decision your CI can enforce.

    Proof, stated honestly: the judge shipped with Assay is measured, not assumed — Cohen's kappa 0.899 for groundedness on a 250-entry, blind human-labeled calibration set, and kappa 0.920 for answer relevance on a balanced 100-entry challenge set, both on the Indonesian legal domain. The skill teaches you to run the same validation on your domain before trusting scores there. Known limitations are documented: 17/20 adversarial cases pass.

    What you need: a RAG endpoint that answers over HTTP, Python 3.11, and optionally your own API key (BYOK) for the judge — keys stay in your own environment. Assay itself is free and open source (Apache-2.0); this skill is the operating manual your agent follows.

    Honest caveat: Assay is v0.1.0 — early development, core pipeline working end to end, not yet production-hardened.

    Assay: https://github.com/gigihsusetyo/assay · https://assay.web.id · Apache-2.0

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Early v0.1.0 development. Judge accuracy is domain-specific. Ragas metrics require extra heavy dependencies. Requires Python 3.11. Local-first architecture.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed today

    • Free with an Agensi account

    Software
    PythonGit

    Permissions

    Terminal / Shell
    Network Access
    Read Files
    Write Files
    Listedtoday

    What's inside

    Frequently Asked Questions

    Assay prevents quality regressions by running your RAG system against versioned 'golden datasets' and comparing the results to a pinned performance baseline. It enforces a hard PASS/FAIL gate in your CI pipeline, ensuring changes to chunking or embeddings don't degrade accuracy or blow your budget.

    Assay is a Python 3.11 CLI tool that connects to any RAG system via a simple JSON adapter contract. It integrates seamlessly with OpenAI-compatible APIs for judging and supports GitHub Actions or other CI tools for automated workflow gating.

    Setting up Assay involves cloning the repository, configuring your RAG endpoint via a JSON adapter, and defining your quality policies in a YAML file. Once your baseline is pinned, you can trigger the gate with a single CLI command in your deployment pipeline.

    Unlike manual scripts, Assay handles the complex statistical work of Cohen’s kappa for judge validation and manages versioned datasets. It provides standardized exit codes for CI/CD and monitors operational metrics like P95 latency and per-query costs alongside semantic quality.

    The skill includes the full CLI tool, support for local (SQLite) and production (PostgreSQL) data storage, and the policy enforcement engine. As an open-source v0.1.0 tool, you receive the current architecture for regression gating, though judge calibration may require domain-specific tuning.

    New in LLM Ops

    Free