More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Search Ranking Eval: nDCG, MRR, recall regression gate

    1

    Did your ranking change make search better or worse? Compare a candidate run against relevance labels and a baseline: nDCG, MRR, precision and recall per query, the worst queries listed, and a pass/fail gate for CI.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Search Ranking Eval: nDCG, MRR, recall regression gate

    Example session with this skill installed

    Evaluate the supplied candidate against the accepted ideal report at k=5 with an absolute 0.05 drop allowance. Show failed gates and prioritize query investigations.

    • Read your context and instructions
    • Compiled the search ranking eval

    Actual stdout, exit 1, from python3 -I -B scripts/search_eval.py references/qrels.json references/candidate.json --k 5 --baseline references/baseline.json --max-drop 0.05 --format markdown:

    Search relevance evaluation

    Status: FAIL; queries: 3; k: 5

    QueryP@5R@5RR (MRR mean)AP (MAP mean)nDCG linear@5nDCG exp@5
    "MEAN"0.2000000.5555560.5000000.4444440.4959770.505351
    "q1"0.4000000.6666670.5000000.3333330.4879320.516054
    "q2"0.2000001.0000001.0000001.0000001.0000001.000000
    "q3"0.0000000.0000000.0000000.0000000.0000000.000000

    Worst queries (linear nDCG ascending; ID breaks ties): "q3", "q1", "q2"

    Diagnostics: {"missing_run_queries": ["q3"], "ignored_run_queries": ["extra"], "no_positive_judgment_queries": [], "duplicate_run_ids_removed": 0}

    Gate: absolute mean drop > 0.05; all six metrics.

    MetricCandidate minus baselineFailed
    precision-0.133333yes
    recall-0.444444yes
    mrr-0.500000yes
    map-0.555556yes
    ndcg_linear-0.504023yes
    ndcg_exponential-0.494649yes

    FAIL: all six means exceed the allowance. First investigate q3's missing ranking; then q1's missing d7 and relevant documents at ranks 2 and 4. q2's only relevant document is already first. These are small made-up example files that show how the check works. They say nothing about a real search system.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem You've changed a reranker. Two queries look better, but a third has quietly dropped out of the export. Before you call the change a win, you need every labeled query scored and compared with the version you trust. This package does that on your saved files and points at the queries where coverage or ordering got worse.

    See it before you buy: real input, real output Command (from the package folder, exit code 1, 0.027 s):

    python3 -I -B scripts/search_eval.py references/qrels.json references/candidate.json --k 5 --baseline references/baseline.json --max-drop 0.05 --format markdown
    

    Before: A saved candidate ranking that covers 2 of the 3 labeled queries (q1 and q2, with an extra unlabeled query), the relevance labels, and an accepted baseline.

    After: lines copied from the script's output (excerpt):

    Status: FAIL; queries: 3; k: 5
    Worst queries (linear nDCG ascending; ID breaks ties): "q3", "q1", "q2"
    Diagnostics: {"missing_run_queries": ["q3"], "ignored_run_queries": ["extra"], "no_positive_judgment_queries": [], "duplicate_run_ids_removed": 0}
    | precision | -0.133333 | yes |
    | mrr | -0.500000 | yes |
    | ndcg_linear | -0.504023 | yes |
    

    What the numbers show

    • Matches 5 worked examples with known answers (scikit-learn docs, university lecture notes, and my own Searchlite test): all 26 numbers compared agreed to four decimal places.
    • Each run took under 0.07 seconds on Python 3.13, startup included.
    • Honest limit: this shows the scores are computed correctly, not how good your search is.

    What you get

    • A standalone Python script and instructions your agent can follow to evaluate your files and explain the results.
    • Six metrics: precision@k, recall@k, MRR, MAP, linear nDCG@k and exponential nDCG@k.
    • JSON or headerless TSV inputs, with JSON or Markdown reports.
    • Diagnostics for missing queries, extra queries, queries without positive labels and duplicate document IDs.
    • A pass/fail check against a report you already accepted. You set how much each average score may drop, and the JSON output shows the change for every query.
    • Format notes, example inputs and 10 saved reports you can reproduce from the package folder.

    Measured on real data (full detail) I checked the math against worked examples with known answers: two from the scikit-learn docs, one from university lecture notes, and a test case from my own Searchlite project. This proves the scores are computed correctly. It says nothing about how good your search is.

    InputExpected scoreScript resultQueries
    scikit-learn example A, linear nDCG@50.69570.69571
    scikit-learn example B, linear nDCG@50.49370.49371
    scikit-learn example B, linear nDCG@40.35200.35201
    University MRR example, k=20.61110.61113
    Searchlite test case, mean recall@50.55560.55563

    Across those five runs I compared 26 numbers (per-query and average scores for each metric) with values I worked out by hand from the formulas. All 26 matched to four decimal places. The scikit-learn page only prints two digits (0.69, 0.49, 0.35), so I recomputed its expected values at full precision from the same inputs. Each run took under 0.07 seconds on Python 3.13, startup included. The commands and sources are in references/benchmarks.md.

    Known limitations

    • You must supply labels and saved rankings. The script doesn't retrieve documents or create judgments.
    • Unjudged documents count as nonrelevant, so incomplete labels can distort the scores.
    • MRR and MAP use the full supplied ranking. The cutoff applies only to precision, recall and nDCG.
    • The pass/fail check looks at averages only. It doesn't test statistical significance, and a pass doesn't mean every query got better.
    • Raw TREC files need converting to the documented JSON or TSV first. If your scores have ties, your export has to fix their order. The script won't average them.

    Who it's for Retrieval engineers and RAG teams comparing retrievers or rerankers on labeled queries.

    Requires Python 3.8+ and an agent that can read files and run terminal commands. Only the standard library is needed. The code sticks to Python 3.8 syntax, but I've only run it on 3.13. Everything runs locally. There's no network access or telemetry, and the script writes nothing unless you redirect output.

    Price and guarantee $12 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    No support for live search, latency tests, or LLM-based answer grading. 16 MiB file limit per input. Requires Python 3.8+. UTF-8 only.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Software
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed15 days ago
    Updated5 days ago

    What's inside

    Frequently Asked Questions

    They're free, they have more metrics, and they read TREC files natively. If you already use them, keep using them. This package is for when your rankings are plain JSON or TSV exports and you want one dependency-free script that flags missing queries and gives a pass/fail against your last accepted run.

    No. Without a baseline, it only means the inputs were valid. With a baseline, it means none of the six averages dropped more than you allowed. Check the per-query changes and how complete your labels are before you ship.

    No. Score the accepted ranking first and save its JSON report, then pass that report as the baseline. It has to use the same labels, query IDs and cutoff. The script rejects mismatched reports but can't tell who made one, so only use reports you trust.

    Agensi's money-back guarantee covers every paid skill: if it doesn't do what this page promises, you get your money back. Everything above is output from the script in the package, so you can see what it does before you buy.

    New in LLM Ops