More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    LM Evaluation Harness: Benchmark Language Models Rigorously

    1

    Benchmark language models with the community-standard harness — fair task sets, reproducible few-shot settings, and comparisons you can trust.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LM Evaluation Harness: Benchmark Language Models Rigorously

    LM Evaluation Harness: Benchmark Language Models Rigorously

    Example session with this skill installed

    We fine-tuned two checkpoints and the numbers jump around every run. Set up lm-eval so the comparison is fair and repeatable, and tell me which checkpoint is actually better.

    • Read your context and instructions
    • Compiled the lm evaluation harness

    Setup: installed the harness and ran the first eval on the base model with a fixed config — tasks arc_easy and hellaswag, 25-shot and 10-shot respectively, recorded in a config file that is now version-controlled. The first run took 40 minutes on the available GPU; the second run on the same config reproduced the numbers exactly, which is the whole point. Task choice: dropped a task the team had been using because it mostly measured formatting, not reasoning, and added mmlu for knowledge and winogrande for commonsense — four tasks that map to the capabilities the fine-tune was supposed to improve. Fair comparison: both checkpoints ran on the identical config, same prompts, same few-shot counts, same backend. Checkpoint A scored 71.2 on the aggregate, checkpoint B 72.8 — but the per-task breakdown showed B's gain came entirely from one task, and A was better on two others. Speed: enabled batching and the faster backend for the large run, and verified the fast config reproduced the slow config's numbers on a 200-example subset before trusting it — speed without that check is just a different experiment. Checkpoint tracking: set a fixed eval schedule — every 500 steps on the same four tasks — so the training curve is comparable across runs instead of a scatter of one-off numbers. Interpretation: the 1.6-point aggregate gap is inside the noise for this task set, so the honest verdict is 'no meaningful difference yet, B is ahead on one task' — and the team kept training instead of shipping a coin flip. The config file, the per-task table, and the noise note all went into the run record.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Two models compared with different prompts, different few-shot counts, and different task versions produce numbers that mean nothing, and this skill exists to stop that from happening. It teaches the lm-evaluation-harness as a discipline: choose the task set that matches the capability you actually care about, configure models and backends for speed without cutting corners, freeze few-shot settings and record them, compare models on identical configs, and track training progress with checkpoint evals on a fixed schedule. It also covers the part everyone skips — interpreting results honestly, including the error bars, the contamination question, and the moment a benchmark stops measuring what you think it measures.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Workflow Automation.

    It covers the harness and the discipline around it, not the design of new benchmark tasks.
    A benchmark measures what its tasks measure; a high score on a contaminated or format-heavy task is not evidence of capability.
    Small task sets have wide error bars — the skill shows how to read them, but it cannot shrink them.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    Terminal / Shell
    Listed3 days ago

    What's inside

    Frequently Asked Questions

    Probably the config, not the model. Few-shot count, prompt template, task version, and backend all change results. Freeze them in a version-controlled config and re-run — identical configs reproduce.

    Not on the aggregate alone. Look at the per-task breakdown and the error bars: if the gain comes from one task or sits inside the noise, it is not a real win. Keep training or evaluate on more tasks.

    Batching and a faster backend, then verify the fast config reproduces the slow config on a subset before trusting it. Never change the task set or the few-shot count to buy speed.

    New in Workflow Automation