Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Post-Training Eval & Safety Pack: Is Your Fine-Tune Better?

    1

    Decides if a fine-tuned checkpoint is really better and safe: eval design, sample sizes, judge calibration, red teaming, ship verdict.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Post-Training Eval & Safety Pack: Is Your Fine-Tune Better?

    Post-Training Eval & Safety Pack: Is Your Fine-Tune Better?

    Example session with this skill installed

    Our new fine-tune wins 58% vs the old model on 100 prompts judged by GPT. Can we ship it?

    • Read your context and instructions
    • Compiled the post-training eval

    Not yet - need more evidence. 58% on 100 items is within noise (roughly ±10 points at 95%), and a single judge run in one order can carry position bias. Fix: ~400 items per key slice, score both orders and count only consistent wins, check the judge against ~100 human labels, and run the paired comparison on the same items. Also missing: safety and over-refusal suites and a regression check on slices you didn't target. Verdict: hold. Next action: re-run the judge with order swapped and report the inconsistency rate.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A fine-tune that "looks better" on 50 examples judged by one LLM is not better. This skill makes your agent act as the evaluation and safety lead that decides whether a checkpoint ships.

    What it does

    • Designs eval suites: metric hierarchy, frozen private regression sets, slice minimums.
    • Gives sample sizes per slice and uses paired comparison on the same items, so you know if a difference is real.
    • Checks LLM judges and reward models: calibration against humans, order-swapped comparisons, reward hacking, judges used both to train and to evaluate.
    • Safety testing: harmful compliance, over-refusal on benign look-alikes, jailbreaks, prompt injection, multilingual gaps.
    • Ends with a verdict (ship / don't ship / need more evidence) against a release checklist.

    Part of the Post-Training series. The full Post-Training Gatekeeper bundles all four packs.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Designs and reviews evaluations; it does not run models or compute statistics itself. Not a substitute for domain-specific safety review in high-risk settings.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 17 days ago

    Permissions

    No special permissions declared or detected

    Listed17 days ago

    What's inside

    Frequently Asked Questions

    Decides if a fine-tuned checkpoint is really better and safe: eval design, sample sizes, judge calibration, red teaming, ship verdict.

    This skill uses the open SKILL.md standard. Use it through an MCP-compatible agent connected to Agensi, and check the compatibility information on this page for confirmed agents and required capabilities.

    Connect your AI agent to Agensi through MCP, then ask it to use Post-Training Eval & Safety Pack: Is Your Fine-Tune Better? on your task. Your agent loads the latest approved version each time. Paid skills require a purchase or active subscription. Skill files are not available for download; your licence does not allow you to save, copy or export any part of a skill.

    New in LLM Ops