Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    analytics model risk and validation reviewer

    1

    This skill provides an accountable, deterministic review of model dossiers by replicating metrics and stress-testing datasets against objective risk criteria.

    $2.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    analytics model risk and validation reviewer

    analytics model risk and validation reviewer

    Example session with this skill installed

    Review the attached model card for 'ChurnPredict-V2' and the evaluation records. Replicate the 0.85 AUC claim and check for training leakage. Impact is medium.

    • Read your context and instructions
    • Compiled the analytics model risk

    I have processed the validation request for ChurnPredict-V2. The internal validation script has recomputed all metrics and audited the dataset overlaps.

    Model Validation Dossier — ChurnPredict-V2, 2023-10-27

    1. Scope, sources and assumptions

    • Sources: Model Card (E1), Test Records (E2), Training Manifest (E3).
    • Purpose: Independent replication of performance claims and leakage audit.
    • Assumptions: Evaluation records represent the intended deployment distribution (user_supplied).

    2. Recommendation (draft for human approval)

    • Decision: needs_review
    • Basis: Claimed AUC replicated, but significant data leakage detected in Section 5.
    • Production release authorised: no
    • Management decision required

    3. Findings

    IdSeverityCodeFindingRecommendation
    1CriticalLEAK_0115% overlap between training and test record IDs.Re-run evaluation with fresh holdout set.
    2MediumDOC_04Missing monitoring plan in deployment section.Define drift thresholds.

    4. Claims and benchmark replication

    ClaimKindMetricClaimedRecomputedDifferenceStatus
    C-01PredictiveAUC0.85000.8502+0.0002Replicated
    C-02PredictivePrecision0.78000.7795-0.0005Replicated

    5. Leakage, robustness and fairness checks

    • Overlap counts: 1,500 records found in both training and evaluation sets.
    • Temporal separation: 0 days (evaluation records are within the training date range).
    • Robustness: 4% drop in AUC during high-noise scenario (within 5% tolerance).
    • Fairness screen: TPR Gap of 0.02 (Screening indicator).

    6. Documentation and deployment safeguards

    • Completeness ratio: 85%
    • Information gap — organizational validation required: Rollback procedures not documented.

    7. Access, exclusions and warnings

    • Exclusions: 0 records excluded (all datasets approved).
    • Warnings: WARN_LEAK_DETECTED, WARN_MISSING_MONITORING.

    8. Items for human review

    • Observation 1: Recomputed AUC matches the claim (computed result).
    • Observation 2: Training leakage (1,500 records) invalidates current evaluation integrity (computed result).
    • Observation 3: Model owner asserts features are available at decision time (user assertion).

    This review supports, and does not replace, the accountable owner's judgment. Any approval or release is a

    Management decision required.

    Next steps

    • Address the LEAK_01 finding by preparing a clean evaluation dataset without training overlap.
    • Provide the missing monitoring plan to complete the deployment documentation.
    • Present this dossier to the Model Risk Committee for final review.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Machine learning models often reach production without independent verification of their performance claims, leading to unexpected failures or bias. This skill provides an accountable, deterministic review of model dossiers by replicating metrics and stress-testing datasets against objective risk criteria.

    What it does

    • Metric replication recomputes AUC, accuracy, precision, recall, F1, and Brier scores from raw evaluation records to verify claimed performance.
    • Leakage detection identifies train-test overlaps, temporal separation issues, and features unavailable at the time of decision.
    • Fairness screening calculates selection-rate ratios and TPR gaps across supplied group labels to flag potential bias.
    • Robustness testing measures performance drops across specific scenarios against defined tolerances.
    • Risk documentation generates a structured validation dossier with a draft go/no-go recommendation for human review.

    How it works

    1. Gather material such as the model card, evaluation records (true labels and scores), and decision context.
    2. Build the request by mapping sources to a request.json file following the provided schema.
    3. Execute validation using the internal scripts/run.py to process the data deterministically.
    4. Analyze the envelope to interpret the findings, warnings, and replication status generated by the script.
    5. Generate the dossier following a strict eight-section format for human-in-the-loop approval.

    Frameworks & tools

    The skill utilizes a self-contained Python validation engine (scripts/run.py) and JSON schema validation. It operates on pseudonymous evaluation records and model metadata.

    Why this beats prompting it yourself

    LLMs frequently hallucinate mathematical results or fail to catch subtle data leakage when reading text descriptions. This skill uses a deterministic script to ensure every decimal is exact and every overlap is counted, preventing the agent from "taking the model's word" for its performance.

    Use cases

    • Pre-deployment audit to verify that a model meets internal performance and fairness benchmarks before release.
    • Independent verification when a third-party or different team provides a model card with unverified claims.
    • Risk mitigation to identify data leakage or robustness issues that could cause production outages.

    Known limitations

    Does not perform model training, tuning, or continuous drift monitoring. Requires raw evaluation records (labels and scores) to replicate metrics. Cannot authorize production releases or certify legal compliance.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse All Claude Skills.

    Requires raw evaluation records for replication. Does not perform training or drift monitoring. Release authorization requires human-in-the-loop approval.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Environment Variables
    Listed3 days ago

    What's inside

    Frequently Asked Questions

    This skill provides an independent, deterministic audit of ML models by replicating performance metrics, detecting hidden data leakage, and screening for fairness biases that manual reviews often miss.

    Unlike general AI agents that might 'hallucinate' or miscalculate statistics, this skill uses a dedicated Python engine to ensure every AUC, F1, and fairness score is mathematically exact.

    You will need the model's metadata (model card), evaluation records containing true labels and predicted scores, and the decision context mapped to the skill's specific JSON schema.

    The purchase includes the core Python validation engine, the JSON schema for data mapping, the execution scripts, and the structured dossier template for generating final reports.

    The skill identifies risks like train-test overlaps and selection-rate gaps, but it does not perform active model training, hyperparameter tuning, or real-time drift monitoring.

    This skill is designed for integration with AI agents capable of executing local Python scripts and handling structured JSON data, such as AutoGPT or custom LangChain implementations.