- Home
- Skills
- Research & Analysis
- Experiment Audit: Integrity and Honesty Checks
More screenshots
Works with the AI Agents you already use
Experiment Audit: Integrity and Honesty Checks
An adversarial integrity audit for finished experiments: ground-truth provenance, score normalization, phantom results, and scope honesty, ending in a PASS, WARN, or FAIL verdict.
$5
Experiment Audit: Integrity and Honesty Checks
Example session with this skill installed
Audit my completed experiment before I write it up. An agent ran a retrieval study for me and reports 0.97 accuracy across a comprehensive benchmark suite. The eval scripts, result files, and tracker are in the project folder; the draft summary citing the numbers is there too.
- Read your context and instructions
- Compiled the experiment audit
INTEGRITY AUDIT — retrieval study, artifacts collected mechanically (6 eval scripts, 14 result files, 1 tracker, draft summary). Reviewer independent of the run. Verdict: FAIL.
Check A, ground-truth provenance — FAIL. The main eval script builds its "reference" answers by querying the same model family under evaluation and caching the outputs as ground truth. The 0.97 headline is agreement with the model's own outputs, reported as accuracy, with no proxy labeling in the summary or the scripts' output headers. This is the cardinal failure: the number measures self-consistency, not performance.
Check B, score normalization — WARN. Per-dataset scores in the cited table are normalized by the best score within each run batch. Raw scores exist in the result files, but the summary cites only the normalized values, which flatters the two weakest datasets.
Check C, result existence — FAIL. The summary's strongest comparison cites results/final_v2.json. That file does not exist; the tracker lists the run as TODO. Two numbers in the table match no key in any result file supplied.
Check D, dead code — WARN. Two metric functions (an exact-match scorer and a latency metric) are defined in the eval package and never called; the reported metric set appears selected after the runs.
Check E, scope honesty — WARN. "Comprehensive benchmark suite" is three datasets, one seed each. Check F, typing: synthetic proxy, mislabeled in the summary as benchmark accuracy.
Claim impact: the 0.97 accuracy claim is unsupported; dataset-level comparisons that trace to real files need qualifiers (proxy evaluation, single seed). Fix path: re-run against the dataset's actual reference answers, cite raw scores, then re-audit the new artifacts. Rewriting the summary around the existing numbers is explicitly not a fix.
Connects securely to your tools. The creator never sees your data.
About this skill
Results produced under optimization pressure fail in characteristic ways, whether the optimizer is a person chasing a deadline or an agent chasing a metric: ground truth quietly derived from the model's own outputs, scores normalized by the model's own maximum, headline numbers cited from result files that were never produced, and a two-configuration pilot described as a comprehensive evaluation. This skill is the integrity check that runs once, after experiments complete and before any claims are written. Its structure is the safeguard: the party that ran the experiment never judges it. The executor collects artifacts mechanically — evaluation scripts, result files, ground-truth sources, the experiment tracker, configs, and the claims narrative — as paths only, and an independent reviewer works under an adversarial brief: assume the evaluation is compromised somewhere, find where. Six checks cover ground-truth provenance (the cardinal failure: agreement-with-yourself reported as performance), score normalization, result existence (phantom results fail outright), dead metric code, scope honesty, and honest evaluation typing — real ground truth, synthetic proxy, simulation-only, human evaluation. The verdict is PASS, WARN, or FAIL with fixed conditions, every check anchored to file-and-line evidence, and every claim in the narrative mapped to supported, needs qualifier, or unsupported, with tags that travel downstream into review and writing stages. A report template, machine-readable verdict format, and a worked example of a FAIL verdict are included.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Research & Analysis.
- It catches common, known integrity patterns, not every possible form of misconduct; a PASS means "no known integrity failure found," never "the science is correct."
- The audit is only as strong as the artifacts supplied — work whose scripts and result files cannot be produced is graded FAIL by absence rather than taken on trust.
- It judges evaluation integrity, not method quality or experimental design; a sound audit verdict and a weak method can coexist.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected