- Home
- Skills
- Research & Analysis
- Paper Claim Audit: Numbers vs. Evidence
More screenshots
Works with the AI Agents you already use
Paper Claim Audit: Numbers vs. Evidence
Reconcile a paper against its own raw result files: every number, comparison, and scope claim extracted, recomputed, and graded FAIL, WARN, or NOTE before submission.
$5
Paper Claim Audit: Numbers vs. Evidence
Example session with this skill installed
Audit the claims in my results section before submission. The abstract says we improve accuracy by 12% over the strongest baseline, the main table reports 84.3 ± 0.4 against the baseline's 83.9, and the text says the method is consistently best across all datasets. My raw result files are in the project folder.
- Read your context and instructions
- Compiled the paper claim audit
CLAIM AUDIT — results section + abstract, auditor working from paper and raw files only. 11 checkable claims extracted from the excerpt; 3 findings.
FAIL — C1, headline number. Claim: "improves accuracy by 12% over the strongest baseline." Evidence: no raw file supports a 12% figure. The run logs show five seeds for the new method (best 89.1, mean 84.3) and a baseline mean of 80.9 in the earliest config; against the baseline actually reported in the main table (83.9), the honest mean-versus-mean delta is 0.4 points, and the 12% traces to comparing the best seed against a superseded baseline run. Fix: replace the headline with the mean-versus-mean figure from the reported comparison, or re-run. Verdict impact: NOT READY while C1 stands.
WARN — C4, comparison wording. Claim: "outperforms the baseline (84.3 ± 0.4 vs 83.9)." The point estimates favor the method, but the baseline's interval (±0.9 in the raw file) overlaps; the unqualified wording overstates a 0.4-point gap. Fix options recorded: qualify ("comparable, with a small mean advantage") or add seeds until the intervals separate. Decision required from the authors; silence does not count as acceptance.
WARN — C7, scope claim. Claim: "consistently best across all datasets." Evidence: best on 4 of 6 datasets; the two losses are in Appendix C runs whose files are present and whose results the main text never mentions. Fix: qualify to "best on four of six datasets" and discuss the losses, or narrow the claim to the benchmark family where it holds.
Verified sample: Table 1's remaining per-dataset numbers match the raw logs exactly, and the ablation delta recomputes correctly. Verdict: NOT READY (1 FAIL, 2 WARN awaiting decisions).
Connects securely to your tools. The creator never sees your data.
About this skill
Papers drift from their evidence in ways authors cannot see: the headline improvement quietly compares a best seed against a baseline mean, an ablation table survives from a configuration the methods section no longer describes, "consistently best" forgets the two datasets in the appendix where the method lost. This skill is the reconciliation protocol that catches the drift. An auditor who is not the author — a colleague, a second model, a fresh session — receives only the paper and the raw result files, and extracts every checkable claim into a ledger: every number, every comparison rewritten as a numerical relationship, every scope word with its set enumerated, every method description, every load-bearing citation. Each claim is mapped to the evidence file that should support it; a claim with no file is not unverified, it is unsupported, and that is a finding. Verification means locating the exact value in the raw file, recomputing every derived quantity (means, deltas, error bars) from raw values rather than trusting the paper's arithmetic, and checking comparisons against the error bars before any "outperforms" is accepted. Findings are graded FAIL (contradicts the evidence; blocks submission), WARN (defensible but misleading; needs a recorded accept-or-fix decision), or NOTE (minor). The verdict follows fixed rules: any FAIL outstanding means NOT READY. Ledger template, audit checklist, severity table, and a worked example of a 47-claim audit are included.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Research & Analysis.
- It verifies that claims match evidence; it does not judge whether the method is sound, the experiments well designed, or the work novel — that is peer review.
- Verification is only as strong as the evidence supplied: result files that were themselves fabricated pass a consistency check, which is why the audit grades provenance problems (missing files, version skew) as findings rather than assuming good faith.
- Citation spot-checks cover load-bearing citations, not the full bibliography.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected