More screenshots
Works with the AI Agents you already use
Citation Checker: Find unsupported claims in AI answers
**The problem** You have a draft, a RAG answer or a report summary with citations, plus the passages it was written from.
$9
Citation Checker: Find unsupported claims in AI answers
Example session with this skill installed
Check the six-sentence synthetic expense answer in references/example.json against its three passages. Run the strict gate, then read the sources for contradictions.
- Read your context and instructions
- Compiled the citation checker
Captured stdout, exit 1, from python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown:
| # | Sentence | Citations | Support | Label | Reason |
|---|---|---|---|---|---|
| 1 | Submit expense reports with itemized receipts within 30 days of the purchase. | expenses#deadline | 1.00 | supported | 9 of 9 content words found in the cited passages |
| 2 | Late reports are returned to the employee and the manager is notified. | expenses#deadline | 0.50 | weakly_supported | 3 of 6 content words found in the cited passages (below 0.80) |
| 3 | Expenses above $1,000 need approval from the finance team. | expenses#approval | 0.86 | unsupported | numbers not in the cited passages: 1000 |
| 4 | Meals on business trips are reimbursed up to $60 per day. | travel#meals (rejected: travel#lodging) | 0.86 | supported | 6 of 7 content words found in the cited passages; rejected citations: travel#lodging |
| 5 | Alcohol is reimbursed on business travel. | travel#meals | 1.00 | supported | 4 of 4 content words found in the cited passages |
| 6 | Most employees find the process easy. | - | 0.00 | unsupported | the sentence has no citations |
Sentences: 6 | supported 3 | weakly supported 1 | unsupported 2 | citation coverage 83%
Lexical heuristic only: review every label before relying on it.
Reviewed verdict: Needs fixes. Sentence 3 says $1,000 where the source says $500. Sentence 5 reverses the source's "Alcohol is not reimbursed" even though the script marked it supported. Sentence 4's meals claim is fine, but its lodging citation points to a passage that wasn't supplied. Sentence 2 adds manager notification, and sentence 6 has no citation. Fix the limit and the alcohol rule first, then the citations and additions. The 83% coverage counts sentences with a valid citation. It says nothing about whether they're true.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem You have a draft, a RAG answer or a report summary with citations, plus the passages it was written from. The citations look fine at a glance, but one sentence quotes a number the source never gives, another cites an ID that isn't there, and a third says the opposite of its source. Reading every claim against every passage by hand is slow, and it's easy to miss things.
See it before you buy: real input, real output Command (from the package folder, exit code 1, 0.02 s):
python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown
Before: Passage and answer sentences (excerpt of references/example.json)
"text": "Your team lead approves expenses up to $500. Anything above $500 also needs approval from the finance team."
"text": "Expenses above $1,000 need approval from the finance team.",
"citations": ["expenses#approval"]
"text": "Alcohol is reimbursed on business travel.",
"citations": ["travel#meals"]
After: lines copied from the script's output (excerpt):
| 2 | Late reports are returned to the employee and the manager is notified. | expenses#deadline | 0.50 | weakly_supported | 3 of 6 content words found in the cited passages (below 0.80) |
| 3 | Expenses above $1,000 need approval from the finance team. | expenses#approval | 0.86 | unsupported | numbers not in the cited passages: 1000 |
| 6 | Most employees find the process easy. | - | 0.00 | unsupported | the sentence has no citations |
Sentences: 6 | supported 3 | weakly supported 1 | unsupported 2 | citation coverage 83%
What the numbers show
- Tested on RAGTruth: 86,566 sentences from 11,592 model-written answers, each checked against all of its source passages.
- The default 'unsupported' flag caught 52% of annotator-marked sentences (3,745 of 7,157); flagging unsupported or weak caught 81% (5,802 of 7,157).
- Honest limit: roughly four out of five flags were sentences the annotators hadn't marked, so the agent reads the sources instead of trusting the label.
What you get
- A small Python checker. For each sentence it checks that the cited IDs exist, how many of the sentence's words appear in the cited text, and whether every number in the sentence appears there too.
- A report in JSON, Markdown or both, with a plain reason on every row ("3 of 6 content words found", "numbers not in the cited passages: 1000").
- Instructions that make the agent read each cited passage before giving its own verdict, so you see the script's label and the reviewed verdict side by side, with fixes in priority order.
- An optional
--strictmode that exits 1 when any sentence isn't supported, so you can use it as a pipeline check. You can adjust the two cut-offs (0.80 and 0.50 by default). - 23 test cases, four saved sample outputs, a worked six-sentence example and the scoring script I used for the numbers below.
Measured on real data (full detail) I ran it on RAGTruth, a public dataset where people marked the made-up parts of 11,592 model-written answers and summaries (86,566 sentences). Each sentence was checked against all of its source passages. The full run took about 40 seconds on one machine.
| What counts as a flag | Marked sentences it caught | Flags that hit a marked sentence |
|---|---|---|
| Labelled unsupported | 52% (3,745 of 7,157) | 19% (3,745 of 19,821) |
| Unsupported or weak | 81% (5,802 of 7,157) | 14% (5,802 of 41,099) |
| A number missing from the source | 22% (1,571 of 7,157) | 17% (1,571 of 9,335) |
In plain terms: the default flag finds about half of the problem sentences, and roughly four out of five flags are sentences the annotators didn't mark. That's why the agent reads the sources instead of trusting the label. I also read ten examples by hand. The number rule caught invented years and durations. The misses were things like a net worth assigned to the wrong person and a hedged finding turned into a flat denial, where every word still matched the source. The annotations cover whole answers, so splitting them into sentences adds some noise.
Known limitations
- Word overlap can't see meaning. A sentence that flips "not reimbursed" to "reimbursed", or credits a fact to the wrong person, can still score 1.00. Every row needs a human or agent read.
- Expect a lot of false alarms. On the public data most flags were on sentences nobody marked, so treat
--strictas a tripwire, not a hallucination detector. - It only checks text you supply. It never fetches URLs, doesn't confirm a paper exists, and needs PDFs, BibTeX or Markdown bibliographies turned into JSON first.
- The word matching uses simple English rules. Other languages and heavy paraphrase score poorly.
Everything runs locally, with no network, no telemetry and no files written unless you redirect the output. The agent may write a temporary input file.
Who it's for Editors, researchers and teams building RAG systems who already have the source excerpts and want a repeatable first pass they can check line by line.
Requires Python 3.8 or newer and an agent that can read files and run Python. The checker uses only the standard library. No API keys, no account.
Check it yourself
23 test cases (passing, failing and invalid inputs) and four saved sample outputs, all re-run against the current script. From the package folder, try python3 -I -B scripts/check_citations.py references/example.json --strict --format markdown, or run the cases listed in references/checks.json. The RAGTruth figures can be reproduced with the included scoring script once you download the dataset. Steps are in references/benchmarks.md.
Price and guarantee $9 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Does not fetch URLs, parse PDFs, or verify source truth. Uses English stemming; not for semantic reasoning. Requires structured JSON input. Maximum row limits depend on available system memory.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
- Software
Python 3
Permissions
Uses file-reading and terminal tools to run Python on the supplied JSON file or standard input. The agent may write temporary input JSON; the checker only prints reports and writes no files itself. No network access, no telemetry, and nothing is sent anywhere. It never fetches URLs.