Works with the AI Agents you already use
Data Forge AI Pro
Provenance-preserving dataset curation, annotation rubrics, leakage controls and reproducible model evaluation.
$9.99
Data Forge AI Pro
Example session with this skill installed
Design an evaluation for an AI customer-support agent across refunds, cancellations and escalation, using 250 labeled conversations.
- Read your context and instructions
- Compiled the data forge ai
I have reviewed the task to create a gold evaluation set from 5,000 support tickets. Based on the requirements, I will focus on preventing temporal and entity-based leakage.
Dataset Evaluation & Split Plan
| Component | Specification |
|---|---|
| Unit of Analysis | Single customer support ticket (Thread ID) |
| Target Labels | Intent (Refund, Tech Support, Billing, Shipping, Other) |
| Deduplication | Exact match on Body text; Fuzzy match (95% Jaccard) on metadata-heavy rows |
| Split Strategy | Chronological: Train (First 70%), Val (Next 15%), Test (Final 15%) |
| Leakage Control | Group by Customer_ID to ensure no user appears in both Train and Test |
| Gold Set Size | 500 manually verified records (10% of total) |
| Metrics | Macro-F1 (imbalance handling), Precision (Billing), Recall (Escalations) |
Provenance & Quality Constraints
- Observed: 5,000 raw records provided via CSV.
- Derived: High likelihood of duplicate tickets from follow-up emails.
Recommendation: Implement a 24-hour window merge for tickets from the same user to prevent near-duplicate leakage.
Privacy Check: FAILED - Raw Email and Phone columns detected. De-identification step required before annotation.
Annotation Rubric Preview
- Label:
Refund_Request - Definition: User explicitly asks for money back or disputes a charge.
- Edge Case: "I want to cancel" without mentioning money is
Cancellation, notRefund. - Conflict Resolution: Tie-break by Senior Annotator if confidence < 0.8.
Next steps
- Approve the chronological split strategy to account for shifting support topics.
- Confirm the PII masking logic for the
EmailandPhonefields. - Provide 20 sample rows to initialize the annotation decision tree.
Connects securely to your tools. The creator never sees your data.
About this skill
Trustworthy datasets and evaluations, not fabricated benchmark wins. Built for ML engineers, data teams and AI product evaluators. Includes a specialist SKILL.md, four operating modes, a 10-stage expert checklist, domain stop rules, a reusable delivery template, an evaluation rubric and practical acceptance prompts. Runs as a portable instruction skill in compatible hosts; host tool permissions and live integrations are not included.
Does not include standalone SaaS, provider API access, or preconfigured third-party integrations. Real host performance not yet independently tested.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Does not execute code or browse the web unless tools are provided. Requires raw data samples to perform audits. Cannot verify legal ownership of third-party data.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago