Annotation Guideline Gap Reviewer

The problem

Data labeling projects often fail due to ambiguous guidelines, overlapping taxonomies, and missing edge cases. Developers and data scientists waste cycles on low-quality datasets because instructions weren't stress-tested before annotators started work.

What it does

• Analyzes label taxonomies to identify definitions that overlap or conflict.
• Reviews existing gold examples and pilot results to find gaps in positive and negative samples.
• Constructs a label decision matrix to clarify boundaries for borderline cases.
• Generates a pilot QA checklist and adjudication workflow for human reviewers.
• Flags safety, bias, and specialist review risks in sensitive domains like medical or legal.

Why this beats prompting it yourself

This skill follows a rigid 15-step analysis framework designed to catch subtle inconsistencies that generic prompts miss. It forces a clear evidence boundary, ensuring it only relies on supplied materials rather than hallucinating rules or "fixing" policy without human approval.

Use cases

• Reviewing text, image, or audio annotation guidelines before vendor handoff.
• Debugging low inter-annotator agreement in pilot labeling results.
• Drafting adjudication notes and escalation paths for complex moderation tasks.
• Identifying missing negative examples in LLM evaluation rubrics.

Known limitations

Cannot access live annotation platforms, production databases, or model training pipelines. Does not provide final legal, medical, or safety certifications.