Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Annotation QA & Rubric Architect

    1

    Audit and draft AI training data against strict project guidelines to increase task acceptance rates.

    Free

    Security scanned
    Annotation QA & Rubric Architect

    Annotation QA & Rubric Architect

    Example session with this skill installed

    I am working on a RLHF project. Here is the guideline: use 3rd person, no emojis, max 3 sentences, and never mention the AI's identity. QA this draft: 'I can help you with that! The user wants a cake.'

    • Read your context and instructions
    • Compiled the annotation qa

    I have audited the draft against your four provided project rules. The draft currently fails on three counts.

    Annotation QA Audit

    Rule IDGuideline ConstraintStatusViolation Details
    R-01Use 3rd person perspectiveFAILUse of "I" in "I can help you" is 1st person.
    R-02No emojis allowedPASSNo emojis detected.
    R-03Max 3 sentencesPASSDraft is exactly 2 sentences.
    R-04Never mention AI identityFAIL"I can help you" identifies the speaker as the AI assistant.

    Recommended Correction:
    "The user requested assistance with a task. A cake recipe or instructions are required."

    Next steps

    1. Update the draft to remove 1st person pronouns and AI self-identification.
    2. Confirm if "The user" is the preferred 3rd person referent for this project.
    3. Run the corrected draft through this QA check once more before submission.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Annotation projects often fail because workers lose sight of 40-page guideline documents over hundreds of tasks. This skill prevents work rejection by anchoring every draft and review to the specific rules, edge cases, and voice constraints of your project.

    What it does

    • Guideline extraction: Converts long PDF or text guidelines into a hard checklist of required voice, tense, and banned phrases.
    • Pre-submission QA: Audits drafts against project rules and flags specific violations with the exact guideline clause broken.
    • Rubric construction: Builds multi-dimensional scoring rubrics with level descriptors and tie-breakers to ensure inter-annotator agreement.
    • Response evaluation: Scores model outputs against rubrics with written justifications for every dimension.
    • Batch consistency: Infers the working voice from previously accepted tasks to maintain a uniform style across a long queue.
    • Edge-case reasoning: Provides documented arguments for tasks that straddle two categories for use in reviewer notes.

    How it works

    1. Rule ingestion: Provide the project guideline document to extract the core checklist.
    2. Drafting or audit: Request a new annotation draft or paste an existing one for a compliance audit.
    3. Grading & Justification: Use the rubric mode to score responses or build new evaluation frameworks from scratch.

    Frameworks & tools

    This skill is framework-agnostic and designed for use within LLM-based development environments. It supports platforms like Outlier, Scale AI, Surge, and internal model evaluation workflows.

    Why this beats prompting it yourself

    Standard prompting often ignores subtle negative constraints or drifts into generic AI voice over long sessions. This skill enforces strict adherence to specific clauses, catching the minor formatting or tone errors that human reviewers use to reject paid tasks.

    Use cases

    • Drafting RLHF prompts that must follow strict persona and length constraints.
    • Reviewing batches of data labels for consistency before final submission.
    • Building calibration sets to align a team of graders on a new project.
    • Generating justification text for edge-case classifications in complex datasets.

    Known limitations

    This skill does not perform external fact-checking or invent data to fill missing fields. It requires a provided guideline document to function effectively and will flag ambiguities rather than guessing.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Prompt & Skill Engineering.

    Requires a source guideline document to establish rules. Does not perform external fact-checking. Ambiguities in guidelines will trigger a request for clarification rather than a guess.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed today

    • Free with an Agensi account

    Permissions

    No special permissions declared or detected

    Listedtoday

    What's inside

    Frequently Asked Questions

    This skill automates the QA process by cross-referencing every annotation against your project's specific 40-page guidelines, catching subtle tone or formatting violations that human reviewers often use to reject tasks.

    Yes, the skill can analyze your previously accepted "Golden Sets" to infer the required voice and style, ensuring all future drafts in a batch remain consistent with the established project standard.

    The skill is framework-agnostic and designed to work within any LLM-based development environment or internal evaluation workflow that supports text and PDF ingestion.

    Simply upload your project's PDF or text guideline document; the skill will automatically extract core constraints, banned phrases, and scoring dimensions to create a functional checklist.

    While it provides rigorous logic for edge cases and rubric-based justifications, it does not perform external fact-checking or fill in missing data points that aren't present in your source material.

    Your purchase includes the full logic for guideline extraction, the multi-dimensional rubric builder, and the pre-submission audit module, along with any future version updates to the core architect.

    New in Prompt & Skill Engineering

    Free