LLM Structure Output Debugger

The model is failing because the prompt lacks a clear delimiter for the output and the schema defines 'priority' as a required enum but the prompt describes it as optional text.

Structured Output Debug Report

1. Contract

• Expected Structure: Flat JSON object.
• Required Fields: task_name (string), priority (enum: "high", "medium", "low").
• Constraints: No markdown formatting; raw string output only.

2. Failure

Markdown Wrappers: The model is wrapping output in json ... blocks, which your parser is not configured to strip.

Schema Mismatch: The 'priority' field is intermittently missing or contains values outside the defined enum (e.g., "urgent").

3. Root Cause

Primary Cause: Prompt-Schema Conflict. The prompt says "Mention priority if relevant," while the schema marks it as Required.

• Contributing Cause: Missing response_format configuration in the API call.
• Evidence Level: Confirmed.

4. Fix

Prompt: Update instructions to: "Output a raw JSON object. Do not use markdown backticks. The 'priority' field is mandatory and must be one of: high, medium, low."

• API Change: Set response_format: { "type": "json_object" } in the OpenAI client configuration.
• Parser: Implement a regex utility to strip backticks as a fallback.

5. Corrected Example

{
  "task_name": "Database Migration",
  "priority": "high"
}

6. Validation

• Test Case 1: Verify parser handles both raw JSON and fenced JSON.
• Test Case 2: Pass a prompt that implies a low-priority task and ensure the enum value "low" is generated.
• Test Case 3: Check for ValidationError when the model returns an empty string.

Next steps

• Update the Pydantic model to include a default value for 'priority' if possible.
• Enable Strict Mode in the OpenAI API call to enforce schema adherence at the provider level.
• Add a unit test to your CI/CD pipeline that validates the LLM prompt against the current JSON schema.