Validation, Retry & Feedback Loops
How to build validation loops that catch semantic errors in Claude's output and retry with targeted error feedback. Covers retryable vs. non-retryable errors, the detected_pattern field for tracking dismissals, and self-correction via calculated vs. stated totals.
Quick Reference
- →Retry-with-error-feedback: append specific validation errors to the next attempt
- →Retries are effective for format/calculation errors but NOT when info is absent from source
- →Always include the original document + failed extraction + specific error messages on retry
- →Set a max retry limit (typically 2-3) to avoid infinite loops and cost explosion
- →Use a detected_pattern field to track recurring dismissal patterns across reviews
- →Self-correction: have Claude compute calculated_total from line items and compare to stated_total
- →conflict_detected boolean flags discrepancies for human review instead of silently choosing one
- →Validation happens AFTER schema parsing -- schemas catch syntax, validation catches semantics
- →Non-retryable errors (missing source data) should escalate to human review, not retry
Why Validation is Separate from Schema
In the previous article, we saw that tool_use schemas guarantee syntactically valid JSON. But valid JSON does not mean correct data. Validation is the second layer: it checks that the VALUES make semantic sense. This is where most production extraction errors actually live.
Extract, validate, and retry with specific error feedback until quality passes
| Error type | Schema catches it? | Validation catches it? | Example |
|---|---|---|---|
| Missing required field | Yes | N/A | No vendor_name in output |
| Wrong field type | Yes | N/A | total is string instead of number |
| Invalid enum value | Yes | N/A | category = 'misc' not in enum |
| Line items don't sum to total | No | Yes | Items sum to $550 but total says $500 |
| Date in wrong format | Partial | Yes | Schema says string, validation checks ISO 8601 |
| Vendor name from wrong section | No | Yes | Extracted 'Bill To' instead of 'Bill From' |
| Fabricated data not in source | No | Difficult | Date fabricated when source has none |
| Missing data from source | No | No | Source has PO number but extraction missed it |
Validation sits between schema parsing (automatic) and human review (expensive). Good validation catches the errors that schemas miss while keeping humans focused on the cases that automation truly cannot handle.