From Expert Judgment to Autonomous Optimization: Encoding Human Expertise into LLM Judges with DSPy
FORMAT: TalkLEVEL: IntermediateLANGUAGE: English
A single misread clause in a reinsurance contract can mean millions in liability. Our LLM pipeline could extract and summarize these documents, but how do you know the output is actually correct? String matching fails ("USD 5,000,000" vs "$5M" scores zero), human review at scale is unaffordable, and a single LLM-as-judge prompt gives inconsistent, uncalibrated scores. The real bottleneck was never generation; it was evaluation. This talk shows how we solved it in two steps, both built entirely in Python.
First, we encoded expert evaluation at scale. We collected structured reviews from five domain experts across different insurance areas, then used DSPy to distill their judgments into a panel of calibrated LLM judges, each targeting a single quality dimension (factual accuracy, task alignment, faithfulness, technical depth, clarity), weighted to reflect what experts actually care about. On our validation set, the panel's per dimension scores stayed within 10% of expert consensus across every dimension, with the same relative shape close enough to trust as a fitness function, not just a diagnostic. Expert disagreement turned out to be our most powerful debugging tool: where reviewers diverged, our definitions were ambiguous; where they agreed, we had ground truth.
Then we closed the loop. Using DSPy's MIPROv2 and GEPA optimizers, we wired the judge panel as a fitness function and let the system rewrite prompts autonomously. Regression guards prevent gaming across dimensions. Humans review only the final score delta and diff through a CI gate, and experts shifted from repetitive review to refining the evaluation rubric itself, which is something only they can do.
The stack is Python-native: DSPy, MLflow, LiteLLM, Pydantic. Every code snippet is production code, not pseudocode. You will leave with a concrete recipe for encoding expert knowledge into automated LLM evaluation and self-improving optimization, applicable to any domain where "correct" is nuanced: legal review, medical summaries, code review, customer support, or your own pipeline.
