What is an LLM judge?

An LLM judge is a model with a narrower job than the ones answering your question. It does not write the answer from scratch. It reads what the other models produced and decides what the final result should be. The name sounds grand, but the work is closer to editing than writing.

Why you would want one

Run the same prompt through Claude, GPT-5, and Gemini and you get three answers that rarely line up word for word. Sometimes one is clearly better. More often each has a part the others missed. Picking a winner by hand means reading all three closely every time, which is the step people skip when they are busy. A judge does that reading for you and gives you one answer, so you are not left sorting through a stack.

What the judge actually does

A good judge does more than vote for one answer. It compares the candidates, keeps the points they agree on, takes the clearest explanation wherever it came from, and drops the parts that do not hold up. When the answers genuinely conflict, it resolves the conflict where it can and flags it where it cannot. The output reads like one considered answer rather than a summary of who said what.

It does not need to be the biggest model

People assume the judge has to be the most expensive model in the run. It usually does not. Judging is a reading-and-deciding task, not a generate-from-nothing task, so a capable mid-tier model handles it well. That keeps the combining step cheap while the responding models do the harder thinking. Mistral is a good fit for this in practice.

Where this shows up

Anytime several models answer and you get back one result, a judge ran in between. In a weave, that is the synthesis step: the responding models answer, the judge combines them, and you read one clean result instead of three. The same idea powers ranking, where the judge scores candidate drafts and keeps or fuses the best instead of blending everything together.

A judge will not turn a weak set of answers into a brilliant one. What it does is spare you the manual reconciling and hand you the benefit of several models without the homework. When the answer matters, that is the difference between a result you trust and three tabs you have to read yourself.

More from the blog