Introduction to (Multimodal) LLM-as-a-Judge

Yinghong Lan

SubscribeSign in

Introduction to (Multimodal) LLM-as-a-Judge<br>Generation–Verification Asymmetry, the Multifaceted Value of LLM-as-a-Judge, and Examples of Multimodal LLM-as-a-Judge

Yinghong Lan<br>Jun 14, 2026

Share

This writeup is an introduction on (Multimodal) LLM-as-a-Judge - a wide overview rather than a deep technical discussion.<br>Generation-Verification Asymmetry

Let’s begin by addressing this common question: if we provide the same context to both the generator and judge, why would a (Multimodal) LLM-as-a-Judge add value? Below are some common reasons:<br>Thanks for reading! Subscribe for free to receive new posts and support my work.

Subscribe

Verification is often easier than generation - a common metaphor here is “more people can critique and appreciate great artwork than create it.” The judge does not need to generate high quality and comprehensive answers - it just needs to recognize quality or gaps in one.

“Providing the same context” is not exactly true . The judge receives the generator’s output - e.g., retrieved frames, reasoning path, and final conclusions - in addition to the original context. Furthermore, compared to the original context, the additional artifact - the generator’s output - tends to be more specific to the actual problem to solve. The judge can compare it against specific guardrails and rubrics, and check consistency and gaps.

You can have multiple judges, one for each specific dimension , thereby breaking down complex matrices of quality and consistency requirements into more tractable metrics. In contrast, generators need to balance all these requirements in their output.

The generator often commits sequentially - token by token, chunk by chunk. The judge, in contrast, can review the final output holistically and catch errors or inconsistencies at a higher level.

Multifaceted Value of LLM-as-a-Judge

Next, let’s demystify a common misconception: that a judge is only useful for evaluation. In practice, LLM-as-a-Judge has many application scenarios across both online / inference and offline / training:<br>Online / Inference time<br>Quality Control : the judge can reject outputs that fail predefined quality rubrics, or escalate them to a human-in-the-loop - e.g., rejecting a multimodal agent’s answer if it isn’t grounded in the retrieved frames.

Best-of-N selection : the judge can pick the best from multiple candidates (or reasoning trajectories) the generator outputs - e.g., sampling five reasoning paths through a video and selecting the one with the highest grounding and consistency scores.

Self-refinement loops : the judge critiques the generator’s first-pass output (”reasoning skipped frames 30–45”) and the same generator revises with the judge’s feedback, iterating until the output clears the predefined quality bar.

Input into a downstream editor / post-processor : similar to self-refinement, except the judge’s feedback - e.g., missing visual elements, weak grounding, hallucinated entities - goes to a separate editor / post-processor, which fixes the issues directly rather than regenerating from scratch.

Agentic step verification : beyond judging the final output, the judge can validate each intermediate action - tool call, retrieved frame, reasoning step - before the agent commits to the next one, catching errors mid-trajectory rather than after the full answer is produced.

Offline / Training time<br>Training data filter : the judge can help filter existing human or synthetic data - e.g., removing flawed, ungrounded, or unverifiable reasoning trajectories - to curate higher quality training datasets.

Synthetic annotator : the judge can help annotate final outputs, trajectories, or intermediate steps - e.g., labeling (query, agent trajectory, final output) triples - to scale training data for the generator beyond what human annotators can produce.

Reward function for reinforcement learning : the judge can provide scalar rewards or preference pairs (chosen vs. rejected) for various RL methods, scaling beyond what human preference labeling can support.

Examples of Multimodal LLM-as-a-Judge

LLM-as-a-Judge can be applied across a diverse set of problems - for this writeup, I want to specifically discuss Multimodal LLM-as-a-Judge for multimodal understanding.<br>MLLM-as-a-Judge (Chen et al. 2024) - the first comprehensive study of Multimodal LLM-as-a-Judge - built human-annotated benchmarks for image-instruction pairs spanning image captioning, math reasoning, text reading, and infographics understanding. It assessed MLLM judgment's alignment with human annotators across three settings: scoring evaluation, pairwise comparison, and batch ranking. It showed that while MLLMs are closer to human judgment on pairwise comparison, there are still significant gaps in scoring and batch ranking . Furthermore, MLLM-as-a-Judge exhibits various biases (position bias, length bias, and self-preference), hallucinations, and...