# LLM-as-Judge: Automated Quality Gate for LLM Outputs in Production > How to use LLM-as-Judge for automated LLM output evaluation. Metrics, judge prompts, DeepEval, Langfuse integration, and CI/CD pipeline setup. > Author: Roman Belov · Published: 2026-03-13 · Source: https://futurecraft.pro/blog/llm-as-judge-automated-quality-gate/ LLM-as-Judge is a pattern where one language model evaluates another model's outputs against defined criteria. An automatic quality gate: every response gets checked before reaching the user, or after, for monitoring. Standard production monitoring metrics (200 OK, latency 340ms, rate limits within bounds) are useless for assessing quality — the model can hallucinate in a meaningful share of responses while HTTP status codes tell you nothing about it. Manual review doesn't scale. One person can handle 100 requests a day. At 10,000, nobody can. And quality degradation usually hits at scale: after a prompt update, a model swap, or a silent change on the provider side. This article covers how LLM-as-Judge works, which metrics to evaluate, and how to plug it into a production pipeline. ## How LLM-as-Judge works and its limitations The judge model receives a prompt with instructions plus the text being evaluated, then returns a score: a number, a category, or structured JSON. The judge doesn't generate content. It classifies and scores. Models handle this more consistently than generation. ``` User: "Recommend cafes in downtown Moscow" | v +--------------------+ | LLM Generator | -> "Here are 5 cafes: Coffemania near Patriarshiye..." | (GPT-5.4-mini) | +--------------------+ | v +--------------------+ | LLM Judge | -> { relevance: 0.9, factuality: 0.7, | (Claude Sonnet) | toxicity: 0.0, completeness: 0.8 } +--------------------+ | v Score < threshold? -> Alert / Block / Log ``` Research by Zheng et al. (2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena") showed that GPT-4 as a judge agreed with human ratings in 80%+ of cases. Two human annotators agreed with each other about 81% of the time. The gap between an LLM judge and a human is roughly the same as the gap between two humans. ## LLM output quality metrics: what to evaluate Metric choice depends on the task. Main categories below. ### Metrics for RAG systems | Metric | What it checks | When you need it | |--------|---------------|-----------------| | **Faithfulness** | Response is grounded in context, no fabricated facts | Always for RAG | | **Answer Relevance** | Response matches the question | Always | | **Context Relevance** | Retriever returned relevant documents | Debugging retrieval | ### Metrics for generative tasks | Metric | What it checks | When you need it | |--------|---------------|-----------------| | **Correctness** | Factual accuracy | When a reference answer exists | | **Completeness** | Response covers all aspects of the query | Complex queries | | **Toxicity** | No insults, harmful content | User-facing products | | **Hallucination** | Model doesn't fabricate facts | Always | ### Metrics for agent pipelines | Metric | What it checks | When you need it | |--------|---------------|-----------------| | **Tool Use Correctness** | Right tool with right arguments | Agent pipelines | | **Task Completion** | End result solves the task | Always for agents | In practice, start with two or three metrics. For RAG: faithfulness + answer relevance. For a chatbot: relevance + toxicity. For an agent: task completion. Add more as you find specific problems. ## Judge prompt structure for evaluation Evaluation quality comes down to the prompt. A working template for faithfulness: ```python FAITHFULNESS_JUDGE_PROMPT = """You are an impartial judge evaluating the faithfulness of an AI assistant's response. Faithfulness means: every claim in the response is supported by the provided context. Claims not found in context = unfaithful. ## Input **User Question:** {question} **Retrieved Context:** {context} **AI Response:** {response} ## Task 1. Extract each factual claim from the AI Response 2. For each claim, check if it is supported by the Retrieved Context 3. A claim is SUPPORTED if the context contains evidence for it 4. A claim is UNSUPPORTED if the context does not mention it or contradicts it ## Output (JSON only) {{ "claims": [ {{"claim": "...", "supported": true/false, "evidence": "..."}} ], "score": , "reasoning": "" }}""" ``` What makes this work: **Specific criteria.** "Rate the response quality" doesn't work. "Check that every fact is backed by context" works. The more specific the instruction, the more stable the scores. **Chain-of-thought.** The model first extracts claims, checks each one, then assigns a score. Without intermediate steps, scores are unstable. **Structured output.** JSON with a fixed schema, score from 0 to 1, reasoning in one sentence. This makes parsing and aggregation straightforward. ## Implementing LLM-as-Judge: three approaches ### 1. Python + LLM API Minimal implementation, no frameworks: ```python import json from litellm import completion def evaluate_faithfulness(question: str, context: str, response: str) -> dict: judge_response = completion( model="anthropic/claude-sonnet-4-6", messages=[{ "role": "user", "content": FAITHFULNESS_JUDGE_PROMPT.format( question=question, context=context, response=response, ) }], response_format={"type": "json_object"}, temperature=0, ) result = json.loads(judge_response.choices[0].message.content) return result eval_result = evaluate_faithfulness( question="Какие кафе в центре Москвы?", context="Кофемания: Патриаршие пруды. Сёстры: Покровка 6.", response="Рекомендую Кофеманию на Патриарших и Пушкин на Тверском бульваре.", ) # score: 0.5 (Кофемания confirmed, Пушкин is not) ``` Pros: full control, minimal dependencies. Cons: you write every metric yourself, no batch processing. If you work with multiple LLM providers, litellm lets you switch between them through a single interface — more on this in the article about [multi-provider LLM architecture](/blog/multi-provider-llm-architecture/). ### 2. DeepEval Open-source framework with built-in metrics. Works like pytest for LLM outputs. ```python from deepeval import evaluate from deepeval.test_case import LLMTestCase from deepeval.metrics import ( FaithfulnessMetric, AnswerRelevancyMetric, HallucinationMetric, ) faithfulness = FaithfulnessMetric(threshold=0.7, model="gpt-5.4") relevancy = AnswerRelevancyMetric(threshold=0.7, model="gpt-5.4") hallucination = HallucinationMetric(threshold=0.5, model="gpt-5.4") test_case = LLMTestCase( input="Какие кафе в центре Москвы?", actual_output="Рекомендую Кофеманию на Патриарших...", retrieval_context=["Кофемания: Патриаршие пруды. Сёстры: Покровка 6."], ) results = evaluate([test_case], [faithfulness, relevancy, hallucination]) ``` 14+ built-in metrics, pytest integration. LLM quality tests run alongside unit tests: ```python # test_llm_quality.py from deepeval import assert_test def test_travel_recommendations(): test_case = LLMTestCase( input="Кафе в Москве", actual_output=run_my_pipeline("Кафе в Москве"), retrieval_context=get_retrieved_docs("Кафе в Москве"), ) assert_test(test_case, [faithfulness, relevancy]) ``` ### 3. Langfuse Evaluations If you already use Langfuse for tracing, evaluations plug in on top. The judge model runs against each trace and attaches a score to it. Scores can be attached to an entire trace or to individual observations. If you haven't set up an observability stack yet, start with the [practical guide to LLM observability with Langfuse](/blog/llm-observability-langfuse/). ```python langfuse.score( trace_id="trace-abc-123", name="faithfulness", value=0.85, comment="1 of 7 claims not supported by context", ) ``` For production monitoring, Langfuse fits better than DeepEval: scores are tied to real traces, visible in the dashboard, with day-over-day quality degradation charts. ## Integrating LLM-as-Judge into CI/CD and production pipelines ### Pre-deploy: prompt regression testing Prompt changed? Run a dataset through the judge model before deploying. Score below threshold — deploy blocked. ```yaml # .github/workflows/llm-quality.yml name: LLM Quality Gate on: [pull_request] jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pip install deepeval - run: deepeval test run test_llm_quality.py env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ``` ### Runtime: gate before the response For high-stakes tasks, evaluate before sending the response: ```python async def generate_with_quality_gate(question: str) -> str: response = await generate_response(question) eval_result = await evaluate_faithfulness( question=question, context=retrieved_context, response=response, ) if eval_result["score"] < 0.7: return "Sorry, I'm not confident in the accuracy of this answer. Try rephrasing your question." return response ``` An extra LLM call per request. GPT-5.4-mini as a judge costs ~$0.15 per million input tokens. At 10,000 requests per day with a ~500-token prompt: about ~$0.75/day. ### Post-hoc: sample-based monitoring The most common scenario. Evaluation runs asynchronously: ```python import random traces = langfuse.fetch_traces(limit=100) sample = random.sample(traces.data, min(100, len(traces.data))) scores = [] for trace in sample: result = evaluate_faithfulness( question=trace.input, context=trace.metadata.get("context", ""), response=trace.output, ) scores.append(result["score"]) langfuse.score(trace_id=trace.id, name="faithfulness", value=result["score"]) avg_score = sum(scores) / len(scores) if avg_score < 0.75: send_alert(f"Faithfulness degraded: {avg_score:.2f}") ``` Cheaper than a runtime gate, but catches trends. Average faithfulness dropped from 0.88 to 0.71 over a week — something broke: the prompt, the retriever, or a model update on the provider side. ## LLM-as-Judge pitfalls and biases ### Position bias Judge models systematically prefer whichever answer appears first in pairwise comparisons. Zheng et al. (2023) measured a shift of up to 10-15%. Fix: run the evaluation twice with swapped order and average the results. Or use pointwise scoring instead of pairwise. ### Verbosity bias Longer answers get higher scores, even when a shorter answer is more accurate. In the judge prompt, explicitly state "response length does not affect the score" and include an example where a short answer receives the highest mark. ### Self-enhancement bias GPT-4 gives higher scores to GPT-4 outputs. Claude prefers Claude outputs. Fix: use a judge model from a different provider than the generator. Generate with GPT-5.4, evaluate with Claude Sonnet. Or the other way around. The broader question of trusting LLM outputs is a topic of its own — more on this in [TruthGuard: when AI agents lie](/blog/truthguard-ai-agents-lie/). ### Cost Every evaluation is an LLM call. A runtime gate on 10,000 requests/day means 10,000 extra calls. Options: a cheap model as judge (GPT-5.4-mini, Claude Haiku), sample-based evaluation, caching scores for similar pairs. ### The judge hallucinates too A judge model can give a high score to a response full of fabricated facts, if the hallucination sounds plausible. Partial fix: chain-of-thought + structured output. There is no complete fix. This is a fundamental limitation of the approach. ## Choosing the right judge model for each scenario | Scenario | Judge model | Why | |----------|------------|-----| | Pre-deploy tests | GPT-5.4 or Claude Sonnet 4.6 | Accuracy matters more than speed | | Runtime gate | GPT-5.4-mini or Claude Haiku | Cheap and fast | | Post-hoc monitoring | GPT-5.4-mini | Bulk processing | Rule of thumb: the judge model should be at least as capable as the generator. GPT-5.4-mini judging GPT-5.4-mini works. GPT-5.4-mini judging Claude Opus is unreliable. `temperature=0` for judge calls is mandatory. ## Tools for LLM evaluation | Tool | Focus | LLM-as-Judge | Self-hosted | Price | |------|-------|-------------|-------------|-------| | **DeepEval** | Testing | 14+ metrics | Yes (OSS) | Free | | **Ragas** | RAG evaluation | Faithfulness, relevance | Yes (OSS) | Free | | **Langfuse** | Observability + evals | Evaluator templates | Yes (OSS) | Free (self-hosted) | | **Phoenix (Arize)** | Observability + evals | Hallucination, QA | Yes (OSS) | Free | | **Braintrust** | Evals + logging | Custom scorers | Cloud | Free tier | For a startup: DeepEval for pre-deploy tests + Langfuse for production monitoring. Two open-source tools cover the entire cycle. ## Production setup for LLM-as-Judge ``` +---------------------------------------------------+ | CI/CD Pipeline | | | | PR with prompt change | | | | | v | | DeepEval: dataset x new prompt -> scores | | | | | v | | Score < threshold? -> Block merge | +---------------------------------------------------+ +---------------------------------------------------+ | Production | | | | User request -> LLM -> Response -> User | | | | | v (async) | | Langfuse trace | | | | | v (cron, hourly) | | Judge evaluation (sample) | | | | | v | | Score dashboard + alerts | +---------------------------------------------------+ ``` ## Where to start: step-by-step plan 1. Pick one metric. For RAG: faithfulness. For a chatbot: answer relevance. 2. Collect 20-30 examples by hand: questions, answers, ratings (good/bad). A golden dataset for calibration. 3. Write a judge prompt, run it against the golden dataset. Agreement with human ratings below 70%? Revise the prompt. 4. Add DeepEval to CI for tests on prompt changes. 5. Set up Langfuse evaluations for production monitoring. From zero to a working quality gate: two to three days. Golden dataset + judge prompt: a couple of hours. --- *Need help setting up LLM quality gates? I help startups build AI products and automate processes — [belov.works](https://belov.works).*