Langfuse Prompt A/B Testing: Offline First, Then Canary

Langfuse supports two useful but different ways to compare prompts. An offline experiment runs variants over a fixed dataset. A production A/B test routes live traffic to two prompt versions identified by labels. Treating those as one procedure creates a dangerous shortcut: a candidate can look good on a dataset yet fail on real traffic, or appear better in production because assignment and measurement were broken.

The practical sequence is simple:

1. test the candidate against a frozen dataset;
2. review failures and guardrails, not only an average score;
3. expose a bounded live cohort to the candidate;
4. keep subjects in one variant;
5. decide with a method chosen before looking at the result.

This guide uses the current Langfuse documentation captured on 30 August 2026 and a tested assignment fixture. The fixture covers the parts we can reproduce locally without an LLM call. It does not pretend to validate your model, dataset, or statistical design.

Offline experiment versus production A/B test

Langfuse calls the first workflow an Experiment. Its SDK runner can use a local or hosted dataset, execute a task for each item, attach item- or run-level evaluators, isolate failures, and create traces. A hosted Langfuse dataset also produces a dataset run that can be compared in the UI.

The separate prompt A/B testing guide uses labels such as prod-a and prod-b to fetch two prompt versions for live traffic. Langfuse can then associate latency, token usage, cost, and evaluation scores with the selected prompt version. It does not repair a bad cohort design for you.

Step 1: run the candidate on a fixed dataset

The dataset should represent the failure modes that matter to this prompt. A support answer prompt might include missing account context, contradictory policy text, unsupported requests, required links, several languages, and cases where the correct response is to stop. A random pile of easy examples is not a regression suite.

Record these boundaries before the run:

• exact dataset revision;
• baseline and candidate prompt versions;
• model, retrieval, tools, and generation settings held constant;
• primary outcome and critical guardrails;
• treatment of timeouts, empty outputs, retries, and evaluator errors.

The current JS/TS SDK exposes a dataset runner in this shape:

const dataset = await langfuse.dataset.get("support-answer-regression");

const result = await dataset.runExperiment({
  name: "support-prompt-v4",
  task: runSupportAnswer,
  evaluators: [hasRequiredLink, refusesUnsupportedRequest],
});

console.log(await result.format());

This snippet follows the official SDK contract; the local FutureCraft fixture does not call Langfuse and therefore does not claim this integration was executed here. Check the current experiment runner reference when implementing it. The SDK and its telemetry setup can change.

Do not compress evaluation into one score

A single mean can hide the exact regression you care about. Keep at least three views:

1. Primary outcome. The behavior the change intends to improve.
2. Guardrails. Failures that can veto release even when the primary outcome rises.
3. Case review. Inputs where variants disagree, evaluators fail, or the candidate produces a qualitatively different answer.

Langfuse supports deterministic code evaluators, human review, SDK/API scores, and LLM-as-judge evaluation. Those methods answer different questions. A JSON-schema check is reproducible but narrow. A human review can catch an invalid rubric. An LLM judge can scale a subjective criterion but introduces its own prompt, model, and calibration problem. See the separate guide to LLM-as-judge quality gates before making one judge score the release authority.

Step 2: label the two production variants

Langfuse prompt deployments use versions and labels. Keep the current prompt under a baseline label and the candidate under a separate canary label:

const promptA = await langfuse.prompt.get("support-answer", {
  label: "prod-a",
});
const promptB = await langfuse.prompt.get("support-answer", {
  label: "prod-b",
});

Link the selected prompt object to the traced generation so that observations remain attributable to its version. Do not overwrite the existing production label merely to start the test. The Langfuse version-control guide explains labels, fetching, and rollback behavior.

Step 3: make assignment sticky

The official A/B example demonstrates random selection. That is enough to show prompt fetching, but repeated subjects need a stronger allocation rule. Choosing with Math.random() on every request lets one account cross between variants. Conversation history, retries, and learned user behavior then leak across the comparison.

The downloadable fixture hashes an experiment key and a subject ID into one of 10,000 buckets:

const assignment = await assignVariant(
  "support-prompt-v4",
  privacySafeAccountId,
  0.1,
);

const selectedPrompt = assignment.variant === "candidate"
  ? promptB
  : promptA;

Use the unit that can actually experience the treatment: account, workspace, session, or request. Do not call them interchangeable. If an account shares context across users, request-level assignment is not independent. If requests are anonymous and one-shot, account-level allocation may be impossible.

The identifier does not need to be a raw email or database key. Generate a scoped, non-reversible experiment identifier and apply the site’s privacy and retention rules before attaching anything to a trace.

Step 4: instrument outcomes and guardrails

For each observation, retain enough information to audit the cohort without exporting private prompt content:

• experiment key and prompt version;
• assigned variant and bucket;
• privacy-safe subject identifier;
• primary outcome;
• guardrail outcomes;
• latency, cost or token usage when relevant;
• timeout, provider, tool, and evaluator failures;
• timestamp and analysis window.

The fixture’s summarizeExperiment() returns each variant’s primary-outcome rate, guardrail-failure rate, average latency, observed candidate share, and deviation from the planned allocation. It deliberately has no winner field.

That omission is a feature. A dashboard should not promote a prompt because one average moved up while a safety failure doubled or half the candidate observations were lost.

Step 5: choose the analysis before opening results

There is no responsible universal rule such as “run 200 examples and use a t-test.” The analysis depends on the unit, outcome, distribution, repeated observations, baseline, minimum useful effect, and stopping plan.

Before launch, write down:

• the smallest improvement worth the operational cost and risk;
• acceptable false-positive and false-negative error rates;
• whether observations are independent or clustered by subject;
• how missing and retried requests are handled;
• whether the test has a fixed horizon or a valid sequential design;
• which guardrail stops the canary regardless of the primary outcome.

Then calculate the required sample size for that design. If the resulting traffic requirement is unrealistic, do not disguise a tiny sample as proof. Keep the result descriptive, run a longer canary, improve the outcome signal, or decide from the offline evidence and risk boundary.

Step 6: promote or roll back explicitly

A clean release decision has four possible outcomes:

• promote because the candidate clears the primary and guardrail criteria;
• continue because the planned observation window is incomplete;
• reject because the candidate fails a guardrail or useful-effect threshold;
• inconclusive because data quality or power is insufficient.

Save the prompt versions, dataset revision, assignment code revision, query window, and analysis output. Langfuse labels make rollback operationally simple: restore the previous version to the production label. A rollback is still a change, so verify the served version and monitor new traces rather than assuming the label update propagated everywhere immediately.

What the downloadable fixture proves

The MIT-licensed TypeScript fixture passed Deno formatting, type checking, linting, and seven tests. It verifies:

• stable assignment for one experiment and subject;
• changed buckets for a different experiment key;
• bounded allocation over a deterministic 1,000-subject test set;
• invalid input and duplicate-subject rejection;
• separate outcome, guardrail, latency, and sample-ratio summaries;
• absence of an automatic winner.

It does not prove Langfuse SDK compatibility, model quality, statistical significance, or production safety. The package includes an EN/RU run checklist under CC BY 4.0 so the tested mechanics and the human release gates stay together.

The shortest honest workflow is therefore: offline dataset → failure review → sticky canary → predeclared analysis → explicit release decision. Langfuse supplies the prompt versions, traces, experiment runs, and scores. Your experiment design still decides whether those records mean anything.