Your AI Agent Returns HTTP 200 With Confidently Wrong Answers — Fix It

Building an Eval Pipeline From Production Traces

The most effective eval pipeline I have built does not start with carefully constructed test cases. It starts with production traces. Real inputs, real agent behavior, real outputs reviewed by a human expert or a strong evaluator model. That dataset is ground truth in a way that synthetic benchmarks never are.

Here is the pipeline architecture. Production traces are sampled, filtered for interest (high latency, low confidence scores, or random sampling for baseline coverage), and routed to a review queue. Reviewers label outputs as correct, incorrect, or borderline. Labeled traces become evaluation cases. The eval suite runs against every deployment candidate. The pass rate becomes the gate.

// trace-to-eval-pipeline.ts — promote production traces to eval cases
interface ProductionTrace {
  runId: string
  input: string
  steps: Array<{ tool: string; input: unknown; output: unknown; latencyMs: number }>
  finalOutput: string
  confidenceScore: number
  timestamp: string
}

interface EvalCase {
  id: string
  input: string
  expectedOutputPattern: string // regex or semantic description
  expectedSteps: string[]       // required tool calls in order
  sourceTrace: string           // runId for lineage
  reviewer: string
  reviewedAt: string
}

async function promoteTraceToEvalCase(
  trace: ProductionTrace,
  review: { correct: boolean; expectedOutput: string; reviewer: string }
): Promise<EvalCase | null> {
  if (!review.correct) {
    // Wrong outputs are MORE valuable as eval cases — they define failure modes
    return {
      id: `eval-${trace.runId}`,
      input: trace.input,
      expectedOutputPattern: review.expectedOutput,
      expectedSteps: trace.steps.map(s => s.tool),
      sourceTrace: trace.runId,
      reviewer: review.reviewer,
      reviewedAt: new Date().toISOString(),
    }
  }
  // Sample correct outputs at 20% — maintain eval diversity
  return Math.random() < 0.2 ? {
    id: `eval-${trace.runId}`,
    input: trace.input,
    expectedOutputPattern: review.expectedOutput,
    expectedSteps: trace.steps.map(s => s.tool),
    sourceTrace: trace.runId,
    reviewer: review.reviewer,
    reviewedAt: new Date().toISOString(),
  } : null
}

The custom evaluator layer is where the real quality signal comes from. An LLM-as-judge evaluator, given a well-defined rubric, can score agent outputs for correctness, relevance, and absence of hallucination at a cost that makes continuous evaluation economically viable. The key is giving the evaluator enough context: the original input, the expected behavior, the actual output, and a structured rubric that converts a subjective quality judgment into a numeric score.

// custom-evaluator.ts — LLM-as-judge for production output quality
interface EvalRubric {
  name: string
  description: string
  scoringGuide: Record<number, string> // 1-5 scale with explicit definitions
}

const FACTUAL_ACCURACY_RUBRIC: EvalRubric = {
  name: 'factual_accuracy',
  description: 'Does the output contain only claims that are verifiable from the input context?',
  scoringGuide: {
    5: 'All claims are directly supported by input context. No hallucination.',
    4: 'All major claims supported. Minor unsupported details that do not affect correctness.',
    3: 'Core answer correct but includes 1-2 unsupported or unverifiable claims.',
    2: 'Mixed accuracy. Key claims are wrong or unverifiable.',
    1: 'Output is predominantly incorrect or hallucinates critical facts.',
  },
}

async function evaluateOutput(
  input: string,
  output: string,
  rubrics: EvalRubric[]
): Promise<Record<string, number>> {
  const scores: Record<string, number> = {}
  for (const rubric of rubrics) {
    const prompt = `You are evaluating AI agent output quality.

Input: ${input}
Output: ${output}

Rubric: ${rubric.description}
${Object.entries(rubric.scoringGuide).map(([score, desc]) => `${score}: ${desc}`).join('\n')}

Respond with JSON: {"score": <1-5>, "reasoning": "<one sentence>"}`
    const result = await callEvaluatorModel(prompt)
    scores[rubric.name] = result.score
  }
  return scores
}

Once you have an eval pipeline running on production traces, the quality alert becomes straightforward: compare rolling average scores against a threshold and alert when you cross it.

// quality-alerting.ts — semantic alerts on eval score degradation
interface QualityAlert {
  metric: string
  currentScore: number
  threshold: number
  windowHours: number
  samplesEvaluated: number
  alertAt: string
}

async function checkQualityAlerts(
  evalScores: Array<{ score: number; timestamp: string; metric: string }>,
  thresholds: Record<string, number>
): Promise<QualityAlert[]> {
  const alerts: QualityAlert[] = []
  const windowStart = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString()

  for (const [metric, threshold] of Object.entries(thresholds)) {
    const recentScores = evalScores.filter(
      s => s.metric === metric && s.timestamp >= windowStart
    )
    if (recentScores.length < 10) continue // insufficient sample

    const avgScore = recentScores.reduce((sum, s) => sum + s.score, 0) / recentScores.length
    if (avgScore < threshold) {
      alerts.push({
        metric,
        currentScore: avgScore,
        threshold,
        windowHours: 24,
        samplesEvaluated: recentScores.length,
        alertAt: new Date().toISOString(),
      })
    }
  }
  return alerts
}

// Usage: alert if factual accuracy drops below 3.5/5.0 over 24h
const QUALITY_THRESHOLDS = { factual_accuracy: 3.5, relevance: 3.8, completeness: 3.2 }

Tool Comparison — Braintrust vs Langfuse vs Arize Phoenix vs LangSmith

I have used all four in production contexts, and they each make a different bet on what the hard problem actually is. Choosing the wrong one does not kill your observability practice, but it does mean fighting the tool’s design instead of working with it.

Braintrust is built eval-first. The core product is a dataset-driven evaluation system with a human review UI, a score tracking timeline, and tight CI/CD integration. The tracing and monitoring capabilities exist as first-class features but feel downstream of the eval philosophy. If your primary problem is measuring and improving output quality across deployments, Braintrust is the best-designed tool for that specific job. It is not the cheapest option and it does not self-host, which matters for teams with data residency requirements.

Langfuse makes the opposite bet: it is built observability-first, with eval as a secondary layer you add on top. It is open source, self-hostable, and the logging SDK is genuinely lightweight. For teams that need data to stay on their own infrastructure — regulated industries, anything involving PII-heavy inputs — Langfuse is often the only viable option. The eval tooling is catching up to Braintrust but is not there yet. The tracing UI is excellent. I run a self-hosted Langfuse instance for the WOWHOW pipelines.

Arize Phoenix is the OSS option with the deepest ML observability lineage. Arize as a company comes from production ML monitoring, and Phoenix inherits that focus on embedding drift, data distribution monitoring, and the kind of statistical analysis that matters when your inputs shift over time. If you are running RAG pipelines or any agent whose quality degrades when the retrieval corpus drifts, Phoenix has capabilities the other three lack. The tradeoff is a steeper learning curve and a UI that is more analytics dashboard than operator console.

LangSmith is the default for anyone running LangChain, and it has earned its position through sheer scale — LangSmith has processed over 15 billion traces. The instrumentation is automatic if you are already in the LangChain ecosystem, and the hub of shared prompts and evaluators is genuinely useful for teams that want community-sourced eval patterns. Outside the LangChain ecosystem it is still capable but loses its key advantage. The pricing scales with trace volume, which can become a constraint at high-throughput production loads.

The practical decision framework: self-hosting requirement points you to Langfuse or Phoenix. LangChain-native stack points you to LangSmith. Eval-first culture or heavy human review workflows point you to Braintrust. All four support OpenTelemetry as an ingestion path, so you are not locked in — the instrumentation you write today works against any of them.

The Shadow Agent Problem

Here is the failure mode that will define the second half of 2026 for organizations that have been aggressively deploying AI agents: shadow agents. An agent built by one team without registering it in a central inventory. An agent a developer spun up for a one-time task that is still running six months later. An agent that was built on top of a deprecated internal API and now silently uses a fallback that nobody intended as a production endpoint.

Shadow agents exist in every organization that has been deploying agents faster than its governance processes have matured. They are not malicious. They are the natural byproduct of capable developers using capable tools to solve real problems quickly. The problem is that a shadow agent, by definition, has no eval pipeline, no quality alerting, no trace retention, and no owner who will respond when it starts producing wrong outputs.

The discovery problem is harder than it sounds. Agents are not like microservices that register themselves with a service mesh. They are processes that make LLM API calls. The only reliable way to find them is to audit your LLM API usage at the account level and cross-reference it against your known-registered agents.

# shadow-agent-discovery.py — find unregistered agents via API key audit
import httpx
import json
from datetime import datetime, timedelta

REGISTERED_AGENTS = {
    "content-research-agent",
    "pricing-sync-agent",
    "seo-executor-agent",
    # ... your known agents
}

async def discover_shadow_agents(api_key: str, lookback_days: int = 30) -> list[dict]:
    """
    Query LLM provider usage logs for API calls not attributable to
    registered agents. Requires usage metadata (user tag or custom header)
    to be set on all registered agent calls.
    """
    cutoff = datetime.utcnow() - timedelta(days=lookback_days)

    async with httpx.AsyncClient() as client:
        resp = await client.get(
            "https://api.anthropic.com/v1/usage",
            headers={"x-api-key": api_key, "anthropic-version": "2023-06-01"},
            params={"start_time": cutoff.isoformat(), "limit": 1000}
        )
        usage_records = resp.json().get("data", [])

    shadow_candidates = []
    for record in usage_records:
        agent_tag = record.get("metadata", {}).get("agent_id", "UNTAGGED")
        if agent_tag not in REGISTERED_AGENTS:
            shadow_candidates.append({
                "agent_tag": agent_tag,
                "model": record.get("model"),
                "input_tokens": record.get("input_tokens"),
                "output_tokens": record.get("output_tokens"),
                "timestamp": record.get("created_at"),
            })

    # Group by agent_tag to see usage patterns
    from collections import defaultdict
    by_agent: dict = defaultdict(list)
    for r in shadow_candidates:
        by_agent[r["agent_tag"]].append(r)

    return [
        {
            "agent_tag": tag,
            "call_count": len(records),
            "total_tokens": sum(r["input_tokens"] + r["output_tokens"] for r in records),
            "first_seen": min(r["timestamp"] for r in records),
            "last_seen": max(r["timestamp"] for r in records),
        }
        for tag, records in by_agent.items()
    ]

The discovery script is the easy part. The harder part is what you do with the results. A shadow agent that has been running for months, used by a business team for a real workflow, cannot be simply shut down without disruption. The practical response is a registration amnesty: give teams a window to register their shadow agents, bring them under the standard observability stack, and assign an owner. Make the bar for registration low enough that compliance is easy. The agents that do not get registered during the amnesty period are the ones you shut down — because if no one registered them, no one is responsible for them, and no one will respond when they start generating wrong outputs at scale.

The connection between shadow agents and the quality crisis is direct. Organizations reporting the highest rates of AI quality problems in production are the same organizations with the weakest agent registration practices. When you do not know an agent exists, you cannot evaluate it. When you cannot evaluate it, you find out it is wrong the same way that customer found out: after the damage is done, with a 200 response code in the logs and no trace of what actually happened.

The investment in observability infrastructure pays back unevenly but reliably. The first few months feel like instrumentation overhead. The first time your quality alert fires two hours after a provider model update and saves you from a day of wrong outputs reaching real users, the ROI calculation becomes obvious. I have had that experience once. It changed how I think about observability from an optional enhancement to a deployment prerequisite. You should not ship an agent to production that you cannot evaluate, and you should not evaluate an agent you cannot trace.

For more on building agents that are production-ready from day one, see the related posts on why 88% of agent pilots never reach production and the 3-layer agent harness pattern for keeping configuration complexity under control. The WOWHOW tools catalog includes utilities for structured agent logging that integrate with all four platforms covered here.