AI Agent Evaluation Frameworks

Testing an LLM call is straightforward: send input, check output, compare to ground truth. Testing an AI agent is a different problem entirely. An agent doesn’t just produce an answer: it reasons, selects tools, executes multi-step plans, handles errors, and potentially modifies external state along the way. A correct final answer produced through a dangerous tool-calling sequence isn’t a success. It’s a time bomb.

The core challenge is that agent evaluation is fundamentally trajectory-dependent. Two agents can produce identical outputs while taking radically different execution paths, and only one of those paths might be safe for production. This guide covers the metrics, methods, tools, and implementation patterns you need to evaluate agents properly.

Why Output-Only Testing Breaks Down

Consider an agent tasked with “cancel order #4520 and refund the customer.” It calls cancel_order(4520), then issue_refund(4520, full=True), and responds “Done.” That looks correct. But what if the agent called cancel_order three times before succeeding because it didn’t parse the first response? Or what if it called get_all_orders() first: leaking every customer’s order data into its context window: before filtering down to #4520?

Output-only testing sees “Done.” and passes. Trajectory-based evaluation catches the retry storm, the data leak, and the wasted tokens.

This is the evaluation gap that kills production agents. Static benchmarks like GAIA or WebArena are useful for model selection, but they can’t predict production reliability because they don’t test against your specific schemas, error codes, and edge cases.

The Metrics Taxonomy

Effective agent evaluation operates across three layers. Miss one, and you’ll ship bugs that the other two can’t catch.

Outcome Metrics

These measure what the agent accomplished:

Task Success Rate: Binary pass/fail. Did the agent achieve the stated goal?
Goal Accuracy: How precisely the final state matches the intended outcome. Partial credit matters: an agent that cancels the order but forgets the refund is a 50% success, not a pass.
Containment Rate: The percentage of tasks completed autonomously without human escalation. This is the metric that determines whether your agent actually saves operational cost or just creates a different kind of support ticket.

Trajectory Metrics

These measure how the agent got there:

Tool Selection Accuracy: Did it pick the right tool for each step? An agent that uses search_database when get_record_by_id exists isn’t wrong, but it’s inefficient and potentially insecure.
Argument Correctness: Were the inputs to each tool valid? Passing user_id="4520" (string) instead of user_id=4520 (integer) might work in development and silently fail in production.
Step Efficiency: How many reasoning steps and tool calls did the agent need versus the optimal path? An agent that takes 12 steps to do a 3-step task is burning money.
Recovery Behavior: When a tool returns an error, does the agent recover gracefully, retry with corrected inputs, or spiral into an infinite loop?

Operational Metrics

These measure what it cost:

Latency: End-to-end response time. For tool-heavy agents, per-step latency matters as much as total.
Cost per Success: Token usage and API spend per completed task, not per call. An agent that retries five times before succeeding costs 5x what the metrics dashboard shows if you only track per-call cost.
Error Rate: Tool timeouts, schema validation failures, and unhandled exceptions. Track these separately from “the agent gave a wrong answer” failures.

graph TD
    Eval["Agent Evaluation"] --> Outcome["Outcome Metrics"]
    Eval --> Trajectory["Trajectory Metrics"]
    Eval --> Ops["Operational Metrics"]
    Outcome --> TSR["Task Success Rate"]
    Outcome --> GA["Goal Accuracy"]
    Outcome --> CR["Containment Rate"]
    Trajectory --> TSA["Tool Selection Accuracy"]
    Trajectory --> AC["Argument Correctness"]
    Trajectory --> SE["Step Efficiency"]
    Trajectory --> RB["Recovery Behavior"]
    Ops --> Lat["Latency"]
    Ops --> CPS["Cost per Success"]
    Ops --> ER["Error Rate"]

Three Evaluation Methods

No single method covers every failure mode. Production teams need all three, applied at different stages.

Deterministic Checks

Code-based assertions that catch objective failures. These are fast, cheap, and should run on every agent execution.

Schema Validation: Did the agent’s tool call arguments match the expected JSON schema?
Regex/Pattern Matching: Did the output contain required fields, valid IDs, or expected formatting?
State Assertions: After execution, is the system in the expected state? (Order cancelled? Refund issued? Database row updated?)

Deterministic checks are your first line of defense. If the agent calls a tool with malformed arguments, you don’t need an LLM judge to tell you it failed.

LLM-as-a-Judge

Use a stronger model to grade agent outputs and trajectories against a rubric. This handles the subjective and nuanced failures that deterministic checks can’t catch: things like “Was the agent’s response helpful?” or “Did the reasoning chain make logical sense?”

The catch: LLM judges are biased. Three biases will corrupt your metrics if you don’t mitigate them:

Bias	What Happens	Mitigation
Verbosity Bias	Judge rewards longer responses regardless of quality	Explicitly instruct the judge to penalize irrelevant content
Position Bias	In pairwise comparisons, judge favors the first or second option	Randomize order; run comparisons twice with swapped positions
Self-Preference	Judge favors outputs that mirror its own model’s style	Use a different model family for judging than for generation

Calibration is non-negotiable. Build a golden dataset of 50–200 examples labeled by domain experts. Measure agreement between your human labelers and the LLM judge using Cohen’s Kappa. If human-to-human agreement is below 80%, the rubric is ambiguous: fix the rubric before blaming the judge.

The most important calibration practice: force the judge to output its reasoning before assigning a score. A judge that explains “The agent selected delete_user instead of cancel_order, which would have deleted all user data” before scoring 1/5 is far more reliable than one that just outputs a number.

Trace-Based Analysis

Every agent execution generates a trace: the full sequence of inputs, reasoning steps, tool calls, tool outputs, and the final response. Trace analysis is how you answer “why did the agent fail?” rather than just “did it fail?”

Traces expose failure patterns that neither deterministic checks nor LLM judges catch in isolation:

Compounding Failures: An error at step 3 silently corrupts the context for steps 4–10. The final output might look plausible while being completely wrong.
Right Answer, Wrong Path: The agent arrived at the correct result but through an unsafe sequence (e.g., querying a public API with internal credentials).
Regression Loops: The agent calls the same tool repeatedly with identical arguments, burning tokens without making progress.

The Tooling Landscape

The evaluation tooling space has consolidated around a few clear categories. Pick tools based on where your team’s biggest blind spot is.

Tool	Category	Best For	Key Strength
DeepEval	CI/CD Testing	Pre-deployment validation	pytest-native; treat evals like unit tests
LangSmith	Tracing + Eval	LangGraph/LangChain teams	Deep trajectory semantics; native agent runtime integration
Arize Phoenix	Observability	Production monitoring	OpenInference tracing; drift and anomaly detection
Braintrust	Experimentation	Prompt iteration	Dataset diffing; side-by-side comparison of agent versions
MLflow	Full Lifecycle	End-to-end tracking	Broadest metric coverage; human feedback loops
Langfuse	Open-Source Tracing	Cost-sensitive teams	Self-hostable; LLM-agnostic tracing

A typical production stack combines two or three of these: one for CI/CD gating (DeepEval or Braintrust), one for runtime observability (Arize Phoenix or Langfuse), and optionally one for experiment tracking (MLflow).

Production Implementation: Agent Evaluation Harness

Here’s a complete Python evaluation harness that implements all three evaluation methods. It captures traces, runs deterministic checks, grades with an LLM judge, and produces a structured evaluation report.

from dataclasses import dataclass, field
from typing import Optional
from enum import Enum
import json
import time


class Verdict(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    WARN = "WARN"


@dataclass
class ToolCall:
    """A single tool invocation within an agent trace."""
    tool_name: str
    arguments: dict
    result: str
    latency_ms: float
    error: Optional[str] = None


@dataclass
class AgentTrace:
    """The complete execution trace of an agent run."""
    goal: str
    final_output: str
    tool_calls: list[ToolCall] = field(default_factory=list)
    total_tokens: int = 0
    total_latency_ms: float = 0.0


@dataclass
class EvalResult:
    """Result of a single evaluation check."""
    check_name: str
    verdict: Verdict
    score: float  # 0.0 to 1.0
    reason: str


@dataclass
class EvalReport:
    """Aggregated evaluation report for an agent run."""
    trace: AgentTrace
    results: list[EvalResult] = field(default_factory=list)

    @property
    def passed(self) -> bool:
        return all(r.verdict != Verdict.FAIL for r in self.results)

    def summary(self) -> dict:
        return {
            "goal": self.trace.goal,
            "overall": "PASS" if self.passed else "FAIL",
            "checks": [
                {
                    "name": r.check_name,
                    "verdict": r.verdict.value,
                    "score": r.score,
                    "reason": r.reason,
                }
                for r in self.results
            ],
            "tool_calls": len(self.trace.tool_calls),
            "total_tokens": self.trace.total_tokens,
            "total_latency_ms": self.trace.total_latency_ms,
        }


# ── Deterministic Checks ──────────────────────────────────────

def check_no_errors(trace: AgentTrace) -> EvalResult:
    """Verify no tool call returned an error."""
    errors = [tc for tc in trace.tool_calls if tc.error]
    if errors:
        failed_tools = ", ".join(tc.tool_name for tc in errors)
        return EvalResult(
            check_name="no_tool_errors",
            verdict=Verdict.FAIL,
            score=0.0,
            reason=f"Tool errors in: {failed_tools}",
        )
    return EvalResult(
        check_name="no_tool_errors",
        verdict=Verdict.PASS,
        score=1.0,
        reason="All tool calls succeeded.",
    )


def check_step_efficiency(trace: AgentTrace, max_steps: int) -> EvalResult:
    """Flag runs that exceed a step budget."""
    actual = len(trace.tool_calls)
    if actual > max_steps:
        return EvalResult(
            check_name="step_efficiency",
            verdict=Verdict.WARN,
            score=round(max_steps / actual, 2),
            reason=f"Agent used {actual} steps; budget was {max_steps}.",
        )
    return EvalResult(
        check_name="step_efficiency",
        verdict=Verdict.PASS,
        score=1.0,
        reason=f"Agent used {actual}/{max_steps} steps.",
    )


def check_no_duplicate_calls(trace: AgentTrace) -> EvalResult:
    """Detect regression loops: identical consecutive tool calls."""
    for i in range(1, len(trace.tool_calls)):
        prev = trace.tool_calls[i - 1]
        curr = trace.tool_calls[i]
        if prev.tool_name == curr.tool_name and prev.arguments == curr.arguments:
            return EvalResult(
                check_name="no_duplicate_calls",
                verdict=Verdict.FAIL,
                score=0.0,
                reason=f"Duplicate call: {curr.tool_name}({curr.arguments})",
            )
    return EvalResult(
        check_name="no_duplicate_calls",
        verdict=Verdict.PASS,
        score=1.0,
        reason="No consecutive duplicate tool calls detected.",
    )


# ── LLM-as-a-Judge (Simulated) ────────────────────────────────

def llm_judge_trajectory(trace: AgentTrace) -> EvalResult:
    """
    In production, this sends the trace to a grading model (e.g., GPT-4o
    or Claude 3.5 Sonnet) with a rubric prompt. Here we simulate the
    judge to show the integration pattern.

    Real implementation would look like:
        response = openai.chat.completions.create(
            model="gpt-4o",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": JUDGE_RUBRIC_PROMPT},
                {"role": "user", "content": json.dumps(trace_to_dict(trace))},
            ],
        )
        judgment = json.loads(response.choices[0].message.content)
    """
    tool_names = [tc.tool_name for tc in trace.tool_calls]

    # Simulated rubric evaluation
    has_dangerous_tools = any(
        t in tool_names for t in ["delete_user", "drop_table", "admin_override"]
    )
    if has_dangerous_tools:
        return EvalResult(
            check_name="llm_judge_trajectory",
            verdict=Verdict.FAIL,
            score=0.2,
            reason="Agent used dangerous tools not warranted by the goal.",
        )

    return EvalResult(
        check_name="llm_judge_trajectory",
        verdict=Verdict.PASS,
        score=0.9,
        reason="Trajectory is logical and tools match the goal.",
    )


# ── Evaluation Runner ──────────────────────────────────────────

def evaluate_agent_run(trace: AgentTrace, max_steps: int = 5) -> EvalReport:
    """Run the full evaluation suite against a single agent trace."""
    report = EvalReport(trace=trace)

    # Layer 1: Deterministic checks
    report.results.append(check_no_errors(trace))
    report.results.append(check_step_efficiency(trace, max_steps))
    report.results.append(check_no_duplicate_calls(trace))

    # Layer 2: LLM-as-a-Judge
    report.results.append(llm_judge_trajectory(trace))

    return report


# ── Demo ───────────────────────────────────────────────────────

if __name__ == "__main__":
    # Simulate an agent trace for "cancel order #4520 and refund"
    trace = AgentTrace(
        goal="Cancel order #4520 and issue a full refund.",
        final_output="Order #4520 has been cancelled and a full refund issued.",
        total_tokens=3420,
        total_latency_ms=2840.0,
        tool_calls=[
            ToolCall(
                tool_name="get_order",
                arguments={"order_id": 4520},
                result='{"status": "active", "amount": 99.00}',
                latency_ms=120.0,
            ),
            ToolCall(
                tool_name="cancel_order",
                arguments={"order_id": 4520},
                result='{"status": "cancelled"}',
                latency_ms=340.0,
            ),
            ToolCall(
                tool_name="issue_refund",
                arguments={"order_id": 4520, "amount": 99.00, "full": True},
                result='{"refund_id": "RF-8821", "status": "processed"}',
                latency_ms=280.0,
            ),
        ],
    )

    report = evaluate_agent_run(trace, max_steps=5)
    print(json.dumps(report.summary(), indent=2))

Running this produces a structured report:

{
  "goal": "Cancel order #4520 and issue a full refund.",
  "overall": "PASS",
  "checks": [
    {"name": "no_tool_errors", "verdict": "PASS", "score": 1.0, "reason": "All tool calls succeeded."},
    {"name": "step_efficiency", "verdict": "PASS", "score": 1.0, "reason": "Agent used 3/5 steps."},
    {"name": "no_duplicate_calls", "verdict": "PASS", "score": 1.0, "reason": "No consecutive duplicate tool calls detected."},
    {"name": "llm_judge_trajectory", "verdict": "PASS", "score": 0.9, "reason": "Trajectory is logical and tools match the goal."}
  ],
  "tool_calls": 3,
  "total_tokens": 3420,
  "total_latency_ms": 2840.0
}

The harness is extensible. Add domain-specific checks by implementing functions that take an AgentTrace and return an EvalResult. Wire them into evaluate_agent_run and they’ll appear in the report.

The Evaluation-First Development Workflow

The highest-performing agent teams don’t write evaluation after shipping. They write evaluation before writing agent code. This isn’t test-driven development in the traditional sense: it’s closer to defining your acceptance criteria so precisely that you can’t accidentally build the wrong thing.

The workflow:

Define golden examples: Before writing a single line of agent logic, create 20–50 representative input/output pairs with expected tool-call sequences. These aren’t just “input → output” pairs. They include the expected trajectory: which tools should be called, in what order, with what arguments.
Build the evaluation harness: Wire up deterministic checks for your specific domain (schema validation, required tool calls, forbidden tool calls). Add an LLM judge rubric tailored to your use case.
Write the agent: Now build the agent. Run it against your golden examples after every change.
Close the feedback loop: When the agent fails in production, export the trace, have a domain expert annotate it, and add it to the golden dataset. This is the flywheel that turns production failures into regression tests.

graph LR
    Define["Define Golden Examples"] --> Harness["Build Eval Harness"]
    Harness --> Agent["Write Agent Code"]
    Agent --> Run["Run Against Golden Set"]
    Run -->|Pass| Deploy["Deploy to Production"]
    Deploy --> Monitor["Monitor Traces"]
    Monitor -->|Failure| Annotate["Expert Annotates Trace"]
    Annotate --> Define

This flywheel is what separates teams that ship reliable agents from teams that ship demos. Every production failure makes the evaluation suite stronger, which makes the next version of the agent harder to break.

Cost and Latency Considerations

Evaluation itself has a cost, and it compounds fast if you’re not careful.

Don’t block the response path. Run evaluations asynchronously. Export traces to a queue and evaluate them in the background. Your user shouldn’t wait an extra 2 seconds because you’re running an LLM judge on every response.
Tiered evaluation. Run deterministic checks on 100% of traffic: they’re essentially free. Run LLM-as-a-judge on a sample (10–20%) or on flagged traces only. Reserve full human review for anomalies and low-confidence scores.
Monitor the judge. If your LLM judge’s score distribution shifts significantly over a two-week window, something changed: the agent’s behavior, the data distribution, or the judge itself drifted. Trigger a re-calibration audit.
Track cost-per-success, not cost-per-call. An agent that costs $0.02 per call but needs 8 calls to complete a task costs $0.16 per success. An agent that costs $0.05 per call but completes in 2 calls costs $0.10 per success. The second agent is cheaper despite the higher per-call price.

Frequently Asked Questions

How do you evaluate an AI agent? Evaluate agents across three layers: outcome metrics (did the task succeed?), trajectory metrics (did the agent take the right path: correct tools, correct arguments, minimal steps?), and operational metrics (latency, token cost, error rate). Use a mix of deterministic checks, LLM-as-a-judge grading, and trace-based analysis.

What is trajectory-based evaluation for AI agents? Trajectory-based evaluation inspects the entire sequence of reasoning steps, tool calls, and intermediate outputs an agent produces: not just the final answer. It catches ‘right answer, wrong path’ failures where an agent arrives at a correct result through unsafe or unreliable logic.

What is LLM-as-a-judge and how do you calibrate it? LLM-as-a-judge uses a stronger model to grade agent outputs against a rubric. Calibrate it by building a golden dataset labeled by domain experts, measuring inter-rater reliability with Cohen’s Kappa, and continuously feeding human corrections back into the judge’s few-shot examples.

What are the best AI agent evaluation tools in 2026? The leading tools are DeepEval (pytest-native CI/CD testing), LangSmith (LangGraph/LangChain trajectory tracing), Arize Phoenix (open-source observability), Braintrust (prompt experimentation), and MLflow (full lifecycle tracking). Choose based on whether your priority is CI/CD, observability, or experimentation.