Eval-Driven Agent Architecture

Designing an agent so it can be measured — traces, spans, structured events, replayable tool layers, and the seams where evaluators attach.

🚀 advanced
⏱️ 120 minutes
👤 Bhanu Pratap Singh

· Agentic AI · 8 min read

📋 Prerequisites

  • Why Agent Evaluation Is Different (previous lesson)

🎯 What You'll Learn

  • Identify the architectural seams an evaluator needs to attach to
  • Instrument an agent loop with traces and spans using a minimal, OpenTelemetry-compatible model
  • Build a tool layer that can run live, mocked, or replayed from a recorded trace
  • Separate agent state from model calls so outcomes can be asserted directly

You Can’t Evaluate What You Can’t See

Most agent code is written to produce an answer. Evaluation needs more than the answer: it needs every decision, every tool call and its arguments, every observation, the state of the world before and after, the token counts, and the timings.

If that information is buried in print statements or lost inside a framework’s internals, your evaluation ceiling is the final string. This lesson is about raising that ceiling architecturally, so that evaluation is a property of how the agent is built, not an afterthought bolted on.

The principle is simple: design the agent so that a complete, structured record of every run falls out for free.

The Four Seams

There are four places an evaluator needs to attach. Design each as an explicit boundary in the code.

            ┌──────────────────────────────────────────┐
  input ──▶ │  Agent loop                              │ ──▶ response
            │   ┌────────────┐      ┌───────────────┐  │
            │   │ Model call │ ───▶ │ Tool executor │  │
            │   └────────────┘      └───────┬───────┘  │
            │         ▲                     ▼          │
            │         └─────── observation ─┘          │
            └───────────────┬──────────────────────────┘
                            ▼
                   Environment / state
  1. Model seam. Every LLM call goes through one function. You can record its inputs and outputs, swap the model, or replay a recorded response.
  2. Tool seam. Every tool call goes through one executor. You can record it, mock it, inject failures, or inject adversarial content.
  3. State seam. The world the agent changes (refunds, tickets, files) lives behind an interface you can snapshot and assert against.
  4. Trace seam. Every step emits a structured event into a trace that evaluators read after the run.

Get these four right and every lesson after this one gets easier. Get them wrong and you’ll spend your time scraping logs.

A Minimal Trace Model

You don’t need a vendor SDK to start. A trace is a tree of spans. Each span has a type, a start and end time, inputs, outputs, and attributes.

from __future__ import annotations
import time, uuid
from dataclasses import dataclass, field
from typing import Any, Literal

SpanKind = Literal["agent", "llm", "tool", "retrieval", "guardrail"]

@dataclass
class Span:
    name: str
    kind: SpanKind
    trace_id: str
    span_id: str = field(default_factory=lambda: uuid.uuid4().hex[:16])
    parent_id: str | None = None
    start: float = field(default_factory=time.time)
    end: float | None = None
    input: Any = None
    output: Any = None
    attributes: dict[str, Any] = field(default_factory=dict)
    error: str | None = None

    @property
    def duration_ms(self) -> float:
        return ((self.end or time.time()) - self.start) * 1000

@dataclass
class Trace:
    trace_id: str = field(default_factory=lambda: uuid.uuid4().hex)
    spans: list[Span] = field(default_factory=list)

    def of_kind(self, kind: SpanKind) -> list[Span]:
        return [s for s in self.spans if s.kind == kind]

    def tool_calls(self) -> list[tuple[str, dict]]:
        return [(s.name, s.input) for s in self.of_kind("tool")]

    def total_tokens(self) -> int:
        return sum(s.attributes.get("tokens.total", 0) for s in self.of_kind("llm"))

The attribute names here mirror the OpenTelemetry GenAI semantic conventions (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) closely enough that exporting to Langfuse, Arize Phoenix, LangSmith, or any OTel backend later is a mapping exercise, not a rewrite.

A context manager makes instrumentation nearly invisible:

from contextlib import contextmanager

class Tracer:
    def __init__(self):
        self.trace = Trace()
        self._stack: list[Span] = []

    @contextmanager
    def span(self, name: str, kind: SpanKind, input: Any = None, **attrs):
        s = Span(
            name=name, kind=kind, trace_id=self.trace.trace_id,
            parent_id=self._stack[-1].span_id if self._stack else None,
            input=input, attributes=dict(attrs),
        )
        self.trace.spans.append(s)
        self._stack.append(s)
        try:
            yield s
        except Exception as e:
            s.error = f"{type(e).__name__}: {e}"
            raise
        finally:
            s.end = time.time()
            self._stack.pop()

The Model Seam

All model calls go through one client wrapper. It records the request and response as an llm span and can be swapped for a replay client in tests.

class ModelClient:
    def __init__(self, tracer: Tracer, model: str, llm_fn):
        self.tracer, self.model, self.llm_fn = tracer, model, llm_fn

    def complete(self, messages: list[dict], tools: list[dict]) -> dict:
        with self.tracer.span("chat", "llm", input=messages,
                              **{"gen_ai.request.model": self.model}) as s:
            resp = self.llm_fn(model=self.model, messages=messages, tools=tools)
            s.output = resp["message"]
            s.attributes["tokens.input"] = resp["usage"]["input_tokens"]
            s.attributes["tokens.output"] = resp["usage"]["output_tokens"]
            s.attributes["tokens.total"] = sum(resp["usage"].values())
            return resp["message"]

llm_fn is whatever calls your provider. Keeping it injected means the same agent runs against Claude, GPT, or a local model, and an eval can compare them with no code changes.

The Tool Seam: Live, Mock, Replay

This is the most valuable seam. Evaluation needs to run tools in three modes:

  • Live: real APIs, for end-to-end smoke tests in staging
  • Mock: deterministic fakes backed by fixture data, for the bulk of offline evals
  • Replay: return exactly what a recorded production trace returned, to reproduce a failure
from typing import Callable, Protocol

class ToolBackend(Protocol):
    def call(self, name: str, args: dict) -> Any: ...

class ToolExecutor:
    def __init__(self, tracer: Tracer, backend: ToolBackend, hooks=None):
        self.tracer, self.backend = tracer, backend
        self.hooks = hooks or []   # fault and adversarial injection, Lesson 9

    def run(self, name: str, args: dict) -> Any:
        with self.tracer.span(name, "tool", input=args) as s:
            result = self.backend.call(name, args)
            for hook in self.hooks:
                result = hook(name, args, result)
            s.output = result
            return result

class ReplayBackend:
    """Returns tool outputs recorded in a previous trace, in order."""
    def __init__(self, recorded: Trace):
        self._queue = [(s.name, s.input, s.output) for s in recorded.of_kind("tool")]

    def call(self, name, args):
        if not self._queue:
            raise RuntimeError(f"replay exhausted at {name}")
        exp_name, exp_args, output = self._queue.pop(0)
        if exp_name != name:
            raise RuntimeError(f"replay diverged: expected {exp_name}, got {name}")
        return output

Replay mode is how you turn a production bug report into a reproducible test case: take the trace, replay the tool outputs, and check whether a new prompt fixes the decision that went wrong.

The State Seam

For Quill, the “world” is customers, invoices, and refunds. Put it behind an interface the mock backend reads and writes, and that evaluators can inspect.

@dataclass
class BillingState:
    customers: dict[str, dict]
    invoices: dict[str, dict]
    refunds: list[dict] = field(default_factory=list)
    escalations: list[dict] = field(default_factory=list)

    def snapshot(self) -> dict:
        import copy
        return copy.deepcopy(self.__dict__)

class MockBillingBackend:
    REFUND_LIMIT = 200.0

    def __init__(self, state: BillingState):
        self.state = state

    def call(self, name, args):
        s = self.state
        if name == "lookup_customer":
            return next((c for c in s.customers.values() if c["email"] == args["email"]),
                        {"error": "not_found"})
        if name == "get_invoices":
            return [i for i in s.invoices.values() if i["customer_id"] == args["customer_id"]]
        if name == "issue_refund":
            inv = s.invoices.get(args["invoice_id"])
            if inv is None:
                return {"error": "invoice_not_found"}
            if args["amount"] > self.REFUND_LIMIT:
                return {"error": "requires_approval"}
            s.refunds.append(dict(args))
            return {"status": "refunded", "refund_id": f"R{len(s.refunds)}"}
        if name == "escalate_to_human":
            s.escalations.append(dict(args))
            return {"status": "escalated"}
        if name == "search_kb":
            return [{"title": "Duplicate charges", "body": "Refund the duplicate within 30 days."}]
        return {"error": f"unknown_tool:{name}"}

Now an outcome grader can ask a precise question: does state.refunds contain exactly one refund, for invoice INV-2291, for $49? No parsing of prose required.

Putting the Loop Together

import json

@dataclass
class RunResult:
    response: str
    trace: Trace
    final_state: dict
    stopped_reason: str     # "final_answer" | "max_steps" | "error"

class QuillAgent:
    def __init__(self, model: ModelClient, tools: ToolExecutor,
                 tracer: Tracer, system_prompt: str, max_steps: int = 10):
        self.model, self.tools, self.tracer = model, tools, tracer
        self.system_prompt, self.max_steps = system_prompt, max_steps

    def run(self, user_message: str, state: BillingState) -> RunResult:
        messages = [{"role": "system", "content": self.system_prompt},
                    {"role": "user", "content": user_message}]
        with self.tracer.span("quill", "agent", input=user_message) as root:
            for step in range(self.max_steps):
                msg = self.model.complete(messages, TOOL_SCHEMAS)
                messages.append(msg)
                if not msg.get("tool_calls"):
                    root.output = msg["content"]
                    return RunResult(msg["content"], self.tracer.trace,
                                     state.snapshot(), "final_answer")
                for call in msg["tool_calls"]:
                    result = self.tools.run(call["name"], call["arguments"])
                    messages.append({"role": "tool", "tool_call_id": call["id"],
                                     "content": json.dumps(result)})
            root.error = "max_steps"
            return RunResult("", self.tracer.trace, state.snapshot(), "max_steps")

Notice what an eval gets from a single run():

  • response: for response-quality graders
  • trace: every LLM and tool span, for trajectory, cost, and latency graders
  • final_state: for outcome assertions
  • stopped_reason: for completion-rate metrics

Architecture Decisions That Help (or Hurt) Evaluation

Structured outputs over free text. If the agent must end with a JSON object ({"resolution": "refunded", "invoice_id": ...}) alongside its message, you can grade the decision deterministically and the prose separately.

Explicit stop reasons. “Final answer,” “max steps,” “tool error,” and “guardrail blocked” are different failures. Collapse them into “it returned something” and your completion metric becomes meaningless.

Config as data. System prompt, model name, temperature, tool list, and max steps should be one versioned config object. Every eval result is then tagged with exactly what produced it, which is what makes comparing two versions possible (Lesson 12).

@dataclass(frozen=True)
class AgentConfig:
    model: str
    system_prompt_version: str
    temperature: float = 0.0
    max_steps: int = 10
    tools: tuple[str, ...] = ("lookup_customer", "get_invoices", "search_kb",
                              "issue_refund", "escalate_to_human")

    def fingerprint(self) -> str:
        import hashlib, json
        return hashlib.sha256(json.dumps(self.__dict__, sort_keys=True).encode()).hexdigest()[:12]

Avoid hidden framework state. Agent frameworks are fine, but make sure you can get the full message history and every tool call out of them. If a framework hides retries or silently truncates context, your traces will lie to you.

Deterministic IDs and clocks in eval mode. Inject the clock and ID generator. Otherwise “refund created at 14:03:22” makes snapshot comparisons flaky.

Where Existing Tools Fit

SeamWhat to use
TraceOpenTelemetry with GenAI conventions; Langfuse, Arize Phoenix, LangSmith, Braintrust, Weights & Biases Weave as backends
ModelYour own thin wrapper; LiteLLM if you need many providers
ToolYour own executor; MCP servers can sit behind it as a backend
StateYour domain code, a test database, or containerized sandboxes for code and computer-use agents

The pattern is the same regardless: own the seams in your code, and let tools consume what flows through them.

Key Takeaways

  • Evaluation quality is capped by what the architecture exposes. Design for observability from day one.
  • Build four explicit seams: model, tool, state, and trace
  • A run should return the response, the full trace, the final state, and a stop reason
  • Tool backends should support live, mock, and replay modes
  • Version the agent’s configuration as data, so every eval result knows exactly what it measured

Exercise

Refactor an agent you’ve built (or implement Quill from the snippets above) so that:

  1. Every LLM and tool call produces a span
  2. Tools run against a mock backend backed by fixture state
  3. run() returns a RunResult with trace, final state, and stop reason
  4. You can take one recorded trace and replay it through ReplayBackend

Print a trace as an indented tree. If you can read the agent’s entire decision process from that tree, you’re ready for Lesson 3.

Part of a structured course

Agent Evaluation & Architecture

Design and build a complete Agent Evaluation Framework — traces, golden datasets, deterministic graders, calibrated LLM judges, trajectory and safety evals, reliability metrics, production monitoring, and CI regression gates. Advanced course for AI engineers and architects shipping agents to production.

Lesson 2 of 13 ⏱ 8 weeks advanced Free

Related Tutorials

🚀advanced ⏱️ 120 minutes

Online Evaluation and Production Monitoring

Sampling live traces, reference-free evaluators, drift detection, user-feedback signals, and the loop that turns production failures into new test cases.

Agentic AI7 min read
agent evaluationonline evaluationmonitoring +3
🚀advanced ⏱️ 300 minutes

Capstone: Build an Agent Evaluation Framework

Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.

Agentic AI6 min read
agent evaluationcapstoneproject +2
🚀advanced ⏱️ 120 minutes

Evals in CI/CD and Regression Gates

Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow.

Agentic AI9 min read
agent evaluationci/cdgithub actions +3
🚀advanced ⏱️ 90 minutes

Deterministic Graders and Assertions

Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight.

Agentic AI8 min read
agent evaluationgradersassertions +2