STEP 7 OF 10

Evals & Monitoring

Two separate pages cover "is this agent/workflow actually working": Evals (/evals) for offline quality measurement, and Monitoring (/monitoring) for live activity.

Evals

/evals has five tabs: Datasets, Runs, Improve, Playground, Annotation Queue.

An eval run dispatches each dataset item through the same real execution path as a manual run — agent_execution_service, through the actual data-plane runtime — not a bare LLM call. Every EvalItemResult links to a real AgentExecution you can open on /runs/{id} and read step by step; cost and latency come from that execution, not an estimate.

Datasets is where you author eval sets — cases don't have to come from a fixture file:

# Create a dataset, add cases by hand
curl -X POST https://your-control-plane-host/api/v1/evals/datasets \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "Matter triage accuracy", "skill": "matter_triage"}'
curl -X POST https://your-control-plane-host/api/v1/evals/datasets/{dataset_id}/items \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"input_payload": {"content": "..."}, "expected_output": "...", "grading_rubric": {"method": "llm_judge"}}'

# Or draft cases from your own knowledge graph
curl -X POST https://your-control-plane-host/api/v1/evals/datasets/{dataset_id}/generate-from-kg \
  -H "Authorization: Bearer $TOKEN"

Generated cases land with source: generated, reviewed: false and are excluded from real eval runs until a human reviews them — POST /datasets/{dataset_id}/items/{item_id}/review. A run's pass rate is never inflated by cases nobody has actually checked.

A completed real run can also become a regression-test case directly:

curl -X POST "https://your-control-plane-host/api/v1/executions/{execution_id}/save-as-eval-case" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"dataset_id": "<dataset_id>"}'

Runs lists eval executions against a dataset, with pass rate and groundedness. The page computes regression detection client-side: if the latest run's pass rate drops 15 points or more below the trailing baseline (last 5 runs), it's flagged as a regression.

# Kick off a new eval run
curl -X POST https://your-control-plane-host/api/v1/evals/runs \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"dataset_id": "<dataset_id>", "agent_id": "<agent_id>"}'

# List / inspect runs
curl "https://your-control-plane-host/api/v1/evals/runs" -H "Authorization: Bearer $TOKEN"
curl "https://your-control-plane-host/api/v1/evals/runs/{run_id}" -H "Authorization: Bearer $TOKEN"

# Trend data for the pass-rate/groundedness charts
curl "https://your-control-plane-host/api/v1/evals/trends" -H "Authorization: Bearer $TOKEN"

Improve is prompt optimization scored against that same real-execution path — this is what used to be the standalone /optimization page (now a redirect here) — run it against an agent and a dataset, and a deployed improvement bumps that agent's spec_hash so the runtime reloads it.

Playground (POST /api/v1/evals/playground) is a one-off: run a single prompt against a chosen agent/model outside of a formal dataset, to sanity-check a change quickly.

Annotation Queue is where a human labels eval outputs the automated grader wasn't confident about:

curl "https://your-control-plane-host/api/v1/evals/annotation-queue" \
  -H "Authorization: Bearer $TOKEN"

curl -X POST https://your-control-plane-host/api/v1/evals/annotation-queue/{item_result_id}/label \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"label": "pass", "note": "Correct, grader missed the citation format."}'

Eval regressions are also checked automatically on a schedule — see the drift-sweep Celery beat task in app/tasks/eval_monitoring.py.

Monitoring

/monitoring has three tabs: Overview, Traces, Threads. Overview polls GET /api/v1/monitoring/overview every 10 seconds for a live fleet-wide view (tile per agent, eval health summary alongside it).

# Fleet-wide overview (polled every 10s in the UI)
curl "https://your-control-plane-host/api/v1/monitoring/overview" -H "Authorization: Bearer $TOKEN"

# One agent's monitoring detail
curl "https://your-control-plane-host/api/v1/monitoring/agents/{agent_id}" -H "Authorization: Bearer $TOKEN"
curl "https://your-control-plane-host/api/v1/monitoring/agents/{agent_id}/actions" -H "Authorization: Bearer $TOKEN"

# Traces -- individual execution traces (LLM calls, tool calls, timing)
curl "https://your-control-plane-host/api/v1/monitoring/traces" -H "Authorization: Bearer $TOKEN"
curl "https://your-control-plane-host/api/v1/monitoring/traces/{trace_id}" -H "Authorization: Bearer $TOKEN"

# Threads -- full conversations, grouped
curl "https://your-control-plane-host/api/v1/monitoring/threads" -H "Authorization: Bearer $TOKEN"
curl "https://your-control-plane-host/api/v1/monitoring/threads/{conversation_id}" -H "Authorization: Bearer $TOKEN"

Per-agent detail lives on that agent's own workspace at /agents/{id} — Run/Runs for execution history and a live timeline, Cost for spend, and the real per-run trace on /runs/{id}. There is no separate /agents/{id}/monitor route; that page's content (approval stats, chat) was folded into the Runs and Cost tabs once runs became a real, persisted AgentExecution record — see Step 8 — Runs & Boundaries.

What's next

Step 8 — Runs & Boundaries: the unified execution record behind every run, with a full step-by-step timeline and per-agent enforcement boundaries.

← Step 6 — Workflows

Step 8 — Runs & Boundaries →