An agent operations dashboard should help an operator answer a small set of consequential questions: Did the requested work finish? Was the result correct enough to use? What did the run consume? Where did it fail, and what should someone inspect next? Those questions require more than a chart of model calls. They require events that describe a task from request to verified outcome, plus measures whose denominators and failure states are explicit.
This tutorial builds that foundation with a portable event schema, illustrative PostgreSQL queries, and an acceptance test for accuracy and access. The example is hypothetical: a mid-market support team uses an agent to draft an answer from approved internal material, and a human checks the draft before it is sent. The design does not assume a particular agent framework or hosting platform. Adapt field names and SQL syntax to your system, and treat every example query as a starting point rather than executed code.
1. Decide what the dashboard is for
An operations dashboard is for people diagnosing and improving live work. It is not a quarterly executive scorecard, a model leaderboard, or a substitute for tracing individual requests. Its useful unit is a business task: one customer question, one research request, or one document-processing job. A task can contain multiple model calls, tool operations, retries, and human decisions. Counting those pieces as separate completed tasks makes the system look busier without showing whether it helped.
Begin by writing down the operator’s decisions. For example: “Should we pause this workflow because too many tasks require correction?” “Which tool is responsible for the rise in failed tasks?” “Are costs increasing because each task uses more steps, because retries rose, or because volume changed?” A dashboard that cannot change a decision is likely displaying telemetry rather than operational information.
Separate the measures into three layers. Work describes task volume and disposition: started, completed, failed, cancelled, or awaiting review. Quality describes whether completed work met an independently defined acceptance rule. Cost and latency describe the resources and elapsed time associated with the task. Keep these layers visible together. A lower cost per run is not an improvement if the system is returning more unusable answers; a high completion rate is not meaningful if “complete” only means that the model stopped generating text.
Define success in user terms before choosing an SLO. SLOs should measure behavior relevant to users, with a defined denominator and time window; a number without those definitions is difficult to interpret or compare. Google’s SRE guidance on implementing SLOs provides useful framing. For this dashboard, an example could be “the proportion of eligible tasks producing an answer that passes the review rule within one business day, measured over a rolling seven-day window.” The threshold is a decision for the team, not a universal value.
2. Prerequisites and setup
Before building charts, make sure the workflow exposes stable task identifiers and that someone can determine the eventual business disposition. You need a database or warehouse that supports the SQL patterns below, an agreed reporting timezone, and a documented definition of each status. You also need a source for cost data. If exact provider charges are unavailable or arrive later, record the cost estimate and its method separately from confirmed charges rather than mixing them silently.
Set up a small reporting dataset with two conceptual layers. The raw event table records what happened and when. A derived task-level view converts those events into one current row per task. Keep the raw events append-oriented where practical: late-arriving corrections and review outcomes should be added as new facts rather than overwriting history. This makes it possible to explain why a historical chart changed and to reconstruct the sequence behind a failure.
Choose event retention and access according to your organization’s policies and the sensitivity of the work. Avoid putting full prompts, customer messages, credentials, or unrestricted tool output in a general-purpose metrics table. Store only what the dashboard needs, such as an internal task key, workflow name, status, timing, counts, and a review label. If deeper debugging needs protected payloads, link them through a controlled system using an identifier, not by copying them into every charting dataset.
For tracing context, an operation can carry a trace identifier and spans can help correlate related operations. Correlation helps an investigator follow a chain; it does not establish that the task succeeded. OpenTelemetry’s tracing concepts describe traces and spans as a way to connect operations. Keep business outcome recording separate from trace collection so a technically complete trace cannot be mistaken for a verified result.
3. Step 1: Define the event contract
Create an event contract before connecting a visualization tool. Each row should represent a meaningful state change or measurement, not an arbitrary log line. A compact starting schema is shown below. Names are illustrative; add fields only when a defined reporting or investigation need requires them. These queries operate on one authorized tenant’s dataset, with globally unique task IDs and one immutable workflow per task. A shared database needs tenant enforcement and tenant-qualified keys, joins and aggregates before these examples are used. The example stores USD only; use separate currency totals or an explicit conversion policy for other currencies.
| Field | Purpose | Example |
|---|---|---|
event_id | Unique identifier for deduplication | evt_01J... |
task_id | Stable identifier for one business task | task_8f2... |
event_time | Time the event occurred, stored consistently | 2026-09-30 14:02:11+00 |
event_type | Controlled event name | task_started, tool_finished |
workflow | Workflow or agent process identifier | support_draft_v2 |
environment | Deployment stage | production |
status | Event-level outcome where applicable | success, error, pending |
operation | Model, tool, review, or orchestration step | search, draft, human_review |
duration_ms | Measured duration for this operation | 1840 |
input_tokens, output_tokens | Usage values when supplied by the system | 1200, 340 |
cost_expected | Whether this event should carry a priced charge | true |
cost_amount | Cost amount, if available | 0.006 |
cost_currency | Currency for that amount | USD |
review_result | Independent task-quality outcome | accepted, corrected, rejected |
trace_id | Optional correlation key for technical diagnosis | 4bf9... |
Use a controlled vocabulary for event_type, status, and review_result. A free-text status such as “worked,” “done-ish,” or “needs another look” cannot be reliably grouped. Document whether task_completed means that the agent produced an output, the workflow ended, or a reviewer accepted the result. These are different milestones and should usually have different event names.
Avoid relying on the latest event alone to define truth. A task might emit task_completed, then fail review and emit task_reopened. Another might time out after a tool operation, while a delayed result arrives later. Store enough history to represent those changes and make the task-level view apply an explicit precedence rule. Do not assume that a missing event means a successful outcome.
A useful minimum event sequence for the hypothetical support workflow is task_started, zero or more operation events, one of task_completed or task_failed, and, when applicable, review_recorded. The review event should identify the rule or reviewer outcome, not include sensitive review notes by default. If work can be cancelled, distinguish cancellation from failure; cancellation may be an expected operator choice rather than a system defect.
4. Step 2: Write events consistently
Create the table in a reporting database. This PostgreSQL-like DDL is illustrative and deliberately leaves indexing, partitioning, and access controls to the team’s platform conventions.
CREATE TABLE agent_operation_events (
event_id text PRIMARY KEY,
task_id text NOT NULL,
event_time timestamptz NOT NULL,
event_type text NOT NULL,
workflow text NOT NULL,
environment text NOT NULL,
status text,
operation text,
duration_ms bigint,
input_tokens bigint,
output_tokens bigint,
cost_expected boolean NOT NULL DEFAULT false,
cost_amount numeric(18, 8),
cost_currency text,
review_result text,
trace_id text,
CHECK (cost_amount IS NULL OR (cost_currency IS NOT NULL AND cost_currency = 'USD'))
);
CREATE INDEX agent_events_task_time_idx
ON agent_operation_events (task_id, event_time);
CREATE INDEX agent_events_time_workflow_idx
ON agent_operation_events (event_time, workflow);
At the point where the application records an event, generate a unique event identifier and reuse the same task identifier throughout that task’s lifecycle. The writer should reject or quarantine events that lack required identifiers or have an invalid event type. A failed write must not silently turn into a missing success measure. Decide whether to retry the write, queue it for later, or alert an operator, and record enough information to detect dropped telemetry without logging sensitive payloads.
Do not treat the primary key as a guarantee that upstream delivery happens only once. A retry may deliver the same event again; the database constraint can reject a duplicate, but the ingestion path still needs a defined response. If the source can produce distinct events with the same task and timestamp, do not use those fields as a substitute for event_id. The queries use event ID only as a deterministic tie-breaker; a production state reducer should use a trusted source sequence when timestamps cannot establish causal order. Validate that the workflow is immutable for each task; the view’s MIN(workflow) is not a conflict-resolution rule. If event ingestion is at-least-once, deduplicate by the stable event identifier before aggregation.
For cost, record what is actually known. If the workflow reports usage but a billing record is delayed, label a value derived from usage as an estimate and retain the derivation method. Mark billable events with cost_expected = true. If those operations have no cost data, represent the amount as unknown rather than zero. Non-billable state events do not create artificial missing-charge counts. A zero implies a measured absence of cost; a null or separate availability flag communicates that the number is missing. Mixing unknown with zero will understate cost per completed task.
5. Step 3: Build a task-level view
The view keeps one row per task across its full event history. It derives disposition only from business-state events and derives the most recent review separately. A later usage or review event cannot erase completion. Filter the start-time cohort in downstream queries, after state has been reconstructed, so a period boundary does not discard earlier state.
CREATE VIEW agent_task_rollup AS
WITH events AS (
SELECT * FROM agent_operation_events
WHERE environment = 'production'
),
state_ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY task_id ORDER BY event_time DESC, event_id DESC
) AS state_rank
FROM events
WHERE event_type IN (
'task_started', 'task_completed', 'task_failed',
'task_cancelled', 'task_reopened'
)
),
review_ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY task_id ORDER BY event_time DESC, event_id DESC
) AS review_rank
FROM events WHERE event_type = 'review_recorded'
),
totals AS (
SELECT
task_id,
MIN(workflow) AS workflow,
MIN(CASE WHEN event_type = 'task_started' THEN event_time END)
AS started_at,
MAX(event_time) AS last_event_at,
SUM(input_tokens) AS input_tokens,
SUM(output_tokens) AS output_tokens,
SUM(cost_amount) AS recorded_cost,
COUNT(cost_amount) AS cost_event_count,
COUNT(*) FILTER (WHERE cost_expected AND cost_amount IS NULL)
AS unpriced_events,
SUM(duration_ms) AS summed_operation_ms
FROM events GROUP BY task_id
)
SELECT
t.*,
s.event_type AS latest_event,
s.status,
r.review_result,
r.event_time AS last_review_at
FROM totals t
LEFT JOIN state_ranked s ON s.task_id = t.task_id AND s.state_rank = 1
LEFT JOIN review_ranked r ON r.task_id = t.task_id AND r.review_rank = 1;Expected result: one row per task, including its latest business-state event, latest review, start time if present, and accumulated usage. A row with no terminal event is not automatically a failure. It may be in progress, stalled, or missing telemetry. Add an age rule to distinguish active work from work that has exceeded an operational threshold, and show those states separately.
There are two important limitations in this simple query. First, summing duration_ms gives cumulative operation time, not end-to-end elapsed time. Parallel operations can make cumulative time longer than the wall-clock task duration. Calculate elapsed time from the appropriate start and finish timestamps when reporting task latency. Second, summing cost is meaningful only if every cost row uses a consistent currency and the event semantics prevent the same charge being recorded twice.
For a production view, build and test a deterministic disposition rule. One option is to derive status from the latest valid business-state event, while retaining the latest technical event separately. That allows a task to be business-complete even if a cleanup operation later errors, or to be technically finished but still awaiting review. Name those columns plainly; a single success flag tends to collapse distinctions that operators need.
6. Step 4: Add dashboard queries
Start with the task-volume and disposition chart. The query below counts tasks started in the last seven days by their current business outcome, using the full-history view. The illustrative interval uses the database’s current time; a production dashboard should make its reporting timezone and interval boundaries explicit.
SELECT
CASE
WHEN latest_event = 'task_completed' THEN 'completed'
WHEN latest_event = 'task_failed' THEN 'failed'
WHEN latest_event = 'task_cancelled' THEN 'cancelled'
ELSE 'in_progress_or_unclassified'
END AS disposition,
COUNT(*) AS tasks
FROM agent_task_rollup
WHERE started_at >= CURRENT_TIMESTAMP - INTERVAL '7 days'
GROUP BY 1
ORDER BY 1;Expected output is a small set of disposition rows and counts. The “in progress or unclassified” bucket is intentionally visible. If it grows, investigate whether the workflow is genuinely slow, emits incomplete events, or uses a terminal event missing from the mapping. Hiding it from the visualization can make an instrumentation gap look like healthy completion.
Next, measure quality using reviewed tasks as the denominator. This query reports each task’s latest review outcome when that latest review occurred in the interval; multiple reviews do not count the same task in several outcome groups. It does not claim that unreviewed tasks are correct or incorrect.
SELECT workflow, review_result, COUNT(*) AS reviewed_tasks
FROM agent_task_rollup
WHERE last_review_at >= CURRENT_TIMESTAMP - INTERVAL '7 days'
GROUP BY workflow, review_result
ORDER BY workflow, review_result;Expected output shows accepted, corrected, and rejected counts by workflow, where those are the values your review contract uses. Add the review coverage rate alongside the outcome distribution: reviewed eligible tasks divided by all eligible tasks. A quality chart without coverage can improve merely because difficult cases were not reviewed. If reviewers sample tasks, record the sampling method and do not present the sample as a census.
For cost, show both recorded total and cost per eligible completed task, and expose the fraction of tasks with complete cost data. The illustrative calculation below assumes a single currency and one deduplicated cost total per task. Do not use it as written if cost rows mix currencies or partial estimates with confirmed charges.
SELECT
workflow,
SUM(recorded_cost) AS recorded_cost_total,
CASE WHEN SUM(unpriced_events) = 0 THEN
SUM(recorded_cost)
/ NULLIF(COUNT(*) FILTER (WHERE latest_event = 'task_completed'), 0)
END AS recorded_cost_per_completed_task,
COUNT(*) FILTER (WHERE unpriced_events > 0) AS tasks_with_unpriced_events,
COUNT(*) AS tasks_in_view
FROM agent_task_rollup
WHERE started_at >= CURRENT_TIMESTAMP - INTERVAL '7 days'
GROUP BY workflow;Expected output is one row per workflow. Interpret cost per completed task alongside failure and quality rates. The numerator includes recorded costs from completed, failed and still-open tasks in the same start-time cohort. Excluding failed-task costs would understate the cost of producing completed work. Missing expected charges suppress the ratio rather than silently becoming zero. Also report cost per attempted task and the costs associated with failed or rejected tasks. Label each denominator directly in the chart title or nearby note.
For latency, calculate the elapsed time between task start and the business completion event, then report a distribution rather than only an average. Median and a high percentile help distinguish a typical run from a tail of slow tasks; choose the percentile based on the decision and traffic volume. Keep timeouts and tasks still open beyond the threshold visible. A latency chart that includes only successful tasks can conceal the slowest failures.
7. Step 5: Make the dashboard operational
A useful first screen should show a current reporting window, task volume, disposition, review coverage and outcome, cost completeness, and a latency distribution. Use a compact trend for each measure, but do not pack every event field into a tile. A person investigating an incident should be able to filter by workflow, environment, date range, disposition, and review result, then move from an aggregate to a task identifier and its event sequence.
The task-level detail should answer “what happened, in what order?” Show timestamps, event types, operation names, status, duration, usage fields, and review outcome. Keep trace identifiers available for technical follow-up without treating them as a quality score. A trace may help locate a slow tool operation or a failed span; the business outcome still comes from the task’s own completion and verification events.
Use explicit labels such as “reviewed-task acceptance rate” rather than “accuracy” unless the measurement actually supports that broader claim. “Accepted” may mean that a reviewer approved a draft under a particular rubric; it does not automatically measure factual correctness across all tasks. Put the rubric version or review policy into the reporting dimensions when changes over time would alter the meaning of the metric.
Here is a small operational flow for interpreting an increase in unaccepted work. The branch is deliberately decision-oriented: it directs the operator to distinguish a real quality decline from a measurement change before escalating a workflow change.
flowchart TD
A["Review outcomes worsen"] --> B["Check review coverage"]
B --> C["Coverage stable?"]
C -->|"No"| D["Inspect missing reviews"]
C -->|"Yes"| E["Segment by workflow"]
E --> F["Inspect task events and traces"]
F --> G["Change, pause, or monitor"]
accTitle: Triage of worsening review outcomes
accDescr: Start with a worsening review measure. Check review coverage; if coverage changed, inspect missing reviews. If coverage is stable, segment by workflow, inspect task events and traces, then choose whether to change, pause, or monitor.A coverage decline is a measurement warning, not evidence that agent quality improved or worsened. If coverage is stable, segmenting can show whether the change is concentrated in one workflow or operation. Inspect examples only through the access path approved for that data. The final action should be recorded with the metric window and the observed evidence so a later operator can tell what prompted it.
8. Step 6: Run an accuracy and access acceptance test
Before relying on the dashboard, test the measurement path with a small, labelled fixture. The following is an illustrative test design, not a claim that any code was executed. Create six synthetic task IDs in a non-production dataset: two accepted after review, one corrected, one rejected, one failed before review, and one completed without a review event. Give each a known event sequence and distinct cost completeness: four have recorded cost, one has a confirmed zero-cost value if that is possible in your system, and one has unknown cost.
Write down the expected outputs before running queries. The disposition query should count six distinct tasks, not the number of events. The review outcome query should include four reviewed tasks, not the failed or unreviewed task. The coverage denominator should include only tasks eligible for review under the stated policy, and the numerator should include only those with a recorded review. The cost-completeness count should identify the task with missing cost rather than quietly counting it as zero.
Then introduce two deliberate data defects in the test copy. Duplicate one event with the same event_id, and add a task whose start event exists but whose terminal event is missing. Confirm that the duplicate is rejected or deduplicated according to the ingestion rule, and that the open task appears in the unclassified or stale-work view. If the dashboard instead reports an extra completed task or hides the open one, the acceptance test has found a concrete measurement defect.
Test access using two roles that represent real operational needs: a dashboard reader and a restricted investigator. The reader should see aggregate counts and only the task fields approved for general operations. The investigator’s access to sensitive payloads, if required, should be granted through the relevant controlled system rather than by widening the dashboard dataset. Verify both what each role can see and what happens when it attempts an action or query outside its intended scope. Record the expected result and the observed result; do not infer access control merely from a dashboard that happens not to display a column.
A simple test matrix makes the acceptance criteria reviewable:
| Test | Expected result | Failure signal |
|---|---|---|
| Duplicate event identifier | One event contributes to aggregates | Counts or usage increase twice |
| Missing terminal event | Task remains open or unclassified | Task disappears or counts as completed |
| No review event | Excluded from reviewed outcomes; coverage reflects it | Counted as accepted or rejected |
| Unknown cost | Marked missing, not silently zero | Cost per task is understated without warning |
| Restricted reader | Sees only approved operational fields | Sensitive detail is exposed or access is unclear |
| Time boundary | Events fall in the documented reporting window | Counts vary with undocumented timezone assumptions |
The access test is about the whole reporting path: source table, derived view, dashboard, and any task-detail link. A restricted dashboard view is not sufficient if the same role can query a broader reporting table. Ask the data owner to verify the intended boundary at each layer and retain a dated test result alongside the dashboard’s metric definitions.
9. Step 7: Diagnose the common misleading patterns
Completion rises while accepted work falls. This often means that operational completion and business acceptance have been combined or that their event definitions changed. Compare the task-level disposition with the review result. Check whether the reviewed population changed, whether the rubric changed, and whether review coverage is stable. Do not “fix” the chart by redefining completion to mean acceptance; preserve both measures and explain their relationship.
Cost per completion drops sharply. First check for missing cost, missing failed tasks, and duplicate or absent completion events. A lower numerator can mean the measurement lost charges; a higher denominator can mean retries or duplicate task IDs are being counted as completed work. Compare cost per attempted task, cost-data completeness, and the number of distinct task IDs before attributing the change to efficiency.
Latency improves while tasks accumulate. Successful tasks may be finishing quickly while a growing group remains in progress, or the dashboard may calculate duration only for terminal events. Plot open-task age and the share of tasks with no terminal event. Define a stale threshold in operational terms, and distinguish a workflow that is still legitimately running from one with a missing completion signal.
A tool error appears to prove that a task failed. An operation-level error and a task-level failure are not interchangeable. The workflow may recover, retry, or complete using another path. Display operation failures as diagnostic signals, then use the business disposition and review events to determine the task outcome. Conversely, a trace with no recorded error does not prove that the answer was useful.
Historical counts change after a refresh. Late events, corrected reviews, replayed ingestion, or a changed status mapping can alter a historical aggregate. Preserve raw events and version the transformation or metric definition. If corrections are valid, show when the outcome was recorded and when the underlying task occurred. A dashboard should explain whether a date means event occurrence, review completion, or ingestion time.
An apparent quality improvement coincides with a new review process. Check whether eligibility, sampling, reviewer guidance, or the acceptance rubric changed. Record review-policy versions and compare like with like where possible. If a policy change makes comparisons invalid, mark the boundary rather than drawing a continuous trend line that implies consistent measurement.
10. Step 8: Put the dashboard into a working cadence
Assign a named operational owner for the event contract and metric definitions, not just the visualization. When a workflow adds a new terminal state or changes review behavior, the owner should update the event mapping and acceptance fixture before the chart is treated as comparable. A small change log with date, field or rule changed, reason, and affected measures prevents silent reinterpretation.
For daily operations, use the dashboard to find unusual changes and route an investigation. For deeper technical context, connect task identifiers to the team’s observability workflow and inspect the relevant operation sequence. The implementation questions around traces, logs, and metrics are covered separately in AI Agents in Production: Agent Observability. Keep the current dashboard focused on task outcomes rather than duplicating a full observability system.
If an agent produces SQL or business analysis as part of its work, validate the meaning of the data it queries separately from whether the query ran. The semantic layer can define governed business concepts that a raw warehouse table name does not explain; see Agents Don’t Query Your Warehouse — They Query Your Semantic Layer for that distinct design concern. This tutorial’s event model records operational facts; it does not make an agent’s analytical answer correct by itself.
Likewise, keep other measurement questions in their proper datasets. Search referral and analytics discrepancies need their own reconciliation rather than being attributed to agent performance; Why Search Console Clicks and GA4 Visits Disagree treats that issue separately. Product and services referral funnels also need explicit segmentation; Measuring AI-Search Referrals Without Mixing Product and Services Funnels covers that measurement problem. These boundaries help prevent an operations dashboard from becoming a catch-all report with incompatible denominators.
When generated SQL is part of the workflow, test its permission boundaries and wrong-answer behavior independently of the dashboard aggregates. The test pack for agent-generated SQL addresses that separate validation task. Here, the relevant connection is to record whether the task was accepted, corrected, or rejected under a defined review process, and to avoid presenting execution success as answer quality.
11. Use this decision artifact before publishing
The following worksheet is designed to be copied into the dashboard’s implementation ticket or metric specification. Fill it in with the people who own the workflow and its reporting. A blank answer is a useful signal that the dashboard is not ready to support the corresponding decision.
| Decision artifact | Record before launch |
|---|---|
| Primary operator decision | What action should a reader take when the measure changes? |
| Task boundary | What business request creates one task_id, and when is that task terminal? |
| Completion definition | Which event means the workflow ended, and which means work was accepted? |
| Quality rule | Who or what assigns each review outcome, under which rubric version? |
| Review denominator | Which tasks are eligible, and how are missing reviews represented? |
| Cost basis | Actual charge, estimate, or mixed; currency; missing-value treatment; deduplication rule |
| Latency definition | Start and finish events, timezone, treatment of open tasks and retries |
| Reporting window | Timezone, inclusive/exclusive boundary, and whether dates use event or ingestion time |
| Access test | Roles tested, fields visible, restricted path checked, expected and observed results |
| Acceptance fixture | Synthetic event cases, expected counts, duplicate handling, and missing-event behavior |
| Change control | Owner, definition version, change date, reason, and affected charts |
Use the worksheet to make a launch decision, not to create paperwork for its own sake. A basic dashboard can publish with a narrow set of measures if its definitions are clear and its known gaps are visible. For example, a team may initially report task disposition and review coverage while withholding cost per task until currency and cost completeness are reliable. That is more useful than a polished cost tile whose denominator cannot be defended.
Prioritize implementation in this order: stable task identity and event vocabulary; task-level disposition and stale-work visibility; review outcomes with explicit coverage; cost and latency with disclosed denominators; then role-based access validation and trend interpretation. Add dimensions only when they support a concrete operational decision. The result should let a reader move from a change in a chart to the specific tasks and events that explain it, without confusing an agent’s claim of success with a verified business result.
If your team needs help shaping the reporting layer or turning these definitions into an operator-facing experience, explore embedded analytics. The useful next step is to bring a representative workflow, its event vocabulary, and the decisions the dashboard should support—not to begin with a list of charts.