The Quarterly Agent Review: Expand, Fix or Retire Each Workflow

A quarterly agent review should end with a decision about a specific workflow: expand it, keep it at its current scope, fix it, pause it, or retire it. The checklist below helps CTOs and operating leaders make that decision from operational evidence rather than from a compelling demo, a rising usage chart, or an isolated success story. It is designed for a review meeting that uses an existing operations dashboard as an input; it does not explain how to build that dashboard.

Bring the workflow owner, a representative operator, the business sponsor and someone able to explain the measured data. Review a defined period, compare it with the previous period where possible, and record unresolved evidence gaps. A quarter is a useful management rhythm, not a reason to wait: a material failure or changed exposure may require an earlier review. For background on the operational signals that can feed this meeting, see AI Agents in Production: Agent Observability. This scorecard turns those signals into a decision; it is not a substitute for investigating individual incidents.

1. Set the decision and review window

• Name one workflow and one decision owner. Write the workflow in operational terms: for example, “classify incoming invoice exceptions and prepare a review packet,” not “use AI in finance.” Name the person accountable for recommending a decision and the person who can authorize any change in scope. A review that covers several unrelated workflows tends to blur their different users, consequences and failure patterns.

• Record the period under review and the comparison period. State start and end dates, the number of eligible cases, and whether the comparison uses the prior quarter, a stable baseline, or another declared period. If a system change, seasonal peak or unusual incident makes a direct comparison misleading, note that before interpreting a trend. A percentage without its period and population is not decision-ready evidence.

• State what the workflow is allowed to do today. Describe its current users, inputs, outputs, action boundary and human checkpoints. Separate “suggests a response” from “sends a response,” and “prepares a transaction” from “executes a transaction.” The review cannot responsibly recommend expansion unless the present operating boundary is explicit enough to compare with the proposed one.

• Write down the decision options before discussing performance. Use expand, maintain, fix, pause and retire. “Maintain” means the workflow remains within its present scope while evidence continues to be reviewed; it does not mean that every unresolved issue is acceptable. “Pause” means stop or constrain the affected operation while a specified condition is investigated. Define what those words mean for this workflow so participants do not mistake a recommendation for an implemented change.

• Identify who will act on the decision and by when. Record the owner for each follow-up, a due date, and the event that will bring an unresolved issue back to the decision forum. If a decision requires a configuration, process or staffing change, name the person responsible for confirming it is in place. A meeting note that says “monitor closely” without an owner or trigger does not create a workable next step.

Begin with the business purpose, not a claim about model quality. State the user problem the workflow is intended to improve and the observable result that would count as useful. “Reduce time spent assembling exception context” is testable if elapsed handling time is measured consistently. “Make finance more efficient” is too broad to connect a dashboard signal to a decision.

2. Confirm that the evidence is usable

• Check whether the dashboard covers the workflow’s actual operating path. Compare its scope with how work really arrives, moves through the agent, reaches a person and ends. Look for manual workarounds, cases completed outside the measured path, retries that appear as new requests, and outcomes recorded in another system. If the dashboard sees only the agent’s responses, it may miss failures occurring before a request is submitted or after an output is handed off.

• Confirm that counts have clear denominators. For each rate, write down what is counted in the numerator and which cases belong in the denominator. For example, “operator corrections” might mean edited outputs divided by outputs reviewed, not edits divided by all incoming cases. Exclusions such as cancelled, duplicate or still-open cases should be stated. If two teams define a “completed case” differently, do not combine their figures until the definitions are reconciled.

• Check missingness and data freshness. Record whether event capture was interrupted, whether key fields were blank, and how long it takes for outcomes to appear. A dashboard may look stable because failed cases never produce the event used to calculate the displayed rate. When records arrive late, distinguish the latest complete period from the current partial period; do not treat an incomplete denominator as a final result.

• Separate system events from business outcomes. A successful run, returned response or completed tool call says something about processing, not necessarily whether the user’s task was completed correctly. Identify which outcome is confirmed by an operator or downstream record, and which is inferred from a technical event. Keep those measures visibly distinct in the meeting materials.

• Mark uncertainty instead of filling gaps with estimates. If an outcome cannot be observed, label it unknown and explain the limitation. Do not silently count missing results as successes or failures. If the review depends on an estimate, write down its source, assumptions and likely direction of error, then decide whether the estimate is strong enough for the proposed action.

• Compare like with like. Check whether the same case types, user groups, operating hours and review practices are represented in both periods. A shift toward simpler cases can raise apparent success even if the workflow has not improved. A newly introduced review step can increase recorded corrections while reducing the number of errors reaching users. Explain material changes in the population before attributing a difference to the agent.

A service objective should describe behavior users care about, and its measurement needs an explicit denominator and time window. The Google SRE workbook’s guidance on implementing SLOs is useful when setting that measurement boundary. For this review, the practical test is simple: can a reader reconstruct what the measure means and what work it includes without guessing?

3. Check user and business outcomes

• Choose one primary user-facing outcome. Identify the result the workflow exists to produce, such as a correctly routed case, a complete draft ready for human review, or a verified record update. State how the outcome is observed and who confirms it. Keep this measure separate from model confidence, response length or the number of tasks attempted; those may help explain behavior but do not establish that the user received value.

• Review quality at the point where mistakes matter. Examine a representative set of completed cases and the cases with the highest consequence, using the organization’s existing review process. Record the type and impact of errors, not only whether an output was marked right or wrong. If the reviewer sample is small or selected because cases looked unusual, say so. A carefully explained small sample is more useful than presenting it as a population-wide accuracy rate.

• Look for user effort displaced rather than removed. Compare the work the workflow was meant to reduce with new work it creates: checking outputs, correcting fields, reopening cases, explaining exceptions, or moving data between systems. Ask operators to identify work that is invisible in event logs. A faster first response can still leave total handling time unchanged if review and repair take longer.

• Check whether the benefit reaches the intended users. Compare experience across relevant teams, shifts, case types or levels of expertise when the data supports it. An average can conceal a group that receives poorer results or bears more correction work. Do not create finely segmented claims from tiny samples; flag the possible disparity and gather sufficient evidence before widening use for that group.

• Verify completion independently of the agent’s own account of completion. Use the business record, an operator confirmation or another appropriate outcome signal. An agent stating that it finished a task is not the same as evidence that the intended change occurred and was accepted. For consequential actions, trace the result to the system or person responsible for the actual business state.

• Check for harmful or misleading success signals. A workflow may appear successful because it returns a plausible answer, closes a ticket early or routes a case out of its own queue. Review reopened work, user complaints, corrections and downstream exceptions alongside headline completion. If these signals conflict, pause the expansion discussion until the team can explain the discrepancy.

Translate outcome evidence into a decision-relevant statement. For example: “During the review period, the workflow prepared packets for these case types; the team confirmed completion for this share, and the remaining cases are unresolved or lack outcome capture.” That is more useful than “the agent performed well,” because it names the observed population and leaves uncertainty visible.

4. Review reliability, recovery and operational load

• Inspect failure patterns, not just an aggregate failure rate. Group failures by their operational cause where records permit: unavailable dependency, invalid input, timeout, incorrect classification, permission denial, or human rejection. Confirm whether the categories overlap and whether they explain all known failures. A single percentage cannot show whether the issue is a recoverable interruption or a repeated defect that produces bad work.

• Trace what happens after a failure. For a sample of failed or incomplete cases, establish whether the request was retried, routed to a person, safely abandoned, or left in an ambiguous state. Record who notices the failure and how long it takes to restore a usable path. If a retry can repeat an external effect, verify the recovery procedure rather than assuming that another attempt is harmless.

• Measure time to detect and recover where it matters. Use timestamps that describe the user-visible interruption and the return to an acceptable service state. Distinguish time until an alert, time until a human response and time until the underlying case is completed. A system can recover technically while a user’s work remains stuck in a queue.

• Check whether manual fallback can handle the observed volume. Ask the operator or service owner what happens when the workflow is paused for a shift or a day. Consider pending cases, queue age, required expertise and the people available to take over. A fallback documented on paper may not be a practical recovery path if the team cannot access the necessary context or capacity.

• Review the operational burden created by routine exceptions. Record who handles escalations, how often they occur, and whether that work is becoming concentrated in one person or team. A workflow that depends on an informal expert to repair edge cases may appear reliable until that person is unavailable. The question is not whether every exception can be eliminated; it is whether the exception workload is visible and manageable.

• Compare operational signals with user reports and incidents. Check whether support tickets, complaints, post-incident notes or operator feedback show problems absent from the dashboard. If incident records use different categories or time periods, keep them separate and explain the mismatch rather than merging them into a misleading total. An unexplained conflict between sources is itself a finding for the decision record.

Use this recovery trace as a working artifact during the review. The case below is hypothetical; its times and events illustrate a method and are not a report of a real deployment.

Trace field	Illustrative record
Case	Invoice exception packet; input received at 09:12
Failure observed	Required supplier reference absent; workflow returned an incomplete packet at 09:13
Detection	Operator noticed the missing field at 09:21 during review
Recovery	Operator checked the source record, completed the packet manually and recorded the exception at 09:34
Business outcome	Packet entered the review queue; no automated posting occurred
Follow-up	Add missing-reference cases to the exception sample; owner: workflow operations lead

• Pass condition: the trace explains the full path to a verified outcome. The record identifies the input, failure, detection, recovery owner and resulting business state. A reviewer can tell whether the case was completed, remains open, or was abandoned. If any of these facts are unknown, mark the trace incomplete and do not use it as proof that recovery is reliable.

• Fail condition: an ambiguous or harmful state is possible without detection. Treat the trace as a failure if the same request could produce a repeated external effect, if the case disappears from both the agent and human queues, or if no one can determine whether the intended change occurred. The response is to contain the specific risk, investigate the path and require a verified recovery test before increasing scope—not to assume that a higher aggregate success rate offsets it.

The trace makes a useful distinction: returning an error is not by itself a safe recovery, and a successful technical retry is not by itself a verified business outcome. A review should be able to explain what the operator sees, what action is safe next, and how the final state is confirmed. If the operational dashboard does not capture these facts, record that as a measurement gap and assign an owner to close it.

5. Examine cost and capacity in relation to value

• Define the unit of business value before comparing cost. Choose a unit the business recognizes, such as a completed case of a defined type, and describe what makes it count. Relate costs to units of business value, state assumptions and account for indirect costs as well as direct ones. (FinOps Foundation: unit economics.) Avoid comparing spend per request with value per completed case unless the conversion between them is explicit.

• List the cost categories included in the calculation. Depending on the workflow, relevant categories may include model or platform usage, supporting services, engineering and operations time, human review, exception handling and rework. Use actual internal measures when available; otherwise show an assumption rather than inventing a precise number. State whether the comparison includes setup work, ongoing maintenance, or only the reviewed period.

• Show the arithmetic and the assumptions. If the team uses a calculation such as total attributable cost divided by verified completed units, display the numerator, denominator, period and exclusions. A hypothetical illustration could assume 1,000 eligible cases, 760 verified completions and $4,000 in total period cost: the illustrative cost is $4,000 ÷ 760, or about $5.26 per verified completion. These invented figures demonstrate the calculation only; they say nothing about expected cost or results for another workflow.

• Compare with the relevant alternative. Establish what the same work costs or requires without the workflow, including manual completion, existing automation or a different process. Keep the alternative’s scope and output quality comparable. If there is no trustworthy baseline, state that the review can describe current operating cost but cannot yet establish relative savings.

• Check whether the unit economics deteriorate at the margin. Ask whether more volume would bring a larger share of difficult cases, added review capacity, additional exception work or new operational support. A current average may not predict the cost of the proposed expansion if the next group of users or cases is materially different. Request a bounded estimate for that next increment, with assumptions, rather than projecting the current average unchanged.

• Separate cost evidence from a claim of business value. A lower cost per case does not establish that the cases are correct, timely or useful to users. A higher cost can still be acceptable if it enables an important outcome, but the business sponsor should state that outcome and the acceptable tradeoff. Keep the choice explicit rather than letting a single financial metric decide on behalf of the business.

The cost review should answer a practical allocation question: what does the organization spend to produce a verified unit of useful work, and what changes if the workflow grows? Avoid false precision when labor time, shared infrastructure or incomplete outcomes are not measured. A transparent range with named assumptions is more decision-useful than a neat figure that hides material categories.

6. Decide whether proposed expansion is bounded

• Describe the proposed change as a measurable increment. Specify the new case types, users, volume, hours, or action boundary under consideration. “Roll it out more broadly” is not a testable scope. A bounded proposal lets the reviewer ask what new evidence will be collected and what condition would stop the change.

• Compare the proposed users and cases with the evidence already reviewed. Note differences in language, data completeness, process rules, expertise, operating hours and consequence of error. If the next group is unlike the current group, existing results may not transfer. Name the evidence needed for that group rather than assuming it will behave like the familiar cases.

• Set an exposure limit and a stop condition. State what volume, time period or type of action is permitted during the next step, and identify who can halt it. Define observable stop conditions such as a specified class of harmful error, missing outcome records above a set threshold, or inability to route exceptions to a person. The threshold should be chosen by the accountable business and operational owners for this workflow, not presented as a universal number.

• Require approval before any consequential execution. Where a workflow can trigger an external or hard-to-reverse action, require a person to approve the exact action payload before execution. Bind approval to the specific payload and make it expire; a general prior approval should not silently authorize a changed amount, recipient, record or action. Record the approval and resulting business state so the review can reconstruct what was authorized.

• Check that the rollback or containment path is real. Identify the person able to stop the change, the mechanism for returning to the previous operating boundary, and how in-flight work will be handled. If already completed actions cannot be reversed, say so and define the compensating process. Do not describe a launch as reversible merely because a configuration can be changed back.

• Specify evidence collection for the next decision. For the proposed increment, name the outcome measures, denominator, review window, exception sample and owner. State when the expansion will be reconsidered and what result would support, block or narrow it. This turns a limited expansion into a learning decision rather than an open-ended rollout.

A small staged change can reduce uncertainty only if exposure is genuinely bounded and the stop path works. Do not assume that a vendor-hosted endpoint can split traffic for a canary: the workflow’s routing design must support the proposed division externally where needed. Confirm that routing and attribution before treating a staged comparison as available. Also avoid claiming that a retry or distributed workflow guarantees exactly-once external effects; design and verify safeguards for duplicate attempts and ambiguous outcomes.

7. Choose the action that fits the evidence

• Expand only when outcomes, operations and scope align. Recommend expansion when the intended user outcome is verified for a relevant population, failures have understood and workable recovery paths, costs are visible enough for the proposed increment, and the new scope has a stop condition. Record the evidence that supports each part. A favorable headline measure alone is not sufficient if the proposed users, actions or case types differ materially from those measured.

• Maintain when current use is acceptable but expansion is not yet justified. This is appropriate when the workflow appears useful within its present boundary, but evidence is too limited, an unresolved measurement gap remains, or the proposed next group has not been assessed. State what must be learned and when it will be reviewed again. “Maintain” should not become a permanent holding pattern with no owner or decision trigger.

• Fix when a specific, bounded issue has a plausible remedy. Name the failure, its user or business impact, the proposed change, the accountable owner and how success will be verified. Keep the workflow’s scope unchanged while the repair is evaluated if increasing exposure would make the problem harder to contain. A fix is not complete when a change is deployed; it is complete when the relevant outcome and recovery path are checked.

• Pause when continued operation could create unresolved harm or an unreliable business state. State which actions stop, what work can safely continue, who will handle the queue and what evidence is needed to resume. A pause should be proportionate to the affected path: a defect in one consequential action may justify disabling that action while preserving a separately safe drafting function, if the boundary can actually be enforced and verified.

• Retire when the workflow no longer merits continued operation. Document the reason, such as a replaced process, insufficient continuing value, unmanageable maintenance burden or a risk that cannot be acceptably bounded. Plan how users will complete the work, how pending cases will be handled, and when the old path is shut down. Confirm that retirement does not leave unresolved requests or an undocumented dependency behind.

• Record dissent and uncertainty without forcing a false consensus. If the business sponsor and operational owner interpret the same evidence differently, capture both views, the assumption each relies on and the decision authority. The meeting can still make a bounded decision, but the record should not rewrite disagreement as certainty. Identify what observation would resolve the disagreement, if it can be resolved.

The branches in this decision flow are intentionally short. Each leads to an accountable action; incomplete evidence is a reason to narrow or defer a change, not a hidden endorsement of expansion.

flowchart TD
%%{init: {"flowchart": {"htmlLabels": true}}}%%
accTitle: Quarterly workflow decision flow
accDescr: Check whether evidence is usable, then choose an action based on outcomes, operational safety, and the proposed scope.
    A["Evidence usable?"] -->|"No"| B["Hold scope; close gaps"]
    A -->|"Yes"| C["Outcomes and recovery acceptable?"]
    C -->|"No"| D["Fix, pause, or retire"]
    C -->|"Yes"| E["Next scope bounded?"]
    E -->|"Yes"| F["Expand with limits"]
    E -->|"No"| G["Maintain current scope"]

If the evidence is unusable, hold the current boundary where safe and assign work to close the gaps; do not treat missing records as a pass. If outcomes or recovery are unacceptable, select the narrowest action that contains the problem, then state what would permit resumption. If current operation is acceptable and a proposed increment has limits, an expansion can be considered with monitoring and an explicit stop condition. If it is not bounded, maintain the existing scope rather than approving an undefined rollout.

8. Test the scorecard against a counterexample

Consider a hypothetical support workflow that drafts responses for account-access requests. Its dashboard shows fewer minutes to first response and a high share of requests with a generated draft. The business sponsor proposes extending it to account closures. At first glance, the metrics look favorable. But the proposed work has a different consequence, the dashboard records draft creation rather than confirmed resolution, and agents report that some requests are reopened after the initial response.

• Ask whether the headline measure tracks the actual user result. In this hypothetical, faster drafts do not establish successful account recovery, and draft creation does not prove the request was resolved. The review should separate response preparation from verified resolution and inspect reopened cases. Until the denominator and outcome are clear, the apparent improvement cannot support the proposed expansion.

• Test whether the new scope is comparable. Account closures may involve different checks and a less reversible action than access guidance. The existing draft-only evidence does not demonstrate that a workflow is suitable for that action. Treat the difference as a new scope requiring its own review, not as an ordinary increase in volume.

• Choose a decision consistent with the gap. A defensible recommendation could be to maintain the existing draft-only boundary while improving outcome capture and reviewing reopened cases. If a harmful issue is found in current use, narrow or pause that path and define a recovery process. The favorable time-to-first-response signal remains relevant, but it cannot cancel the unresolved outcome and scope questions.

This counterexample prevents a common reasoning error: using evidence that supports one narrow claim to authorize a different claim. A measure can be real and still be insufficient for the decision under discussion. Record what it establishes, what it does not establish, and what additional evidence would make the next decision possible.

9. Produce a decision record people can operate

• Write the decision in one sentence with its scope. Example: “Maintain agent-generated drafts for the current request types; do not add account-closure actions this quarter.” Avoid vague outcomes such as “continue cautiously.” Include the effective date and whether the decision changes actual operation now or is only a recommendation awaiting implementation.

• Attach the evidence summary, not a dashboard screenshot alone. Record the review window, eligible population, outcome definition, notable exclusions, operational failures, recovery findings and cost assumptions. Point to the dashboard or underlying records used, where the organization’s normal process permits. A screenshot can preserve what participants saw, but it rarely carries definitions and limitations by itself.

• List actions with owners, due dates and completion evidence. For each action, state what will change, who is responsible, when it is due and how completion will be checked. For example, “operations lead to review reopened cases from the prior period by [date], then report the verified resolution outcome.” Replace the placeholder with an agreed date before the record is finalized.

• Set the next review trigger. Use a date, volume threshold, incident, material workflow change or evidence milestone that fits the risk. A scheduled quarterly meeting can coexist with an earlier review trigger when conditions change. Make clear who convenes the review and what evidence must be available for it to make a decision.

• Close the loop with the people doing the work. Tell affected operators what decision was made, what changes and where to route exceptions or questions. Ask the workflow owner to verify that the practical operating instructions match the decision. A formally approved boundary that is not understood by the people handling cases is not a reliable boundary.

The record should distinguish a decision from its implementation. If a recommendation awaits a change, note the current operating state and the person responsible for making it effective. If implementation cannot be verified, do not report that the workflow has already moved to the new boundary. This small distinction avoids a gap between executive intent and the work users actually encounter.

For broader operational ownership beyond this quarterly decision, see The Post-Launch Ownership Framework: Who Runs It After We Leave. For a distinct treatment of how delivery claims should be supported, What Good AI Delivery Evidence Looks Like in a Case Study addresses that subject. A model change is another decision, not a shortcut around this workflow review; What a CTO Should Ask Before Approving the Next Frontier Model covers that separate approval question.

10. Printable quarterly review summary

Use this section as the meeting’s compact record. Complete it for one workflow, attach the operational evidence, and leave unresolved fields visibly marked rather than filling them with assumptions.

• Workflow and owner: Name the user task, workflow owner, business sponsor, operator representative and decision authority.

• Review window and population: Enter the dates, eligible case count, comparison period, exclusions and any material change in case mix.

• Current boundary: Describe users, inputs, outputs, actions, human checkpoints and fallback path. Note any difference between written instructions and observed practice.

• Primary outcome: Define the user-facing result, its numerator and denominator, the confirmation source and the time window. Mark unverified or missing outcomes as unknown.

• Operational evidence: Summarize failure patterns, detection and recovery, manual exception load, incidents and conflicting signals. Include at least one trace for a consequential or representative failure where available.

• Cost and alternative: State the unit of useful work, included cost categories, period, arithmetic, assumptions and comparable alternative. Mark estimates and missing indirect costs clearly.

• Proposed scope change: Specify exactly what would change, what evidence supports transfer to that scope, the exposure limit, stop condition, rollback or containment path, and required approval before consequential execution.

• Decision and rationale: Select expand, maintain, fix, pause or retire. Name the evidence supporting the choice, the uncertainty that remains and any dissent that affects interpretation.

• Actions and next trigger: Assign every action an owner, due date and completion check. Record the next scheduled review and any earlier event that should bring the workflow back for a decision.

The completed summary should allow a leader who missed the meeting to understand what was decided, for which users and actions, based on which evidence, and what would change the decision. If it cannot do that, return to the missing field before approving an expansion.

A quarterly review is useful when it changes what the organization does: it limits exposure where evidence is weak, funds a repair where a fix is testable, and permits expansion only when the next scope is bounded. Keep the scorecard focused on the workflow’s actual user outcome and its operating path. If the team needs help turning those requirements into an observable, maintainable production design, consider production platform engineering as a practical next step.