What a CTO Should Ask Before Approving the Next Frontier Model

A new frontier model deserves a decision, not an automatic upgrade. The CTO’s job is to determine whether a candidate improves a defined business workflow without creating unacceptable quality, security, availability, cost or operational risk. A release announcement, vendor benchmark or compelling demonstration can help select what to evaluate; none is a production approval by itself.

Use this checklist to decide whether to reject a candidate, gather more evidence, run a limited production trial or approve a controlled migration. It applies whether the candidate replaces one model or enters a system that can route work among several. It does not prescribe a universal score threshold: acceptable limits depend on the task, user impact and cost of failure.

The example and worksheet below are designed to be adapted. Treat the sample values as illustrative, not as measured results or recommended thresholds. Record the exact provider-issued model IDs and the date of each evaluation. A model name alone may not identify what was tested clearly enough for a later review.

How to use this checklist

Assign an accountable lead to assemble the evaluation, but require decisions from the people who own the affected workflow, production service and risk controls. A CTO can sponsor the review without personally judging every sample. The approver needs enough evidence to understand what changed, what remains uncertain, who can stop a rollout and what happens if the candidate fails.

Mark an item complete only when its evidence is findable and tied to the specific candidate and workload. “Tested” is not evidence unless the record identifies the test set, evaluation method, result and decision. An unresolved item should be marked as a gap with an owner and a disposition: block, defer, constrain the rollout or explicitly accept a bounded risk.

This release review is narrower than ongoing workflow ownership or a recurring portfolio review. For responsibilities after launch, see Who Owns an AI Agent After Launch?. The quarterly decision to expand, fix or retire a workflow belongs in The Quarterly Agent Review: Expand, Fix or Retire Each Workflow.

flowchart TD
    accTitle: "Frontier model release decision"
    accDescr: "Define the workload and limits, evaluate the candidate, hold it if critical gates fail, otherwise canary it behind reversible routing and promote only if production signals remain within limits."
    A["Define workload and limits"] --> B["Run paired evaluation"]
    B --> C{"Critical gates pass?"}
    C -->|"No"| D["Hold candidate or restore incumbent"]
    C -->|"Yes"| E["Canary behind reversible routing"]
    E --> F{"Production signals within limits?"}
    F -->|"No"| D
    F -->|"Yes"| G["Approve limited promotion"]

The diagram is a gate sequence, not a claim that a successful test proves a model safe or suitable for every task. “Critical gates” should be defined before the evaluation begins. A failed gate means the candidate remains out of the affected production path; it does not require abandoning it for every possible workload. A production canary is appropriate only when the service can identify candidate traffic, observe relevant outcomes and restore the incumbent without waiting for a new model evaluation.

1. Define the decision before evaluating a candidate

• Name the specific production decision.

  State whether the proposal is to replace the default model, use the candidate for one task class, add a fallback, or route selected requests to it. These are materially different changes. “Adopt the new model” is too broad to approve: it hides which users, prompts, tools and business actions could be affected. Tie the request to a service, workflow and proposed traffic boundary.

• Identify the workflow outcome that must improve or remain acceptable.

  Describe success in business terms that can be checked: for example, a support response meets a policy rubric, a document field is correct, or a generated code change passes the relevant tests. Pair that outcome with operational measures such as completion time and failure rate. Do not substitute fluency, preference ratings or a vendor’s general capability description for the result the workflow actually needs.

• Write down the incumbent and candidate identifiers.

  Record the exact provider-issued model IDs, provider, evaluation date, relevant configuration, and the identifier of the currently deployed version. Capture changes to prompts, tools, context assembly, retry policy and output validation too. If several of those change at once, a better or worse result cannot be confidently attributed to the model alone.

• Set decision thresholds and blocking conditions in advance.

  Define which measures may trade off and which are hard stops. A modest latency improvement should not compensate for a material increase in a critical error. Set thresholds using the workflow’s existing service objectives and risk tolerance, not a convenient number selected after seeing results. Record who can accept a non-blocking exception and what evidence supports that exception.

A useful starting point is to separate release gates into three classes: required outcomes, operating limits and open questions. A required outcome might be passing a set of high-consequence cases. An operating limit might be a maximum acceptable tail latency. An open question might be how the candidate behaves on a rare input that the team cannot yet evaluate reliably. Keeping these distinct makes “promising overall” less likely to obscure a failure that matters.

2. Build an evaluation set that represents the work

• Use representative inputs from the real task mix.

  Sample the workload across common cases, difficult cases, different input lengths and relevant customer or document types. Remove or transform sensitive data according to your organization’s controls, and preserve the features that affect the task. A set consisting only of clean demonstrations is likely to reward a candidate that performs well on ideal inputs while missing the conditions production actually presents.

• Include costly mistakes and ambiguous cases.

  Label examples where a wrong answer could trigger an incorrect business action, expose information, mislead a user or cause expensive rework. Include cases where the correct response is to request clarification, abstain or hand off rather than guess. Score these separately from ordinary cases; averaging them into a large set can make a serious regression appear small.

• Keep a held-back set for confirmation.

  Use one portion of the evaluation set to find prompt adjustments and investigate failures, and reserve another portion for the final comparison. If the team repeatedly tunes against every example, the final score describes familiarity with the test set more than performance on unseen work. Keep versions of both sets so future reviews can distinguish a genuine model change from a changed test.

• Test the complete workflow, not only a prompt in isolation.

  Include relevant retrieval, formatting, output parsing, validation, tool use and downstream handling. A candidate can produce an attractive answer that fails when a parser expects a specific field, or return valid structured content that leads the surrounding system to take the wrong action. Record both model-level observations and end-to-end task outcomes.

• Choose enough cases to make the decision credible.

  Size the evaluation according to task diversity, error severity and the decision being made. A small set may expose obvious regressions but cannot establish reliable performance on rare events. Where labels are subjective, define a rubric, use more than one reviewer on a subset and resolve disagreements before comparing candidates. Report sample size and uncertainty rather than presenting a single percentage as definitive.

For a practical approach to selecting release measures, connect this work to Benchmarks That Actually Matter for New Model Releases. The key decision is not which metric looks most impressive; it is whether the evaluation can detect the failure modes that would change the approval.

3. Compare quality and capability on the actual task

• Score outcomes against explicit acceptance criteria.

  Use a task-appropriate rubric or objective checks. For classification, compare decisions with reviewed labels. For generated code, use the relevant build and test process. For summaries, assess whether required facts are preserved and unsupported claims are absent. Keep raw outputs for review, but do not let a handful of memorable examples override a systematic comparison.

• Review regressions by category, not just by average.

  Break results down by task type, input characteristics, failure severity and user group where those distinctions matter. A candidate can raise the overall pass rate while worsening a small but high-impact category. Review representative failures side by side and record whether the cause appears related to the model, prompt, context, tool interaction or an evaluation defect.

• Measure repeatability where output variation affects the decision.

  Re-run a suitable subset when repeated outputs could alter a classification, route or downstream action. A single successful sample does not establish that the workflow will behave consistently. Decide how to compare variable answers before rerunning: use exact-match only where exactness is required, and use a defined rubric where multiple answers can be acceptable.

• Separate model capability from execution controls.

  A model’s apparent ability does not establish what a particular product configuration can execute. For example, GPT-6 Astra is an OpenAI model; its capability is separate from Codex permissions and execution controls (OpenAI model information). In your own system, test the model together with the permissions, tools, validators and routing that actually govern the proposed workflow. Do not infer execution authority from a model label or a successful text response.

For agent workflows, judge outcomes as well as transcripts. An agent evaluation harness and the harness that runs the agent are distinct concerns, so a convincing interaction trace is not itself proof that the task completed correctly (Anthropic’s discussion of agent evaluations). In practice, define the business state that counts as completion, then verify it through an independent check when the workflow permits one.

4. Examine safety, data handling and execution boundaries

• Map the data the candidate will receive and return.

  List the actual input categories, retrieved material, outputs and any data sent to external tools. Identify whether the proposal changes what data is included, how long it remains in system logs, or which operational teams can inspect it. Ask the relevant privacy and security owners to assess the concrete data flow against existing organizational requirements; a model upgrade is not a substitute for that review.

• Test the workflow’s known misuse and instruction-conflict cases.

  Include attempts to elicit prohibited content, manipulate instructions through supplied documents, produce unsupported claims or misuse connected tools, as applicable to the workflow. Define the expected system response and inspect the full interaction, including what the application does after receiving the model output. Do not assume that a model’s general safety reputation proves that your prompts and controls work.

• Verify that consequential actions have an independent control.

  For an operation that changes an account, sends a message, moves money or otherwise creates an external effect, document how the application limits and validates the action. Where human approval is required, bind that approval to the exact proposed action and payload, and make it expire so an old approval cannot authorize a materially changed request. A model-generated explanation is not a control over execution.

• Check output validation and failure handling.

  Confirm what happens when the model returns incomplete, malformed, contradictory or unexpected content. The receiving system should fail in a defined way rather than quietly treating an unvalidated response as a valid instruction. Test timeouts, retries and duplicate submissions against the actual workflow; do not promise exactly-once effects for external actions unless the end-to-end system can establish them.

• Record unresolved risk instead of hiding it in a score.

  For each material gap, state the affected input or action, plausible consequence, available control and decision owner. A limitation may be acceptable for a restricted trial while remaining unacceptable for broad use. If the issue cannot be measured with confidence, say so directly and narrow the proposed deployment rather than turning absence of evidence into a pass.

This section is about the model’s place in a system, not a generic certification exercise. The approval record should make clear which safeguards are application controls, which depend on people and which remain assumptions. If the candidate’s intended use changes, repeat the relevant assessment; an approval for a bounded drafting task does not automatically cover a workflow that takes actions.

5. Confirm availability and operational fit

• Identify dependencies that could make the workflow unavailable.

  Map the model call alongside the application, network path, retrieval components and any tools needed for completion. Determine how users experience a provider error, timeout or capacity constraint. The relevant question is not whether a model is generally available, but whether this particular service can meet its operating needs when one dependency is degraded.

• Exercise timeout, retry and fallback behavior.

  Test what happens when a request is slow, fails, returns an unusable answer or cannot be retried safely. Retries can increase latency and cost, and may duplicate an external effect if the workflow is not designed to prevent it. A fallback should be checked against the same task requirements; switching to another model is not automatically a safe or equivalent response.

• Measure the whole request path under realistic conditions.

  Include time spent assembling context, calling the model, validating its response and completing any downstream step. Track distributions, not only averages, so occasional slow requests are visible. If a task has a user-facing deadline or a service objective, evaluate against that limit and decide what the system does when it cannot finish in time.

• Confirm the team can detect and respond to a degraded release.

  Name the signal that will reveal a problem, the person or rotation that receives it, and the action available to restore the previous behavior. Ensure the operational view distinguishes candidate traffic from incumbent traffic. A dashboard that combines both can conceal a release-specific regression until users report it.

A release decision should be specific about the scope that can operate reliably, not just the capability the team wants to use. If the service depends on routing between providers or models, assess that routing as part of the change. The design considerations in Multi-Model Production Stacks in 2026: Routing Between Claude Opus 4.7 and GPT-5.5 are relevant when the proposal includes selective routing rather than a single-model replacement.

6. Check cost, latency and capacity with explicit assumptions

• Calculate cost per completed task, not only price per token.

  Use the expected mix of inputs, outputs, retries, tool calls and failed tasks. A model that generates longer responses or causes more retries may cost more in practice even if one component of its use appears cheaper. State the assumed volume and workload mix, and keep the calculation separate from any unverified commercial terms.

• Compare cost against the outcome the business receives.

  Estimate cost per accepted result or completed workflow, not just per request. If a candidate improves task success but needs more review, include that review effort in the decision. If it is cheaper but produces more rework, show where the apparent saving goes. Label estimates as estimates and identify which inputs should be replaced with observed data during a trial.

• Measure latency at the points users and systems experience it.

  Track time to a usable response and end-to-end completion where both matter. A quick first response may not help if the total task takes longer, and a slower response may be acceptable in a background workflow but not in an interactive one. Set separate limits for the distinct paths instead of allowing one average to stand in for every use.

• Check capacity against a plausible peak, not only a quiet test window.

  Estimate the concurrency and burst pattern the service needs to handle, then establish how the application behaves if it cannot sustain that demand. Avoid relying on a rate, quota or availability assumption that has not been confirmed for the specific account and deployment. If peak behavior is unknown, record the uncertainty and constrain the rollout until it is resolved.

A useful comparison keeps quality, cost and response time side by side while preserving task-level detail. Do not collapse them into a composite score unless the weights have been agreed and reviewers can still inspect the underlying measures. For a high-stakes workflow, a weighted average can make a severe correctness regression look like a reasonable trade.

7. Make the evaluation reproducible and interpretable

• Freeze the comparison conditions.

  Run the incumbent and candidate against the same eligible examples with equivalent context, application logic and output handling. Record configuration changes that are necessary for the candidate, but do not silently give one model a different task. If prompts must differ, preserve both versions and explain why the comparison remains useful.

• Use separate execution and evaluation responsibilities where appropriate.

  The component that runs a workflow and the process that judges its outcome answer different questions. Keep the evaluation method explicit: identify which checks are automatic, which require a reviewer and which rely on a proxy. For an agent, judge whether the intended result occurred rather than treating a plausible transcript as sufficient evidence.

• Store the evidence needed to repeat or challenge the decision.

  Keep the test-set version, exact model IDs, date, relevant configuration, scoring rubric, aggregate measures and selected failure examples in a location the team can retrieve. Protect sensitive data and retain only what organizational policy permits. A result without its conditions may be impossible to reproduce after the model or application changes.

• Distinguish measured results from judgment and assumption.

  Label direct observations, reviewer judgments, forecasts and unresolved assumptions separately. For example, a measured latency from a test run is not a guarantee about future load; a reviewer’s quality score is not an objective ground-truth label. This distinction makes the approval useful to executives without overstating what the evaluation established.

• Ask whether the test can detect a failure that matters.

  Before accepting a pass, have a reviewer identify a plausible regression the evaluation could miss. If the answer is “we did not test that input type” or “we cannot verify the downstream result,” the decision should reflect the gap. Add a focused test, narrow the scope, add a runtime control or defer approval; do not disguise a blind spot as a clean result.

For evidence to be useful beyond the release meeting, it should show the question asked, the comparison made, the limitations and the decision that followed. What Good AI Delivery Evidence Looks Like in a Case Study covers evidence quality in a distinct context; for this review, the practical standard is whether another responsible person can understand and challenge the release record.

8. Plan a reversible rollout before granting approval

• Confirm that candidate traffic can be identified and limited.

  Establish how the application distinguishes candidate requests from incumbent requests and how the percentage or workload scope is controlled. Test that control before using it for a live release. If the deployment cannot separate traffic or restore the incumbent promptly, treat that as a missing operational prerequisite, not as a detail to solve after approval.

• Start with a scope that limits plausible harm.

  Choose the first live population by task, user group, traffic share or another meaningful boundary. The safest starting point is not necessarily an arbitrary percentage: one sensitive workflow at low volume may carry more risk than a larger share of routine, reversible requests. State why the initial scope is informative while limiting exposure.

• Define rollout gates, observation windows and stop authority.

  Set the signals that must remain within bounds, the length or volume of observation needed to assess them, and who can pause or reverse the change. Include quality signals where they are available, not only infrastructure metrics. Make escalation practical: the person on call should not need a new executive decision to stop a release that crosses a pre-agreed critical threshold.

• Verify restoration of the incumbent and downstream state.

  Confirm how routing returns to the previous version, whether in-flight work is affected, and how any partial or completed external actions are reconciled. A model rollback does not reverse a message already sent or a record already changed. Document the operational steps for those cases and test the control path before relying on it.

The decision branches are deliberately asymmetric: a failed critical evaluation gate means hold; a successful offline evaluation permits a bounded trial, not unrestricted traffic. Only stable production signals support a larger release. The candidate should remain reversible until the team has enough evidence for the next scope, and a successful canary should not erase unresolved issues outside the canary’s workload.

9. Work through a hypothetical release decision

Consider a hypothetical customer-support workflow that drafts answers for a human agent. The incumbent model is incumbent-model-id-example; the proposed candidate is candidate-model-id-example. These are placeholders, not real provider identifiers. Assume the evaluation date is 2026-09-24, and assume the organization has a held-back set containing routine questions, ambiguous requests and cases where an incorrect answer could create a customer commitment. The workflow requires a human agent to review drafts before sending.

• Check whether the candidate improves the defined support outcome.

  Suppose, for illustration only, that the candidate receives better rubric scores on routine cases but performs worse on ambiguous requests. The team should not approve the change based on an overall average alone. It should inspect whether the weaker cases share a pattern, confirm that the held-back set was not used to tune prompts and decide whether the candidate can be limited to the routine category without weakening existing review controls.

• Test whether the operating change creates an acceptable trade.

  Suppose the illustrative run also shows slightly faster draft completion but more output tokens and additional review on a subset of cases. The decision depends on observed workload mix and staff impact, not a claim that “faster” or “better” is inherently worthwhile. Calculate the estimated cost per accepted draft using the assumed mix, then mark the calculation as provisional until the proposed trial measures real usage.

• Check the failure path, not only the ordinary draft.

  Suppose a candidate response omits a qualifying condition that the support rubric treats as essential. The team must decide whether a reliable validator can detect that omission, whether a reviewer can reasonably catch it before sending and whether the candidate should be excluded from that category. If none of those controls is credible, the failure is a blocker for that task, even if the candidate performs well elsewhere.

• Set a bounded first release with an observable stopping rule.

  A reasonable hypothetical proposal might route only eligible, low-risk draft requests to the candidate, keep an agent’s review before sending, compare candidate and incumbent signals, and stop if a pre-defined critical error or operational limit is crossed. The actual scope and thresholds must come from the organization’s workload and service requirements. This example does not establish that the candidate passes; it shows how a partial pass can become a constrained decision rather than a blanket approval.

This example also illustrates a counterexample to “the new model scored higher, therefore replace the old one.” If the score increase comes from many easy cases while a rare but consequential category regresses, a full replacement may be the wrong decision. The appropriate response could be to hold, gather targeted evidence, route only the demonstrated safe category or retain the incumbent. A model release decision can be selective without pretending that every task needs the same answer.

10. Complete the approval record and printable summary

The decision artifact should fit into the release record, not depend on someone remembering the meeting. Fill in the table for every candidate and update it when the evaluated configuration changes. The example identifiers and date are intentionally placeholders for illustration; replace them with the exact provider-issued IDs and actual test date before making a real decision.

• Record a decision that is narrower than the evidence when necessary.

  Choose approve, approve with scope limits, defer for evidence or reject. State the approved task classes, prohibited uses and conditions that would invalidate the decision. A conditional approval should identify what must be completed before the condition is lifted, who owns it and how completion will be verified.

• Identify the accountable operational and business owners.

  Name the people responsible for the workflow and its technical operation during the rollout. This is not a substitute for a broader ownership framework; it is the minimum needed to make this particular release actionable. The release record should show who receives the signal, who can pause traffic and who decides whether the proposed business outcome is still worth pursuing.

• Set a re-evaluation trigger.

  Reopen the decision if the model ID, prompt, tool permissions, data flow, task mix or downstream action changes materially, or if production signals cross a stated boundary. A calendar date can prompt a review, but a material system change should not wait for the calendar. Keep this release artifact distinct from the separate recurring portfolio decision to expand, repair or retire workflows.

• Make the summary printable without losing its meaning.

  For a meeting or paper review, print the decision table together with the gate results, the most important failure examples and the proposed rollout boundary. The table alone is not a scorecard: reviewers need enough accompanying context to understand why a result passes, what it does not establish and what action follows if a stop condition occurs. Store the full evaluation record with the summary.

A complete approval is not a claim that the model is universally better. It is a bounded decision that identifies the candidate, workload, evidence, limits and action owners. If the team cannot state what would cause it to stop, what the old path is and which measured result supports the change, the proposal is not ready for production approval.

For organizations that need help choosing an evaluation scope, comparing candidate approaches or turning a release review into an operating decision, AI strategy and model selection is an appropriate next step.

WANT TO TALK THROUGH YOUR SITUATION?

Talk it through with us. Thirty minutes, no slides.