Table of Contents
- The Client Problem: Why This Decision Mattered
- The PADISO Framework for Production AI Selection
- Model Selection: Picking the Right Brain
- Vendor Selection: Beyond API Calls
- Guardrails: The Safety and Quality Net
- Deployment, Tuning, and Measurable Results
- Summary and Next Steps
The Client Problem: Why This Decision Mattered
A mid-market logistics operator with $180M in annual revenue came to us with a deceptively simple ask: “We need an AI feature that triages incoming freight tickets, suggests routing corrections, and drafts customer replies in real time.” The catch—this wasn’t a hackathon prototype. It had to ship to production within 12 weeks, integrate with a legacy IBM iSeries backend, pass an ISO 27001 audit as part of a PE roll-up, and demonstrate measurable EBITDA lift for the portfolio board.
At PADISO, we see this pattern every week across our fractional CTO and CTO advisory engagements in San Francisco, New York, and other major hubs. Mid-market CEOs and private equity operating partners are no longer asking whether AI can generate value—they’re demanding a clear, defensible path from model selection to production ROI. Yet most off-the-shelf advice stops at “just use GPT.” The real work—the selection process that separates a write-off from a revenue driver—is far more nuanced.
This teardown walks through exactly how we, led by founder Keyvan Kasaei, chose the model, the vendor, and the runtime guardrails for one real client. It’s not theory. It’s the same framework we apply in our AI Strategy & Readiness engagements and Venture Architecture & Transformation work for PE-backed portfolios.
The PADISO Framework for Production AI Selection
Before we evaluated a single model, we anchored the entire selection in two principles that are often skipped: start with the business outcome, and use a multi-lens evaluation that goes far beyond benchmark scores.
Starting with the Business Outcome
The client’s board didn’t care about parameter counts or training data cutoffs. They cared about three measurable outcomes:
- Reduce average ticket handling time by enough to reallocate three full-time coordinators to higher-value exception management.
- Improve routing accuracy to cut misdirected freight expenses, which were bleeding low-seven-figure sums annually.
- Achieve audit readiness for ISO 27001 with a clear chain of evidence for every AI-assisted decision—non-negotiable for the PE firm’s compliance timeline.
We translated each outcome into a set of technical acceptance criteria. For example, “improve routing accuracy” became a benchmark of 92% exact-match on a curated test set of 1,200 historical tickets with known correct routings. This gave us a non-opinionated way to compare models and vendors later.
Our Four-Lens Evaluation Model
We evaluate AI components across four interdependent lenses: performance, operational fitness, economic fit, and governance readiness. Performance covers accuracy, reasoning, and hallucination rate. Operational fitness looks at latency under load, deployment modality (API, self-hosted, edge), and integration with existing data pipelines. Economic fit isn’t just per-token cost; it’s the total cost of ownership including retraining, fine-tuning infrastructure, and ongoing monitoring. Governance readiness maps directly to the compliance and audit requirements the PE firm mandated.
This lens model is core to our CTO as a Service practice. It forces clients and engineers to confront trade-offs early, before sunk-cost attachment to a particular vendor takes hold.
Model Selection: Picking the Right Brain
We shortlisted five families: Anthropic’s Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5, Fable 5; and OpenAI’s GPT-5.6 Sol and GPT-5.6 Terra. We also evaluated open-weight models as a control, but they dropped out early due to the client’s lack of in-house ML engineering to host and fine-tune them securely.
Accuracy and Reasoning Benchmarks
We built a custom evaluation harness using the 1,200-ticket dataset mentioned earlier. The tasks required multi-step reasoning: a ticket might say “customer needs urgent redelivery to alternate dock due to weather at original destination; note that this shipment crosses a state line.” The model had to infer the best rerouting, check compliance constraints, and generate a draft email that explained the change without causing alarm.
Claude Opus 4.8 hit 94% exact-match routing accuracy and consistently produced the most context-aware emails, with zero hallucinated regulatory references. GPT-5.6 Terra came close at 91% but occasionally introduced extraneous “policy” language that didn’t exist in the client’s documentation. Claude Sonnet 4.6 performed almost identically to Opus but with slightly higher variance on edge cases. Haiku 4.5 was a strong performer for simpler classifications but struggled with multi-step reasoning beyond two hops. For a production system that required consistent high-stakes decisions, we narrowed the choice to Opus 4.8 and Terra.
Latency and Infrastructure Fit
The client’s ticket volumes peak at 8 a.m. ET, with spikes of 60 requests per second. The IBM iSeries backend is chatty—each ticket lookup takes 400–600 ms. We needed the AI response time to stay under 2 seconds end-to-end to avoid coordinator frustration. Self-hosted models were infeasible on the current budget, so we stuck with API offerings.
Anthropic’s API delivered Opus 4.8 responses consistently in 1.2–1.5 seconds for our prompt length. OpenAI’s Terra averaged 1.8 seconds, and occasionally hit 3-second p95s during US business hours. That latency delta might seem small, but multiplied across tens of thousands of monthly tickets, it would either slow workers or force us to add expensive caching layers. The operational win went to Opus.
Cost and Licensing Considerations
We modeled three-year total cost of ownership. Opus 4.8 was priced at a premium per token, but the higher accuracy meant fewer human reviews, which reduced downstream labor costs more than enough to offset the API bill. GPT-5.6 Terra was cheaper per token, but the additional oversight hours and potential rework from lower routing accuracy erased the savings. We also scrutinized licensing terms: Anthropic offered a standard commercial API with no data-use for training by default; OpenAI required an explicit opt-out and enterprise addendum. For a PE-backed firm with client confidentiality clauses, that distinction mattered.
The Client’s Workload and Our Final Call
The logistics operator’s workload demanded high reasoning fidelity and low latency, with a clear audit trail. We chose Claude Opus 4.8 as the primary model, with Claude Sonnet 4.6 as a fallback for peak traffic and a Claude Haiku 4.5 lightweight classifier that pre-filtered obvious “no-action” tickets before they hit the main model. This cascade saved 35% on inferencing costs while keeping end-to-end latency under the 2-second SLA.
Vendor Selection: Beyond API Calls
Choosing Anthropic as the vendor didn’t follow automatically from picking Opus 4.8. We could have hosted the model ourselves or used a managed reseller. We ran a full vendor evaluation that paralleled the models themselves.
Security, Compliance, and Audit Trails
Because the client was in the middle of an ISO 27001 audit readiness initiative facilitated by our Security Audit (SOC 2 / ISO 27001) service, we needed a vendor that could provide role-based access controls, encryption at rest and in transit, and immutable request logs. Anthropic’s enterprise tier delivered SOC 2 Type II reports and audit logs exportable to the client’s Vanta instance—exactly what the compliance team required. For any firm approaching platform engineering in San Francisco or building production AI platforms with diligence expectations, this degree of transparency is table stakes.
We often reference the AI Vendor Evaluation Checklist from Safe AI Australia during this phase; it forces you to define success criteria before sales conversations. The legal guide on key considerations for evaluating an AI vendor from Morgan Lewis was also invaluable in ensuring our contract didn’t inadvertently grant data rights.
Enterprise Tiers and Support Rigor
We compared Anthropic’s enterprise support against OpenAI’s equivalent. Anthropic offered a dedicated Slack channel and a 99.9% uptime SLA that aligned with our 24/7 logistics operation. OpenAI’s enterprise offering was comparably robust but required a multi-year commitment that the client wasn’t ready to make before proving initial ROI. The flexibility to scale up without a long-term lock-in was a deciding factor.
Integration and Ecosystem Lock-In
The client already used AWS heavily, and we were migrating some workloads toward a platform development approach in the United States that leveraged AWS Bedrock for multi-model access. Anthropic’s models were available in Bedrock, which meant we could start with the API, prove value, and then transition to a more integrated deployment without rewriting the integration layer. This reduced vendor lock-in risk and pleased the PE operating partner who wanted to keep the architecture fungible.
The Decision Matrix
We summarized the vendor evaluation with a weighted scorecard based on the CIO’s 7-dimension vendor evaluation framework and the CTO’s strategic guide to de-risking AI software vendor selection. Dimensions included data security, integration depth, support responsiveness, and contractual clarity. Anthropic scored highest on the dimensions that mattered most for a regulated, mid-market operator.
A note on the enterprise AI vendor checklist for regulated industries from Vidizmo: it provided an excellent cross-check for deployment and compliance that we built into our internal playbooks.
Guardrails: The Safety and Quality Net
No model, however accurate, operates in a vacuum. The client’s freight tickets often contained PII, internal cost data, and competitive details. We needed runtime guardrails that prevented prompt injection, data leakage, and generation of unapproved commitments.
The Taxonomy of Runtime Guardrails
Research from the taxonomy of runtime guardrails in foundation models paper gave us a structured vocabulary. We implemented three categories:
- Input filters that detect and redact PII before it reaches the model, and block prompt injections using a fine-tuned classifier.
- Output validators that check generated routing codes against a known good list and flag any draft email that contains pricing promises or contractual language not approved by the legal team.
- Operational monitors that track drift in model behavior over time, triggering a review if accuracy on the validation set drops below 90%.
We coupled these with the security evaluation principles from the F5 white paper on evaluating enterprise AI security, which emphasizes independent validation of detection effectiveness, not just vendor claims.
Implementation Architecture
Below is the guardrail flow we deployed. Every user query passes through the input filter, then to the model (Opus 4.8, with fallback), then through the output validator before reaching the coordinator’s screen.
graph TD
A[Coordinator submits ticket] --> B[Input Filter]
B --> |Cleaned & validated| C{Model Router}
C --> |Primary: Opus 4.8| D[Anthropic API]
C --> |Fallback: Sonnet 4.6| D
D --> E[Output Validator]
E --> |Valid| F[Show draft to coordinator]
E --> |Invalid or flagged| G[Manual Review Queue]
G --> H[Human corrects and logs]
H --> |Feedback loop| B
style B fill:#f9f,stroke:#333,stroke-width:2px
style E fill:#f9f,stroke:#333,stroke-width:2pxThis pattern is one we frequently recommend in our platform development engagements on the Gold Coast and Darwin, where remote operations require additional reliability layers.
Testing and Continuous Monitoring
We ran a red-team exercise before go-live, trying to trick the model into revealing customer data or authorizing a discount. The input filter caught 100% of our injection attempts; the output validator flagged three cases where the model hallucinated a phone number. By launch, we had a zero-critical-issue baseline. Post-launch, we use Vanta’s monitoring integrations to feed guardrail logs into the ISO 27001 evidence trail, which means the PE firm’s compliance officer can pull an audit report on demand without asking engineering.
Deployment, Tuning, and Measurable Results
We went live in week 10, two weeks ahead of schedule. The first month saw 15,000 tickets processed automatically. Average handling time dropped from 6 minutes to under 90 seconds for the 78% of tickets that the AI handled end-to-end. The human coordinators now focus on complex exceptions, which has improved both job satisfaction and service levels.
Routing accuracy, measured weekly on the holdout set, has stayed above 93%—meeting the 92% benchmark. Most importantly, the EBITDA lift from reduced misdirected freight and labor reallocation became visible in the quarterly financials, giving the PE operating partner a concrete number to report at the next board meeting. Qualitative gains, like the ability to handle higher volumes without adding headcount, have strengthened the investment thesis for the roll-up’s next add-on acquisition.
These results reflect the approach we bring to every engagement, whether it’s fractional CTO advisory in Sydney for a scale-up or platform development in San Francisco for a venture-backed SaaS. The selection process isn’t a one-off; it’s a repeatable asset.
Summary and Next Steps
Choosing the model, vendor, and guardrails for production AI is not a beauty contest of benchmark charts. It’s an exercise in aligning technical capability with business outcomes, compliance demands, and operational realities. For this logistics client, Claude Opus 4.8, Anthropic as the vendor, and a layered guardrail architecture delivered measurable ROI while meeting the rigid standards of a PE portfolio.
If you’re a CEO or board member of a mid-market company thinking about an AI feature that can’t fail, or a PE operating partner needing to consolidate tech and drive EBITDA across a portfolio, the same methodology applies. We’ve executed similar selections for defense and advanced-manufacturing clients in Adelaide, mining and energy firms in Perth, and tourism and health operators on the Gold Coast.
Here’s what you can do next:
- Book a discovery call with our fractional CTO practice. We offer CTO as a Service in Brisbane, Melbourne, and across the United States.
- Request a case study from our case studies page to see how we’ve delivered similar outcomes for other firms.
- Download our AI vendor evaluation template that builds on the frameworks cited here; it’s the same one we use internally.
We’re especially eager to hear from private equity firms executing roll-ups. The combination of tech consolidation and AI value creation is where our venture studio model excels. Reach out through padiso.co/services and let’s discuss how we can turn your next AI investment into a boardroom win.
This teardown reflects a real engagement, though some details have been generalized to protect client confidentiality. All model names and capabilities are current as of this writing: Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5, Fable 5, and GPT-5.6 (Sol and Terra).