Table of Contents
- Understanding Custom Tools in AI Agents
- Architecting Tools for Reliability at Scale
- Implementation Patterns and Code-Level Recommendations
- Security and Governance in Tool Authoring
- Testing and Evaluation of Custom Tools
- Operational Monitoring and Observability
- Operational Quirks Teams Hit at Scale
- Integrating Custom Tools into a Broader AI Strategy
- Summary and Next Steps
Understanding Custom Tools in AI Agents
Custom tool authoring is the single most leveragable engineering capability for production agent deployments. When mid-market brands and scale-ups move beyond proof-of-concept, they quickly discover that off-the-shelf tool integrations fail to capture the nuanced business logic that drives real value. At PADISO, we see this pattern repeatedly across AI Strategy & Readiness engagements: the difference between a promising prototype and a reliable production agent often comes down to how thoughtfully you write your own tools.
What Makes a Tool Agent-Friendly
Anthropic’s guide on writing tools for agents cuts to the chase: an agent-friendly tool is one that a model can reason about with minimal ambiguity. That means leaning heavily into OpenAPI contracts, typed input/output schemas, and descriptions written for a developer—not an end user. When we author tools for clients on AI & Agents Automation projects, we enforce a rule: every parameter must include a description field that would make sense to a senior engineer who knows nothing about the immediate domain. The model ingests these descriptions and uses them to decide when and how to call the tool.
Microsoft’s documentation on authoring agents in code reinforces this, stressing that tool definitions should be explicit about side effects. If a tool writes to a database or sends an email, the agent needs to know that. Without this metadata, the agent will treat every tool as a read-only function, leading to destructive operations in production. One of our fractional CTO clients in New York discovered this the hard way when an agent inadvertently launched 300 EC2 instances because the provisioning tool lacked a clear “mutating” flag.
Tool Design Principles
Start with purpose-built tools, not Swiss-army knives. A tool should do one thing well and be trivially composable. Think Unix philosophy: small, focused utilities that pipe together. In agentic AI, this translates to a tool that takes 2–4 inputs and returns a predictable, typed output. Avoid tools that return free-text success messages; agents struggle to parse natural language feedback reliably. Instead, return structured JSON with a status field, error code, and payload.
OpenAI’s practical guide to building agents advises keeping tool interfaces consistent across an organization. When multiple teams build tools for the same agent ecosystem, adopting a shared style guide for tool definitions drastically reduces prompt engineering overhead. In our Platform Design & Engineering work in San Francisco, we enforce a canonical tool template that every custom tool must follow—this alone cut tool-related production incidents by over 40% for one Series B startup.
Architecting Tools for Reliability at Scale
Production tool authoring is as much about the surrounding infrastructure as it is about the function body. A tool that works perfectly in a local Jupyter notebook can crumble under concurrent agent requests. This is where classic platform engineering principles apply: circuit breakers, retry budgets, and timeout-aware calls.
Core Architecture Pattern
We recommend a three-layer pattern: tool interface layer (the definition the agent sees), execution layer (the business logic), and transport layer (how we reach external APIs or data stores). The interface layer is pure configuration—think OpenAPI spec or equivalent. The execution layer runs inside a serverless function or container with strong isolation. The transport layer handles authentication, rate limiting, and error mapping. This separation means the agent never sees network-layer errors; they are translated into meaningful tool-specific error codes the agent can reason about.
graph TD
A[AI Agent] -->|Structured Tool Call| B[Tool Interface Layer]
B -->|Validate & Route| C[Execution Layer]
C -->|Domain Logic| D[Business Services / APIs]
D -->|Data| C
C -->|Tool Response| B
B -->|Structured JSON Response| A
E[Observability & Logging] -.-> B
E -.-> C
F[Secrets Manager] -.-> C
style A fill:#e1effe,stroke:#3b82f6
style B fill:#fef3c7,stroke:#f59e0b
style C fill:#d1fae5,stroke:#10b981This architecture allows us to swap out the underlying execution logic without re-authoring the tool interface. When a client on our CTO as a Service retainer needed to migrate a tool from a legacy SOAP endpoint to a GraphQL layer, the agent continued to function identically—the tool definition never changed.
Error Handling and Resilience
You cannot assume the agent will gracefully handle every error. Expect it to retry calls that should not be retried. A common anti-pattern is returning a 5xx HTTP status on a tool failure; the agent sees a server error and retries, sometimes causing double-orders in financial systems. Instead, map all business failures to a tool_error field inside a successful 200 response. The model then reads the error payload and decides the next action.
Google Cloud’s guide on building AI agents with custom tools emphasizes idempotency keys. Any mutating tool should accept an idempotency key and check for duplicates. This is non-negotiable in high-stakes environments. For a private-equity-backed logistics firm in Brisbane, we engineered tools that embed the original agent call ID as an idempotency key; this eliminated a recurring bug where inventory adjustments were applied twice.
Implementation Patterns and Code-Level Recommendations
Moving from architecture to code, there are several patterns that repeatedly prove themselves in production. These are pragmatic recommendations born from shipping agentic AI in mid-market environments where time-to-ship and reliability compete aggressively.
Structured Tool Definitions
Every tool should be defined as a typed schema, not a creative prompt. We prefer the Anthropic tool-use format, which is converging with OpenAI function calling. A minimal tool definition looks like:
{
"name": "lookup_customer",
"description": "Retrieve a customer record by account number. Does not modify data.",
"parameters": {
"type": "object",
"properties": {
"account_number": {
"type": "string",
"description": "Unique account number, 10-digit numeric string."
}
},
"required": ["account_number"]
}
}The key is the description fields: they constrain the model’s behavior. Without them, the agent might hallucinate account numbers or call the tool for non-customer entities. We see far fewer hallucinations when descriptions include data type hints and example values.
Stateless vs Stateful Tools
Production tools should be stateless whenever possible. Statefulness introduces ordering dependencies and makes concurrency tricky. However, some business processes inherently span multiple calls—like a multi-step approval workflow. In these cases, externalizing state to a durable store (DynamoDB, Redis) and passing a conversation_id or run_id as a tool parameter is essential. The tool retrieves its current state from the store and updates it atomically. Engineers at a Melbourne health scale-up we advise wrapped stateful tool calls in optimistic locking patterns to prevent two concurrent agent steps from stepping on each other.
Tool Composition and Chaining
Don’t force the agent to chain tools manually if a higher-level abstraction can combine them safely. A composite tool that orchestrates a sequence of sub-tools reduces the number of reasoning steps the agent must perform, lowering latency and token costs. For example, a process_refund tool might internally call validate_customer, check_order_status, execute_refund, and send_confirmation. The agent sees only the combined tool, which is treated atomically. This is a pattern we use extensively in AI for Financial Services engagements in Sydney, where regulatory compliance demands strictly ordered operations with audit trails.
Security and Governance in Tool Authoring
Tools are the boundary at which an AI agent touches real-world systems. Every tool call is a potential vector for prompt injection, data leakage, or unauthorized action. Mid-market companies often underestimate this risk until a security auditor—or worse, a real incident—highlights the gap.
Least Privilege and Scope Limitation
A tool’s permissions should be scoped to exactly what the agent needs for its intended task. Never give a tool blanket access to a database; instead, define read-only queries with row-level security. For write operations, use parameterized mutations that limit the blast radius. In one US-based venture studio project, we implemented a “scope token” that restricts a tool call to the specific customer record and timeframe relevant to the current conversation. Even if the agent is tricked into making a rogue call, the damage is contained.
Auditability and Compliance
For clients targeting SOC 2 or ISO 27001 audit-readiness, every tool invocation must be logged with sufficient context: who made the call (the agent’s run ID), what parameters were passed, the response, and timestamps. This is non-negotiable. We use structured logging (JSON over stdout) and instrument tools with OpenTelemetry to feed into Vanta dashboards. A Perth mining services firm we support now has full traceability from agent decision down to individual SQL statements, which satisfied their ISO 27001 auditor.
To be clear: we don’t promise regulatory outcomes; we provide the technical scaffolding for audit-readiness. The distinction matters for legal reasons, and our Security Audit service explicitly frames it this way.
Testing and Evaluation of Custom Tools
Traditional software testing doesn’t fully capture agent-tool interactions. You need to test both the tool in isolation and the agent’s ability to use the tool correctly.
Unit Testing Tools Independently
Every tool should have a standard set of unit tests: positive cases, invalid inputs, timeout scenarios, and authorization failures. But for tools called by agents, you must also test “confusing” inputs—parameters that are technically valid but likely to be produced when the agent misinterprets user intent. For example, a customer lookup tool might receive a name when it expects an account number. The tool should return a clear, typed error, not silently return an empty result.
Evaluating Tool-Augmented Agent Performance
Addy Osmani’s post on writing good specs for agents advocates for self-verifying specifications. We extend this to tools: define an eval harness that sends a diverse set of conversational inputs to the agent and checks whether the correct tool is selected and whether the returned data is used appropriately. These evals should run in CI/CD. When a tool is updated, you can catch regressions where the agent suddenly misuses it. One Adelaide defence-tech client integrates these evals into their deployment pipeline; a failing eval blocks the release.
Tool documentation is another lever. When tool interfaces evolve, the documentation must be updated automatically. A Relevance AI agent for generating technical documentation can keep internal SDKs in sync, but we caution that generated docs still need human review for accuracy in high-stakes environments.
Operational Monitoring and Observability
Once tools are live, observability becomes your primary defense against degradation. The agent’s behavior can drift as models update or as usage patterns change.
Key Metrics to Track
At a minimum, monitor per-tool:
- Latency (p50/p95/p99): Slow tools cause agent timeouts.
- Error rate by type: Distinguish business errors (invalid input) from infrastructure errors.
- Tool call frequency: Sudden spikes may indicate a prompt injection attack or a reasoning loop.
- Token consumption per call: Tools that return verbose output drive up cost.
We build custom Superset + ClickHouse dashboards (a specialty of our platform development teams) that show these metrics against baselines. When a tool’s p95 latency drifts above the SLO, an alert pages the on-call engineer.
Debugging Tool Failures in Production
Mermaid diagrams can clarify complex debugging flows. Here’s a typical decision tree when a tool returns an unexpected error:
flowchart TD
A[Agent reports tool error] --> B{Is error from tool code?}
B -- Yes --> C[Check tool logs]
C --> D[Is input schema mismatched?]
D -- Yes --> E[Update tool definition]
D -- No --> F[Check external API health]
F --> G[Apply circuit breaker]
B -- No --> H[Check agent reasoning]
H --> I[Was tool description ambiguous?]
I -- Yes --> J[Refine tool description]
I -- No --> K[Add few-shot examples to agent prompt]We’ve seen cases where the agent consistently misused a tool because its description was slightly misleading. Refining the description fixed the issue without touching a line of tool code. This highlights how tightly tool authoring ties into prompt engineering—a key focus of our AI advisory work in Sydney.
Operational Quirks Teams Hit at Scale
Deploying AI agents with custom tools at scale surfaces friction points that small-scale prototypes never reveal. Here are the top quirks we assist mid-market CTOs and PE portfolio managers with.
Token Budget Explosion
Large tool outputs consume context window fast. An agent that calls a list_invoices tool and gets back 200 records will burn thousands of tokens before the model can even consider the next step. We enforce strict output schemas with pagination. Tools must support a limit and cursor parameter, and the agent is instructed via its system prompt to request increments of 10–20 items. This is a core practice in our Venture Architecture & Transformation work.
Indeterminate Tool Selection
When an agent has many similar tools, it sometimes picks the wrong one—or oscillates. A canonical case: a customer support agent with both search_knowledge_base and search_faq tools. The solution is to ensure tool descriptions are maximally differentiable. Use negative examples in descriptions: “Use this tool when looking for recent technical articles, not for policy-like FAQ entries.” Further, implement a “tool cooldown” mechanism: if the same tool is called three times in a row unsuccessfully, force the agent to try a different approach.
Input Drift and Schema Evolution
Tools evolve, but agent prompts often lag. When a tool gains a new required parameter, agents fail with cryptic errors because the model wasn’t informed. Agent-friendly API documentation is an evolving practice, but we recommend versioning tool schemas and maintaining a changelog that the agent can read. Some forward-thinking teams include a version field in the tool response; the model can then detect when its instructions are stale.
Integrating Custom Tools into a Broader AI Strategy
Custom tool authoring doesn’t happen in a vacuum. It is one piece of a broader AI transformation—whether you’re a PE firm driving portfolio value creation or a scale-up looking for measurable AI ROI. We consistently see that the firms with the strongest tooling culture are those that invest in platform engineering and fractional CTO leadership. They treat tools as shared assets, governed like internal APIs.
Private equity operating partners should view tool authoring as a consolidation lever. Across a roll-up, standardizing on 50 well-crafted tools can replace 500 bespoke scripts, delivering direct EBITDA lift through headcount reduction and error elimination. Our case studies show how this works in practice.
Summary and Next Steps
Custom tool authoring is the engineering discipline that separates production AI agents from science projects. It demands a structured approach: typed tool definitions, layered architectures, rigorous error handling, security-first design, and continuous evaluation against agent performance. The quirks at scale—token budget management, tool selection ambiguity, schema drift—are manageable once you know to look for them.
If your team is shipping agents that rely on homegrown tools, start with a tool audit: are your definitions explicit enough for a model to reason about? Are you logging every invocation? Do you have idempotency and circuit breakers in place? If the answers are “no,” you have immediate cost-saving and reliability-improving work.
For CEOs, boards, and PE firms, the strategic question is whether to build this capability in-house or partner with a firm that has done it repeatedly. PADISO brings that pattern recognition through our CTO as a Service and Venture Architecture & Transformation practices. We don’t write decks—we ship tools that drive EBITDA, whether you’re in New York, Dallas, Sydney, or Melbourne. Book a call.