How does an AI agent call a third-party API without ever holding the API key itself?

The agent never gets a real credential — it gets a tool name and a JSON blob of arguments, and the platform's execution plane is what actually calls the provider. The real OAuth token lives in a token vault, encrypted under a key held in a KMS (a service that keeps the master encryption key in hardware and never lets it leave), and the executor fetches and decrypts it only for the instant of the call, on the tenant's and end user's behalf. This is the same shape Stripe Connect and Plaid use for a human developer's app acting on a user's payments or bank account: the platform is the one party that ever touches the live credential.

What is MCP (Model Context Protocol) and how does it relate to a tool-calling platform like this one?

MCP is a wire protocol, not a platform — the Model Context Protocol's own docs describe it as "an open-source standard for connecting AI applications to external systems," and compare it to "a USB-C port for AI applications": one plug shape, many devices. It standardizes two JSON-RPC calls, tools/list (what can I call?) and tools/call (call this one, with these arguments), plus a result envelope. None of that says anything about where a token vault lives, how retries work, or how a provider's rate limit gets shared fairly across tenants — that is everything this page builds. A real platform can expose its entire catalog through an MCP server as a thin adapter in front of the same router, vault, and executor described here.

Why can't you just hand the model every tool schema in a 10,000-tool catalog?

Because each schema is billed and counted like any other input token, on every single call, whether the model uses it or not — both Anthropic's and OpenAI's own docs are explicit that tool definitions "count against the model's context limit and are billed as input tokens." A minimal schema runs under 100 tokens, but a well-documented real one commonly runs several hundred; even a conservative average across 10,000 tools adds up to more tokens than most models' entire context window, spent before the conversation starts. OpenAI's own guidance is blunt about the practical ceiling: "aim for fewer than 20 functions available at the start of a turn." The fix is a router that loads only the handful of tools a given request plausibly needs — Anthropic ships exactly this as a dedicated tool-search capability, built to "work with thousands of tools by discovering and loading them on demand."

How do you stop one tenant's traffic burst from exhausting a shared third-party rate limit for every other tenant?

Real provider limits are not generous relative to a multi-tenant platform's traffic — GitHub caps an authenticated token at 5,000 requests an hour, and Slack tiers its Web API methods from roughly 1 to 100-plus calls a minute per method per workspace, returning a 429 with a Retry-After header once a tier is exhausted. Route every tenant through one shared app credential with no accounting, and a single tenant's burst can eat the whole hourly budget, so everyone else starts seeing 429s that have nothing to do with their own traffic. The fix is two independent limiters in the executor: one bucket per provider (never send more than the provider actually allows, in aggregate) and one bucket per tenant within that provider (so no single tenant can claim more than its fair share of the shared budget).

What happens when a provider returns a 429 in the middle of an agent's tool call?

The executor does not simply retry immediately — synchronized immediate retries from many callers are exactly what turns a brief provider hiccup into a pile-on. It computes a jittered backoff delay (AWS's well-known "full jitter" formula: sleep = random(0, min(cap, base × 2^attempt))) and drops the call back onto its own retry queue, carrying the same idempotency key so a duplicate delivery can never double-book the same action. If failures on that provider keep piling up, a circuit breaker trips open and short-circuits further calls immediately — the classic closed → open → half-open state machine — so the platform stops hammering a provider that is clearly down and lets it recover instead of amplifying the outage.

Design a Tool-Calling Platform for AI Agents

Thousands of tenants, each running agents that need to act on a real SaaS account on a real end user's behalf — a shared connector catalog, a token vault, and an execution plane that survives one provider having a bad day.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design a Tool-Calling Platform for AI Agents — built to be understood, not skimmed.

Questions

How does an AI agent call a third-party API without ever holding the API key itself?: The agent never gets a real credential — it gets a tool name and a JSON blob of arguments, and the platform's execution plane is what actually calls the provider. The real OAuth token lives in a token vault, encrypted under a key held in a KMS (a service that keeps the master encryption key in hardware and never lets it leave), and the executor fetches and decrypts it only for the instant of the call, on the tenant's and end user's behalf. This is the same shape Stripe Connect and Plaid use for a human developer's app acting on a user's payments or bank account: the platform is the one party that ever touches the live credential.
What is MCP (Model Context Protocol) and how does it relate to a tool-calling platform like this one?: MCP is a wire protocol, not a platform — the Model Context Protocol's own docs describe it as "an open-source standard for connecting AI applications to external systems," and compare it to "a USB-C port for AI applications": one plug shape, many devices. It standardizes two JSON-RPC calls, tools/list (what can I call?) and tools/call (call this one, with these arguments), plus a result envelope. None of that says anything about where a token vault lives, how retries work, or how a provider's rate limit gets shared fairly across tenants — that is everything this page builds. A real platform can expose its entire catalog through an MCP server as a thin adapter in front of the same router, vault, and executor described here.
Why can't you just hand the model every tool schema in a 10,000-tool catalog?: Because each schema is billed and counted like any other input token, on every single call, whether the model uses it or not — both Anthropic's and OpenAI's own docs are explicit that tool definitions "count against the model's context limit and are billed as input tokens." A minimal schema runs under 100 tokens, but a well-documented real one commonly runs several hundred; even a conservative average across 10,000 tools adds up to more tokens than most models' entire context window, spent before the conversation starts. OpenAI's own guidance is blunt about the practical ceiling: "aim for fewer than 20 functions available at the start of a turn." The fix is a router that loads only the handful of tools a given request plausibly needs — Anthropic ships exactly this as a dedicated tool-search capability, built to "work with thousands of tools by discovering and loading them on demand."
How do you stop one tenant's traffic burst from exhausting a shared third-party rate limit for every other tenant?: Real provider limits are not generous relative to a multi-tenant platform's traffic — GitHub caps an authenticated token at 5,000 requests an hour, and Slack tiers its Web API methods from roughly 1 to 100-plus calls a minute per method per workspace, returning a 429 with a Retry-After header once a tier is exhausted. Route every tenant through one shared app credential with no accounting, and a single tenant's burst can eat the whole hourly budget, so everyone else starts seeing 429s that have nothing to do with their own traffic. The fix is two independent limiters in the executor: one bucket per provider (never send more than the provider actually allows, in aggregate) and one bucket per tenant within that provider (so no single tenant can claim more than its fair share of the shared budget).
What happens when a provider returns a 429 in the middle of an agent's tool call?: The executor does not simply retry immediately — synchronized immediate retries from many callers are exactly what turns a brief provider hiccup into a pile-on. It computes a jittered backoff delay (AWS's well-known "full jitter" formula: sleep = random(0, min(cap, base × 2^attempt))) and drops the call back onto its own retry queue, carrying the same idempotency key so a duplicate delivery can never double-book the same action. If failures on that provider keep piling up, a circuit breaker trips open and short-circuits further calls immediately — the classic closed → open → half-open state machine — so the platform stops hammering a provider that is clearly down and lets it recover instead of amplifying the outage.

Design a Tool-Calling Platform for AI Agents

Questions

Related explainers