Deploying AI Agents to Production

Short, bounded turns — a quick classification, a single retrieve-and-answer step — fit serverless functions beautifully. They autoscale to zero, you pay per invocation, and a spiky workload costs nothing while idle. The catch is execution limits: most serverless platforms cap how long a function can run and how long it can stream, which collides head-on with an agent that loops for minutes.

For those longer runs, reach for long-lived containers or a queue-and-worker pattern. A thin API accepts the request, drops a job on a queue, and returns immediately; a pool of worker containers pulls jobs and executes the full agent loop for as long as it needs, streaming progress over a websocket or storing results for the client to poll. This decouples the user-facing latency budget from the agent's actual wall-clock time and lets you scale workers independently of the API.

Most mature deployments end up hybrid: serverless at the edge for fast turns and cheap fan-out, a durable queue in the middle, and container workers for the heavy, long-running reasoning. Whatever you pick, the workers must be stateless — all run state lives outside the process, which is the subject of the next section.

Production agents carry several kinds of state, and each needs a durable home outside the worker process. Conversation state — the running message history and tool results — must survive a reconnect or a worker swap. Working memory — the scratchpad, plan, and intermediate findings within a single run — should be checkpointed so a crashed run can resume instead of starting over. And long-term memory — facts the agent should recall across sessions — belongs in a database or a vector store the agent queries on demand.

The practical rule is that any worker should be able to pick up any run by loading its state from storage. A relational or document database handles structured run state and checkpoints; an object store holds large artifacts; a vector store backs retrieval and long-term recall. Checkpoint after each step so a failure costs you one step, not the whole run — and so you can insert a human approval pause and resume exactly where you left off.

Treat memory as something to curate, not just append. Summarize and prune old turns to keep context small and cheap, expire stale facts, and keep provenance on stored memories so you can trust and audit what the agent recalls. Read more in the deep dive on observing agent state.

Never put credentials in prompts, code, or model context. Inject them at runtime from a secrets manager, scope them tightly, and rotate them on a schedule. The model should be able to invoke a tool that uses a credential without ever seeing the credential itself — the secret lives in the tool's execution environment, not in the conversation.

Then scope each tool to least privilege. A retrieval tool should read one index, not your whole warehouse; a write tool should touch one tenant's records, not everyone's. Because an agent decides for itself which tools to call — and can be steered by prompt injection from untrusted content — assume any tool it holds can be triggered by an adversary, and grant accordingly. Pair narrow permissions with input validation on tool arguments and allow-lists on destinations.

Finally, separate read tools from write tools, and gate the dangerous writes. Reversible reads can run freely; irreversible or high-impact actions should require an explicit approval step, covered below. Log every tool call with its arguments and result so you have an audit trail when something goes wrong.