Deploying to Production + Capstone

Agents often fail quietly in production. They can make things up with full confidence. They can call the same API hundreds of times before anyone notices. That is the runaway loop from the tool-calling lesson, at full size. So safety limits and budget limits are part of the foundation. They are not extras to add before launch.

Four levels of guardrails

• Input checks: reject or clean what comes in before it reaches the model or a tool.
• Output checks: check the structure, safety and consistency of what the model produces. The schema checks from the prompting lesson belong here.
• Limited access: give each agent only the tools and permissions its task needs.
• Instructions in the prompt: the rules you write in the system prompt. These are the weakest layer and the last line of defense, not the first.

Safety and security are two different problems

Safety is about mistakes nobody intended: made-up facts, wrong actions and runaway costs. There is no attacker. Security is about people on purpose trying to make the agent misbehave, for example with prompt injection or by flooding it with tokens. A real agent needs protection for both. A control built for one does not cover the other.

Budget limits in layers

One cap in your app is not enough. You need limits for each session, each user, each tool, and also at the infrastructure level. Then a single runaway session cannot get past the app's check and burn through everything. For simple safety checks, you can use small, cheap models instead of your most expensive one.

Policy as code

Tools such as Open Policy Agent (OPA) let every tool call pass through a policy checkpoint first. The checkpoint tests things like the kind of resource, the size of the inputs and the compliance labels, against written rules. There is no doubt about what is allowed. That is far better than scattering permission rules through prompts that the model might ignore.

async function authorize(call: ToolCall, ctx: SessionContext): Promise<boolean> {
  const decision = await policyEngine.evaluate({
    tool: call.name,
    args: call.arguments,
    user: ctx.userId,
    sessionSpend: ctx.spendSoFar,
    limits: ctx.budgetLimits,           // per-session / per-user / per-tool
  });
  return decision.allow;                 // deny by default on any failure
}

Capstone project

Build and ship a small agent of your choice that shows everything in this path working together. It needs a real message loop with tool calling. It needs at least one hard budget limit, enforced in more than one layer. It needs input and output guardrails. And it needs basic tracing that you can point to and explain. The goal is not size. Every piece must be real, not faked.

• Required: tool calling with answers checked against a schema.
• Required: at least one hard limit on budget or steps that really stops the agent.
• Required: input and output guardrails, plus one policy check before any tool runs.
• Required: a trace you can walk through from start to finish for one full run.
• Optional, if it fits your task: search over documents (RAG), a named framework, or MCP.

Going deeper: a production design

Most production agents end up with a boring design on purpose. A thin API accepts a request and returns a run ID. A queue holds the work. Workers that keep no memory of their own run the agent steps and save their progress in a state store. Results are streamed or sent back when ready. Long agent runs do not belong inside a single web request.

Client -> API (auth, rate limits, validation) -> enqueue(run_id) -> Queue
                                                        |
                                    Workers (stateless, autoscaled)
                                      |  load state(run_id) from State store
                                      |  call model / tools with timeouts + retries
                                      |  checkpoint state after each step
                                      v
              Result store  <-  emit events (SSE / websocket / webhook) -> Client
Cross-cutting: secrets manager, tracing + usage ledger, policy layer, kill switch, cost circuit-breaker

• Stream partial progress to the user. How fast it feels matters as much as how fast it is.
• Keep secrets out of prompts. Add logins in the tool layer, from a secrets manager.
• Use separate keys and budgets for development, testing and production.
• Make runs resumable. Workers can die. A run must continue from its last saved step.
• Launch behind a feature flag with an off switch. Release prompts and models as versioned bundles.
• Set limits per user and overall, plus spending caps, before launch and not after the first surprise bill.

Capstone: how it will be judged

• Works from start to finish on 5 realistic tasks you did not tune it on (30%).
• Guardrails: hostile inputs are blocked or contained, and the budget stops a deliberate loop (25%).
• Tests: at least 20 cases, run several times, with a written pass mark (20%).
• Observability: a trace you can follow to explain a failure (15%).
• Documentation: a one-page README with what it can do, its limits, its budgets and known failures (10%).

Here are some capstone ideas that show real skill. A support-triage agent with approval gates. A research agent that returns briefs with sources and a separate fact-check step. A codebase question agent with hybrid search and a search test. A document agent that pulls out structured data and sends doubtful cases to a person. Publish the code, the test results and your write-up of what failed. The failures show your judgment.