Cloud & System Design Questions

In plain words: Run it in a safe, locked box with limited access. Let it search, edit and test in a loop. Record every step, check the result and have a person approve it.

Full answer · Clarify scope (repo size, languages, allowed actions, turnaround) then: API returns a run ID; a queue and durable workflow drive the loop; each run gets an isolated sandbox container with only the repo at a pinned commit, no secrets and an egress allow-list; the agent loops search → read → edit → run tests with budgets on steps, time and spend; every command and diff is appended to an immutable audit trail; gates run (tests, linter, static analysis, dependency and secret scans, diff size); the result is a branch and pull request for human review. State is checkpointed so a killed worker resumes; cancellation stops side effects; traces and cost per task are reported.

In plain words: A queue holds jobs in a waiting line so a slow part cannot crash the rest. Use it for uploads and background work.

Full answer · To decouple producers from workers and absorb bursts: asynchronous ingestion of documents, queued agent runs and tool executions, fan-out work items, and retry handling with dead-letter queues for poison messages. It gives durability and backpressure, so a slow model or downstream doesn't take down the API. Use visibility timeouts and idempotent consumers because delivery is at-least-once.

In plain words: Use it for long, multi-step jobs with waits and retries. One function would time out and lose track of where it was.

Full answer · When the workflow is multi-step, long-running or needs waits (such as human approval), built-in retries and error handling, parallel branches and an auditable execution history. A single Lambda has a maximum run time and holds compute while waiting on slow calls; a state machine persists state between steps and is visible and resumable. Use plain Lambda for short, single-purpose tool functions.

In plain words: Pass one ID through every service, record traces, metrics and logs, and set alarms for errors and stuck jobs.

Full answer · Propagate a run ID as a correlation ID through API Gateway, Lambdas, containers, queues and workflow executions; emit traces with X-Ray or OpenTelemetry, structured logs to CloudWatch, and custom metrics for tokens, cost, tool errors and steps per task. Store redacted prompt metadata, build dashboards for success, latency percentiles and spend, alarm on error rates, DLQ depth and spend velocity, and keep an audit table of actions and approvals.

In plain words: Count usage per customer, set limits and slow them down gently before stopping them. Plan this before customers arrive.

Full answer · Meter tokens, model calls, retrieval, storage and tool usage per tenant at the model gateway and API layer into a usage ledger; enforce plan quotas and rate limits with graceful degradation (smaller model, shorter outputs) before hard stops; track cost per successful task and margin per tenant; alert on anomalies; and provide tenant-level overrides. Design metering before the first customers arrive.

In plain words: Do not make one function wait. Start the job, save its progress and let a workflow engine run each short step.

Full answer · Don't make one function wait on a long LLM or tool chain. Make the entry point return a run ID, then drive the work with durable orchestration (Step Functions or a queue-based worker on containers), with each step a short function and state checkpointed in a database. Use callbacks or polling for slow external calls and move truly long-lived processes to ECS/EKS.

In plain words: Put all model calls through one gateway you control, so you can change models or providers by changing settings.

Full answer · Create an internal model gateway or provider abstraction that exposes your own request/response contracts, handles authentication, routing, quotas, logging and fallbacks, and keeps model names and parameters in versioned configuration. Applications depend on the contract, so you can migrate providers or models by changing configuration and passing the eval gate, not by rewriting services.