Eight steps to build an AI voice agent that answers a phone line: choose one job, connect the call, write the limits, handle delay, add a human hand-off, test with real calls and go live safely.
You will use:A phone number providerA speech-to-text serviceA language modelA text-to-speech service
A voice agent answers phone calls with a spoken conversation. This playbook builds one in eight steps, in the order that avoids the most common mistakes. Use it as a checklist, and swap in whichever services you prefer.
Choose one narrow job, such as booking an appointment or answering opening-hours questions. Write down three things:
There are two common designs.
Start with the chain. You can move to a live voice model later, once you know what the calls look like.
Rent a phone number from a telephony provider, and point it at your agent so that call audio streams to your code and your answers stream back. Test with your own phone before anything else. Check that you can hear the agent, and that it can hear you.
Write a short prompt that says the job, the tone, the limits from Step 1 and what to do when it is unsure. Give the agent only the tools it needs, such as look up a booking. Let your code check every tool call before it runs.
# A sketch of one turn. Replace the helper functions with your own services.
MAX_TURNS = 20
def handle_turn(call, audio_chunk):
text = speech_to_text(audio_chunk) # hear
if wants_a_person(text) or call.turns >= MAX_TURNS:
return hand_off(call) # send to a human
reply = ask_model(call.history, text) # think (may ask for a tool)
if reply.tool:
check_allowed(reply.tool, reply.args) # guard in code
result = run_tool(reply.tool, reply.args)
reply = ask_model(call.history, result)
call.history.append((text, reply.text))
return text_to_speech(reply.text) # speakA pause of more than about a second feels strange on a phone. Stream audio in small pieces, start speaking before the whole answer is ready, and stop talking at once when the caller speaks. If a tool call is slow, say a short line such as "One moment while I check that."
Always give an easy way to reach a person. Hand over when the caller asks, when the agent is unsure twice in a row, when the topic is outside the limits, or when the call runs long. Pass the person a short summary, so the caller does not have to repeat everything.
Test with real recorded calls from your own callers, in their languages and with background noise. Check names, numbers and codes by reading them back. Keep a list of twenty calls and replay it after every change. Mark each call pass or fail, and fix the failures that matter most.
Want the fundamentals first?
The agentic engineer course teaches tools, memory, evaluation and safety, and the AI Radar keeps you current.