Voice AI Agents for Phone Calls: How They Work and When They Fit

A voice AI agent answers a phone call, talks in natural speech, and can look things up or book a slot while the caller waits. It fits narrow, repeated calls such as booking, order status and after hours messages. It does not fit upset callers or complicated cases. Three things decide whether callers accept it: how fast it replies, how it handles being interrupted, and whether a person is always one step away.

What a voice agent is made of

Every call passes through the same parts.

1. A phone line. This is a phone number from a provider. Providers such as Twilio can stream the raw audio of a live call to your software over WebSockets in near real time. That audio is phone quality, not studio quality. Twilio's streams use 8,000 samples per second.
2. Speech to text. It turns the caller's voice into text as they speak.
3. A language model. It reads the text, decides what to say, and calls tools, such as your calendar, order system or knowledge base.
4. Text to speech. It turns the answer back into a voice.

The other design is one speech to speech model. OpenAI's voice agents guide describes it as one model that listens, decides and answers in speech, with the session managing turns, tools, interruptions and handoffs. For the chained design, the same guide says to use it "when each stage needs to be visible or replaceable". An example is storing transcripts and running policy checks before the agent speaks.

	Chained pipeline	Speech to speech model
How it works	Speech to text, then a text model, then text to speech	One model listens and speaks
Strength	You can see and swap each stage and check the text before it is spoken	One model handles turns and interruptions in one session
Watch for	More parts, so more places for delay	Fewer places to inspect the words before they are spoken

Neither is better in every case. Choose the one you can test and measure.

Why speed and interruptions matter

People answer each other fast. A well known study of ten languages found a mean gap of 208 milliseconds between turns, and a median of 100. Danish speakers were the slowest on average at 469, and Japanese speakers the fastest at 7. No machine matches that, but it explains why a two second silence feels like a bad line.

Three habits keep the wait short:

• Keep replies short. Long answers also give callers more to interrupt.
• Say a short line while a lookup runs, such as "One moment, I am checking the calendar."
• Measure the wait callers feel, and look at the slow calls. OpenAI's guide suggests tracking backend time separately and comparing the median with the 95th percentile.

Interruptions are the second test. Callers talk over agents all the time, so the agent must stop speaking at once. With Twilio's two way streams, your server sends a clear message to empty the audio that is queued. In a speech to speech session, the guide says the session handles interruptions for you.

Handing the call to a person

A good handoff matters as much as a good answer. Send the call to a person when:

• the caller asks for one, every time and at once;
• the agent has failed to understand twice;
• the topic is a complaint, a money dispute or anything urgent;
• a tool the agent needs is down.

Phone providers can usually transfer a live call to a normal number. Pass a short summary first: the caller's name, the reason and what was tried, so the caller does not repeat everything. If nobody is in, take a message and a call back number, and keep the promise. Our post on chatbot best practices explains why handoff is the most important feature of any bot.

Recording, consent and telling callers it is an AI

Here is what three real rules say.

• United States, AI voices. The FCC adopted a declaratory ruling in February 2024. It confirms that the Telephone Consumer Protection Act's limits on an "artificial or prerecorded voice" cover current AI technologies that generate human voices. Calls that use them need the prior express consent of the called party, unless there is an emergency or an exemption. The ruling is about calls you start. If you plan outbound calls, ask a lawyer first.
• United States, recording. Federal law, 18 U.S.C. 2511, lets a person record a call they are part of, or where one party has consented, unless it is for a crime or a wrong. States can add their own rules, and some ask for everyone on the call to agree.
• European Union, disclosure. Article 50(1) of the AI Act says providers must design AI systems that talk to people so that people are told they are dealing with an AI, unless that is obvious to a reasonably well informed person. The duty sits with the provider, the party who builds the system and places it on the market.

Our habit is simple. Say it at the start: "This is the automated assistant for Harbor Dental. This call may be recorded." Keep recordings and transcripts only as long as you need them, because they hold personal data.

What drives the cost

We do not quote prices, because they depend on the setup. These are the cost drivers:

• Call minutes from your phone provider.
• The AI parts. Speech to text, the model and text to speech, or one speech model, usually billed by use. Check each provider's current price page.
• Length of calls. A chatty agent costs more per call.
• Integrations. Each system the agent uses takes work to connect and test.
• Testing and monitoring. Someone must listen to calls and fix mistakes.

The number that counts is the cost per call the agent finishes alone, compared with a person handling the same call. Calls that end with a person still cost you twice.

Good first uses

Choose a job with a narrow goal, an answer in a system, and a cheap mistake.

• Appointment booking and changes. Easy to confirm back to the caller.
• Order status. One question and one lookup.
• After hours calls. Opening times, a message, or a booking for tomorrow. A person would not answer anyway, so the risk is low.
• Routing. Ask the reason for the call and send it to the right team.

The same booking and order checks also work in text. See our posts on what AI agents can do for business and an AI agent on WhatsApp.

When not to use one

• Upset callers, bad news and emergencies.
• Medical, legal or money advice, where a wrong word can cause harm.
• Rare, complex questions that have no clear answer.
• Very few calls. A voicemail and a call back may cost less.
• Cases where you cannot meet the consent or disclosure rules.

Questions and answers

Can callers tell they are talking to an AI?

Often they can, and in some places you must tell them. We advise saying it in the first sentence. Callers who know are usually more patient and ask for a person sooner when they need one.

Should I use one speech to speech model or the chained design?

It depends on what you need to see. The chained design lets you store transcripts, check text and swap parts. The speech to speech design keeps turns and interruptions in one session. Pick one, test it on real calls, and measure the wait.

How fast must it answer?

People reply in a fraction of a second, so aim to keep pauses short. Track the median and the slowest calls. Use a short spoken line while the agent looks something up.

What happens when the agent does not understand?

Ask once more in different words, then hand the call to a person. Never loop. A caller stuck in a loop will hang up and remember it.

Do I need to record the calls?

No, but you need some record, such as transcripts, to find and fix mistakes. Recording has legal rules, so say so at the start of the call and ask a lawyer how it applies where you work.

Can the agent call customers first?

Technically yes, but the FCC ruling above says calls with AI generated voices in the United States need prior express consent. Other countries have their own rules. Ask a lawyer before you build outbound calls.

References and further reading

• FCC Declaratory Ruling 24 17 docs.fcc.gov The ruling on AI voices under the Telephone Consumer Protection Act, adopted 2 February 2024.
• 18 U.S.C. 2511 law.cornell.edu The US federal law on intercepting calls, including the consent exception.
• EU AI Act, Article 50 artificialintelligenceact.eu The transparency duties for AI systems that talk to people.
• Voice agents guide developers.openai.com OpenAI's guide to speech to speech and chained voice designs, with advice on measuring waits.
• Twilio Media Streams twilio.com How live call audio is sent to your application.
• Universals and cultural variation in turn taking in conversation pmc.ncbi.nlm.nih.gov The study of reply gaps across ten languages.

A voice agent works when the job is narrow, the handoff is clean and the legal basics are checked first. If you want to test one on your own calls, see our AI agents service and our wider AI services.