An AI voice agent is software that holds a real phone conversation: it listens, understands, decides what to say or do, and speaks back in a natural voice. You can build a capable one today without writing much code by pairing a Twilio phone number with ElevenLabs' conversational agents platform. We run one ourselves. It is a Twilio number connected to an ElevenLabs agent that places outbound calls on the owner's behalf, such as confirming appointments or getting a simple answer from a vendor.

This guide walks through how the pieces fit, how to set them up, and the decisions that matter more than the technology.

How the pieces fit

There are three layers:

  • Telephony. Twilio provides the phone number and carries the call over the phone network.
  • The agent. ElevenLabs' agents platform (branded ElevenAgents, and previously called Conversational AI) handles speech recognition, runs a language model with your instructions, and generates the spoken reply in the voice you choose.
  • Your systems. Tools and integrations let the agent check a calendar, look up a record, send a summary or transfer the call.

ElevenLabs offers a native Twilio integration, so you do not have to build the audio streaming yourself. You import the Twilio number into ElevenLabs, assign an agent to it, and ElevenLabs configures the number in Twilio for you.

Step 1: Get a Twilio number

Create a Twilio account and buy a phone number with voice capability in the country and area code you want. Twilio charges a monthly fee per number plus per-minute usage, and both vary by country and number type, so check Twilio's current pricing.

Two options worth knowing:

  • A purchased Twilio number can make and receive calls through ElevenLabs.
  • A verified caller ID (an existing business number you verify in Twilio) can be used for outbound calls only. It cannot receive calls or be assigned to an agent for inbound. This is useful if you want the agent's outbound calls to show your existing main number.

For credentials, ElevenLabs accepts either your Twilio Account SID and Auth Token or a Twilio API key and secret. Use an API key. It can be revoked on its own without changing your main account credentials, and the ElevenLabs documentation recommends a key with limited permissions.

Step 2: Create the agent in ElevenLabs

In the ElevenLabs agents dashboard, create a new agent. The core settings are:

  • First message. What the agent says when the call connects. For outbound calls this matters a lot, because the agent speaks first. Keep it short and say who is calling and why.
  • System prompt. The agent's instructions: who it is, who it represents, what it is trying to accomplish, what it must not do, and when to end or transfer the call.
  • Voice. Choose a voice from the library or design one. Pick something clear on a phone line rather than something dramatic.
  • Language model. ElevenLabs lets you choose the underlying model. Faster models reduce awkward pauses; stronger models follow complex instructions better. Test both on your real script.
  • Knowledge base. Upload documents or add URLs and text the agent can draw on, such as hours, service descriptions and policies.

Step 3: Write the prompt like a job description

The prompt is where most voice agents succeed or fail. We write it in plain language with clear sections:

Role: You are the scheduling assistant for [Company], calling on behalf of [owner].
Goal: Confirm the appointment on [date] at [time], or find a new time.
Rules:
- Say you are a virtual assistant if asked.
- Never agree to pricing, contracts or payments.
- If the person asks for [owner], offer to have them call back and take a message.
- Read back any date, time, phone number or email before ending the call.
- If you do not know the answer, say so and offer to have someone follow up.
Ending: Thank them, confirm next steps in one sentence, and end the call.

Short sentences, explicit limits and a defined ending produce better calls than a long, clever prompt. Write the way you would brief a new assistant on their first day.

Step 4: Add tools

Tools let the agent act during the conversation. ElevenLabs supports several kinds:

  • System tools built into the platform, such as ending the call, transferring to another number or agent, detecting voicemail, and pressing keypad tones to get through a phone menu.
  • Server tools (webhooks) that call your own API or an automation platform. For example, the agent can check calendar availability or write a call summary to your CRM.
  • Dynamic variables that inject per-call details, such as the person's name or appointment time, into the prompt.

Start with the minimum. A useful first agent often needs only end call, transfer, voicemail detection and one webhook that sends a summary somewhere a person will read it. We often point that webhook at an automation workflow (we use n8n) that posts the result to email or a sheet.

Step 5: Import the number and connect the agent

In the ElevenLabs dashboard, open the phone numbers section, choose the Twilio option, and enter a label, the phone number and the Twilio credentials. ElevenLabs detects whether the number supports inbound, outbound or both. For inbound, assign the agent to the number. Calls to that number now reach the agent.

The ElevenLabs Twilio integration documentation has the current details.

Step 6: Place outbound calls

You can start an outbound call from the dashboard by choosing the number, the agent and the destination. That is how you should test. For real use, ElevenLabs provides an API endpoint to start an outbound call through Twilio, which lets an automation or another AI system trigger calls with per-call variables. There is also a batch calling feature for calling a list.

In our setup, the owner asks an AI assistant to make a call, the assistant fills in the details (who, why, what to confirm), and the voice agent places the call and sends back a summary. The owner reviews outcomes rather than dialing.

Step 7: Test like a skeptic

Before any real person hears your agent, run through:

  • A cooperative call where everything goes to plan.
  • A confused caller who answers the wrong question.
  • Someone who interrupts constantly.
  • A voicemail greeting and an automated phone menu.
  • A request the agent must refuse, such as a discount or a payment.
  • Bad audio: speakerphone, background noise, a weak signal.

Read every transcript. ElevenLabs keeps a conversation history with a transcript of each call, which makes it straightforward to find the moment a call went wrong and fix the instruction that caused it.

Compliance and consent

Outbound AI calling is regulated. In February 2024 the FCC ruled that AI-generated voices are "artificial" voices under the Telephone Consumer Protection Act, which brings the consent rules for prerecorded calls into play. State laws can add disclosure and consent requirements, and some states require all-party consent to record calls. ElevenLabs' own documentation on batch calling points users to the TCPA and applicable state laws.

Practical rules we follow:

  • Have the agent identify who it is calling for at the start.
  • Have it disclose that it is a virtual assistant when asked, and preferably up front.
  • Keep a record of the consent or business relationship behind each outbound call.
  • Get legal advice before any campaign-style calling.

Inbound calls carry fewer obligations, but recording and disclosure rules still apply.

What it costs

You pay Twilio for the number and call minutes, and ElevenLabs for agent minutes on your plan. Depending on plan and model, language model usage may be included or billed separately. All of these change, so check current pricing from both providers and run a small pilot to see your real per-call cost.

Common mistakes

  • Making the first version do too much. One clear job beats five half-built ones.
  • Skipping the handoff. Every agent needs a way to transfer or take a message.
  • Letting the agent guess. Instruct it to say it does not know rather than invent an answer.
  • Using account-wide Twilio credentials instead of a scoped API key.
  • Not reviewing transcripts after launch. The first two weeks of real calls teach you more than any test plan.

If you want inbound coverage rather than outbound calling, our guide to AI receptionists covers which calls to automate and which to leave with people.

Where we help

We build and maintain AI voice agents end to end: Twilio setup, agent design, tool connections to your calendar, CRM or automations, testing and ongoing tuning. See our AI voice agents service for details.

Frequently asked questions

Do I need a developer to build an AI voice agent with Twilio and ElevenLabs?

Not for a basic agent. Importing a Twilio number, writing the prompt, choosing a voice and testing calls can all be done in the dashboards. You will want technical help for custom tools, such as webhooks into your CRM, and for triggering calls from other systems through the API.

Can I use my existing business phone number?

For outbound calls, you can verify an existing number as a caller ID in Twilio and use it with ElevenLabs. Verified caller IDs cannot receive calls through the agent. For inbound, you need a number that routes to Twilio, either a number purchased there or one connected through SIP trunking.

Can the agent transfer a call to a person?

Yes. ElevenLabs includes a transfer tool that can send the call to another phone number or another agent. Define clear conditions for transfers in the prompt so the agent hands off when a caller asks for a person or the request is out of scope.

How natural does it sound?

With a well-chosen voice and a fast model, conversations sound natural, with short pauses between turns. Latency and interruptions are the main things that make a call feel robotic, so test different models and turn-taking settings on real phone lines.

Is it legal to make outbound calls with an AI voice?

It can be, with the right consent and disclosures. The FCC treats AI-generated voices as artificial voices under the TCPA, and state laws vary. Get legal advice before calling people who have not agreed to hear from you.