An AI phone assistant that makes our calls and emails us what happened

The problem

A lot of the work in running several businesses is small phone calls. Confirm an appointment. Get a quote. Check on an order. Chase a vendor who does not answer email. Each one takes ten minutes and a context switch, and most of them happen during business hours when there is real work on the desk.

We already had AI handling email, documents and research. The phone was the last channel still running through a person.

What we built

An assistant with its own phone number that Claude can send out on an errand, the same way you would hand a task to an office manager.

  • A Twilio number connected to an ElevenLabs voice agent, so the assistant can call out and take calls in.
  • An n8n workflow that Claude triggers. Claude passes the objective, who to call, the number and any details the assistant needs, and n8n starts the call.
  • A system prompt written like a job description: who the assistant works for, what it can do (scheduling, confirmations, quotes, status checks, messages) and what it must never do.
  • Structured results on every call: whether the person was reached, whether the objective was met, the outcome in a sentence and any follow-ups.
  • An email to us as soon as each call ends with that summary, so nothing has to be checked in a dashboard.

The rules that mattered more than the voice

The assistant gives out no information about the person it works for. No schedule, no address, no personal details. It was never given a calendar, because a calendar it cannot keep current is worse than none.

If it reaches voicemail, it leaves a short message and asks people to call the assistant back, not us. When they do, it handles the callback and we get the same email summary.

Tuning it until people stop noticing

The first version worked and sounded wrong. Most of the effort after launch went into making the calls feel normal, one change at a time, each tested on a copy of the agent before the live one changed.

  • The first voice was too expressive, then too slow. We moved to a calm, natural pace and a custom-designed voice with a slight Southern accent.
  • Long pauses before answers made people say "hello?" mid-call. A faster model fixed most of it. We tried quicker turn-taking, then set it back to normal because it started cutting people off.
  • It opened by announcing it was an AI. We dropped that from the introduction. It still answers honestly if asked.
  • On outbound calls it started talking the instant the line connected. Now it waits for the other person to say hello and only speaks first if they do not.
  • We added faint office background sound, turned it down, then removed it. Clean audio sounded better.
  • It laughed too easily. Now it laughs only when someone actually makes a joke.
  • In November we moved it to ElevenLabs' newest realtime model with voicemail detection, after a side-by-side test on a copy.

What changed

Since it went live in October the assistant has placed and answered more than 50 calls, including the test calls we used for tuning. Errands that used to wait for a free ten minutes now go out when we think of them, and the result lands in the inbox as a short summary with any follow-ups listed.

What we would do again

Our take: write the rules before you pick the voice. What the assistant must never say matters more than how it sounds.

Give every call a structured result. "Reached, objective met, outcome, follow-ups" turns a recording nobody listens to into something you can act on.

Tune on a copy. Small changes to voice and timing change how people react, and you want to hear that on a test call, not a vendor's.

The technical setup, from the Twilio number to the ElevenLabs agent, is in our guide to building an AI voice agent with Twilio and ElevenLabs.