I Tried Adding a ConversationRelay AI Participant to a Twilio Video Room and Having a Conversation

Introduction

What happens when you add an AI to an online meeting as a participant?

In August 2026, Twilio published a sample architecture that lets a ConversationRelay AI join a Programmable Video Room. In this article, I used this architecture to talk with the AI in Japanese and checked how the experience differs from a one-on-one voice call.

As a result, conversation with the AI worked inside a Room too, and the wait time before a response was about the same as in one-on-one. On the other hand, in a Room, there were cases where the AI answered utterances that were not addressed to it.

What is ConversationRelay?

ConversationRelay is a mechanism that handles speech recognition and speech synthesis in Twilio voice calls, so you can build a voice AI using only text exchange. On a WebSocket server, developers receive the recognized utterances and return response text.

Test environment

• Twilio Programmable Video (Group Room)
• Twilio Video JavaScript SDK 2.36.0
• Twilio Voice JavaScript SDK 2.18.5
• ConversationRelay (both speech recognition and speech synthesis by Google)
• Amazon API Gateway (WebSocket API) and AWS Lambda (Node.js 22)
• Claude Haiku 4.5 on Amazon Bedrock
• macOS and Google Chrome

Target audience

• Those who have experience building a voice AI with ConversationRelay
• Those considering adding an AI to meetings or group conversations

References

• Add an AI Voice Assistant to a Twilio Video Room with Conversation Relay (Twilio Blog)
• Adding Programmable Voice Participants to Video Rooms (Twilio Docs)
• TwiML Voice: <ConversationRelay> (Twilio Docs)

Background and challenges

ConversationRelay is built on the assumption that one person who calls in talks with an AI. Since there is only one other party, every utterance that arrives can be considered addressed to the AI.

Twilio's official material introduces an architecture that connects this ConversationRelay to a Video Room along with the Voice call. In this article, I actually tried it and decided to observe it, focusing on how the conversation experience changes when the same AI is moved from one-on-one to a multi-person conversation setting.

Architecture

The AI participant is built from two call legs (legs) connected by a single Voice call. One leg joins the Room with audio via <Connect><Room>, and the other leg connects to ConversationRelay via a TwiML App. The Room's audio is converted to text by ConversationRelay and delivered to my own server.

Inviting the AI takes just one call to calls.create. The following code is the code used in the test with variable names simplified. twiml is the leg that joins the Room, and the app: in to is the leg that connects to ConversationRelay.

async function inviteAi(room) {
  const twiml = new twilio.twiml.VoiceResponse();
  twiml.connect().room({ participantIdentity: 'ai-assistant' }, room);
  return client.calls.create({ from: aiFromNumber, to: `app:${twimlAppSid}`, twiml: twiml.toString() });
}

The Voice URL of the TwiML App returns the ConversationRelay TwiML. To speak Japanese, I specified the language and voice. interruptible="speech" is a setting that stops the reading aloud when someone speaks while the AI is talking.

<Response>
  <Connect>
    <ConversationRelay url="wss://<WebSocket API の URL>" language="ja-JP" ttsProvider="Google" voice="ja-JP-Chirp3-HD-Kore" transcriptionProvider="Google" interruptible="speech" welcomeGreeting="AIアシスタントです。何かお手伝いできることはありますか。" />
  </Connect>
</Response>

The role of the WebSocket server is the same as in the official article's sample. It passes the received utterances to Bedrock and returns the generated text to ConversationRelay little by little. The only instruction given to the AI was to answer in short spoken language within two sentences.

Test method

From the browser, two participants, a human and a simulated participant, entered the Room, invited the AI participant, and conversed freely. The simulated participant is a participant that plays pre-prepared Japanese audio files with a button.

For comparison, I also called the same server from the browser using the Voice JavaScript SDK and conversed with the AI one-on-one.

Results

One-on-one conversation

The following conversation is an excerpt of the utterances transcribed by ConversationRelay and the AI's responses. For responses that were interrupted midway, the part read aloud until the interruption is shown.

人間: 日本で1番高い山はどこですか？
AI: 日本で一番高い山は富士山で、標高は3776メートルです。(ここで割り込み)
人間: あ、ちょっと待って 天気のいい日におすすめの過ごし方が知りたい。
AI: 天気のいい日は、公園でピクニックをしたり、ハイキングに行ったり、海や川で遊ぶのが (ここで割り込み)
人間: 今日の会議の予定教えて。
AI: 申し訳ありませんが、私はあなたの予定表にアクセスできないので、今日の会議の予定は確認できません。(後略)

When I interrupted, the reading aloud stopped and the AI answered the next question right away. My impression is that the conversation felt natural, no different from the usual experience of a voice AI built with ConversationRelay.

Conversation in a Room

In a Room too, the AI answered the questions of both the human and the simulated participant.

人間: 来週の会議の議題が3つ 考えて。
AI: 来週の会議の議題として、まず上半期の売上目標の進捗状況を確認することはどうでしょうか。次に、新しいプロジェクトのスケジュール調整と、チームメンバーの業務分担の見直しを議題にするのがいいと思います。
(中略)
擬似参加者: あ、アシスタントさん 天気の良い日におすめの過ごし方を1つ教えてください。
AI: 天気がいい日は、外に出て散歩をするのがおすすめです。(後略)

The wait time from the end of an utterance until the AI started speaking was 3.1 to 4.2 seconds. In values measured automatically beforehand as well, no large difference was seen between the wait times in one-on-one and in a Room.

On the other hand, the AI also answered utterances that were not addressed to it. In the following conversation, the simulated participant is speaking to the human.

擬似参加者: ところで、昨日の資料はもう見ましたか？
AI: 申し訳ありませんが、私はAIアシスタントなので資料を見ることはできません。ただ、もし資料の内容について質問があれば (ここで割り込み)
擬似参加者: ところで、昨日の資料はもう見ましたか？
人間: 見ました。
AI: そうですか、ありがとうございます。資料について何か質問や気になる点があれば、いつでも聞いてくださいね。

The last utterance is one where the question replayed by the simulated participant and the human's reply "I did" were joined and reached the AI as a single utterance. The AI participant receives a single stream of audio mixing the voices of all the other participants. For that reason, the AI cannot distinguish speakers.

It is interesting to have the AI participating, but since it answers even when it does not understand if you throw a question at it, I felt it would be difficult to control. In a meeting, it seems likely to prompt "I wasn't asking you, the AI."

Interruption

When interrupted, the AI stops reading aloud, but not immediately. By feel, it stopped after a delay of 1 to 2 seconds. For three interruptions in the Room, I measured in the browser the time from the start of the interrupting utterance until the AI's audio stopped.

In the server logs, for 5 of the 6 interruptions across the Room and one-on-one, the interruption notification (interrupt) arrived 6 to 101 milliseconds before the transcription (prompt) of that utterance. In the third case, the AI continued talking for about 8 seconds over the two people's utterances.

Discussion

If you use an AI in a Room, I think you need to build in a design that makes it judge who an utterance is addressed to, such as instructing the AI to answer only utterances that contain a calling word, or adopting an architecture that can handle each participant's audio separately.

Even so, the experience of having an AI in a multi-person conversation was interesting. For example, if its role is narrowed down, like a meeting-support assistant that answers only when addressed, it seems it could add a new experience to video meetings.

Summary

I put a ConversationRelay AI participant into a Twilio Video Room and conversed in Japanese while comparing with a one-on-one voice call. Conversation worked inside the Room too, and the wait time was about the same as in one-on-one. On the other hand, since the AI receives the voices of all participants without distinguishing them, building the judgment of who is being addressed into the design becomes the challenge when using it in a Room. I hope this article helps as material for decision-making when adding an AI to a multi-person conversation.