Twilio ConversationRelay now supports Deepgram Flux, so I tried testing backchannel interruption and language detection

Introduction

There are two common complaints heard in the field of multilingual phone support and voice AI bots. One is that bots interrupt callers in response to backchannels (words like "yes" or "uh-huh"), making conversations feel unnatural. The other is that callers must select their language before the call begins, with prompts like "For Japanese, press 1; English speakers press 2."

On May 6, 2026, Twilio ConversationRelay announced support for Deepgram's new speech recognition model, Flux. This model integrates speech recognition and turn detection (utterance boundary detection) into a single model, and Twilio claims it reduces response latency by 200 to 600 milliseconds and cuts false interruptions by approximately 30 percent.

This article examines how much these two pain points are actually resolved in real calls, by observing data arriving at a WebSocket server. As a result, language detection between Japanese and English worked stably for normal-length utterances. On the other hand, we were unable to confirm any effect on suppressing backchannel interruptions or improving response speed in this verification.

What is ConversationRelay?

ConversationRelay is a mechanism that connects Twilio voice calls with voice AI (speech recognition, LLM, and speech synthesis) via WebSocket. Speech during a call is converted to text and delivered to your own server, and text you return is synthesized into speech and delivered to the caller.

Verification Environment

• WebSocket server: Plain Node.js + ws + twilio package
• Node.js runtime: v24.18.0
• Deployment target: Vercel Functions
• Phone: US Twilio phone number

Target Audience

• Those building voice AI bots with Twilio ConversationRelay
• Those who want to verify the effects of Deepgram Flux for themselves before adopting it
• Those struggling with false interruptions from backchannels or multilingual support

References

• ConversationRelay now supports Deepgram Flux (Twilio Changelog)
• Deepgram launches Flux Multilingual (Deepgram press release)
• ConversationRelay (TwiML reference)
• WebSocket message reference

Background and Challenges

Support for speech recognition in ConversationRelay has progressed in stages. Deepgram's Nova-3 model became available in September 2025, automatic language detection settings (transcriptionLanguage="multi") were added in October 2025, and in May 2026, support for Flux — which integrates speech recognition and turn detection — was added. According to Deepgram's announcement, Flux Multilingual supports 10 languages including English and Japanese, with turn detection under 400 milliseconds and native support for mid-conversation language switching (code-switching).

Among the messages exchanged over WebSocket, the prompt message sent from the Twilio side contains the transcribed text (voicePrompt) and the detected language (lang). Whether subsequent processing can be branched based on this lang is the key to resolving the language selection challenge.

False interruptions from backchannels significantly affect the perceived quality of voice AI bots. If a bot stops mid-sentence — say, while saying "Thank you for calling us today" — just because the caller says "uh-huh," the conversational tempo breaks down and the experience feels mechanical.

Forcing language selection is another factor that degrades the experience. Even in a multilingual service, making callers press a number to choose their language right after connecting is an extra step they shouldn't need. Automatic language detection would eliminate this friction.

Verification Method

On the TwiML side, we prepared three configurations that differ only in the speechModel setting. The third option, ignoreBackchannel, is an attribute that only takes effect when combined with speechModel="flux". It is a setting that prevents short utterances like backchannels from being treated as a trigger to stop the bot's speech.

The variant is passed as a query parameter in the Voice URL. Since TwiML is finalized at the start of a call and configuration cannot be switched mid-call, we redialed for each of the three configurations when comparing them.

On the server side, received messages such as setup, prompt, interrupt, and text were recorded directly via console.log, and a short confirmation response was returned each time a prompt was received.

function replyFor(lang, voicePrompt) {
  if ((lang || '').startsWith('ja')) {
    return `はい、かしこまりました。今「${voicePrompt}」とおっしゃった内容を、こちらで確認しております。少々そのままでお待ちくださいね。`;
  }
  if ((lang || '').startsWith('en')) {
    return `Sure, I heard you say "${voicePrompt}". I'm just noting that down, so please stay on the line for a moment.`;
  }
  return `了解しました。「${voicePrompt}」について確認しています。少々お待ちください。`;
}

There were two main types of verification scenarios.

1. Scenarios to observe language detection
   Five patterns were tested with the flux configuration: Japanese only, English only, back-and-forth between Japanese and English, short utterances, and mixed-language utterances.
2. Scenarios to observe interruptions and response speed
   Cases where backchannels were spoken during the bot's speech and cases where normal questions were asked were each tested twice across all three configurations.

To avoid the influence of ordering familiarity, the order of configurations was reversed for the second round.

For interruptions, beyond subjective perception, the following four points were individually verified from logs.

1. Whether an interrupt message arrived to stop the bot's speech
2. Whether the backchannel itself was recognized as a prompt
3. Whether an unnecessary response was returned after recognition
4. The caller's subjective experience

For response speed, since server-side processing time from receiving a prompt to sending a response text is nearly 0 milliseconds and is unrelated to the time Twilio spends on speech recognition or synthesis, it was not measured. The caller's subjective experience was treated as primary data.

Verification Results: Language Detection Behavior

The Japanese-only, English-only, and back-and-forth scenarios all performed as expected.

On the other hand, short utterances went unrecognized in all three attempts. Three calls were tested where "もしもし" was followed by "Hello." No prompt messages arrived at all, and the connection dropped approximately 27 seconds later. No error notifications from Twilio were recorded either, and from the server logs alone it was impossible to determine whether recognition failed or whether it was caused by the call state or utterance timing.

Next, the mixed-language utterance "ありがとうございます、Thank you" was tested twice. One attempt went undelivered; the other was received as voicePrompt="ありがとうございます。 Thank you." with lang=ja — recognized as a single Japanese utterance without being split in two.

Verification Results: Interruptions and Response Speed

In scenarios where backchannels were spoken during the bot's speech, behavior differed significantly across the three configurations.

The baseline consistently misidentified the content of backchannels as the English Hi.. Flux correctly identified recognized utterances as Japanese, but there were also calls with no response at all. With flux_ignore_backchannel, interrupt occurred consistently, but whether those utterances led to a response varied by call.

In scenarios where a normal question — "今日の営業時間を教えてください" — was spoken, a simpler pattern emerged.

The flux-based configurations (flux and flux_ignore_backchannel) failed to receive a single prompt across all four attempts with this question. This means that not only short backchannels, but also normal-length utterances, may go unrecognized.

For the two baseline calls that were recognized, responses returned in approximately 2 to 3 seconds by the caller's perception, with the comment that "it would feel more human if it were a bit faster." Since recognition itself failed for the flux-based configurations, no material for comparing response speed was obtained.

Implementation: Branching Logic Based on lang

By reading the lang in the prompt message, you can branch the subsequent response wording or the language used for routing to a live operator. The branching in this verification server was as follows.

if (message.lang === 'ja') {
  // Response or routing for Japanese
} else if (message.lang === 'en') {
  // Response or routing for English
}

The lang value contains only the BCP-47 primary language tag such as ja or en, not a format like en-US. No field indicating confidence in the detection was included in any of the prompt messages received.

Discussion

The cause of recognition failures in the flux-based configurations cannot be identified from server-side logs alone. Determining whether it is due to Deepgram Flux's turn finalization threshold or the impact of audio quality through the telephone network would require additional verification with recorded call audio. Furthermore, since this verification was conducted by a single author with a small number of trials, results could change with repeated attempts.

It should also be noted that the baseline's backchannel misrecognition (はい。 as Hi.) is a problem that can occur regardless of whether Flux is introduced. Automatic language detection accuracy tends to be sensitive to utterance length, and this is likely a persistent issue across all configurations.

Summary

We verified Twilio ConversationRelay's support for Deepgram Flux through actual phone calls. Language detection between Japanese and English worked stably for normal-length utterances. On the other hand, suppression of false interruptions from backchannels and improvements to response speed could not be confirmed in this environment. In fact, there were multiple instances where calls ended without short utterances or normal questions being recognized at all, resulting in overall instability of recognition itself in the flux-based configurations.

If you are considering adopting Deepgram Flux, we recommend testing with the actual utterance patterns and call environment you intend to use. We hope the verification method and results in this article serve as useful reference for your adoption decision.