I Tried Using Gemini to Transcribe the Pronunciation of Names into Katakana Without Interrupting a Twilio ConversationRelay Conversation

Introduction

Twilio ConversationRelay's Japanese speech recognition transcribes names in kanji. For example, "Shoji Akito" may be transcribed as a kanji string, and from that string alone you cannot tell whether the family name is read "Shoji" or "Tokairin", or whether the given name is read "Akito" or "Akihito". In workflows that look up reservations by the reading of a name, the reading is required.

ConversationRelay's settings have no option to make it return transcriptions as readings (katakana). So in this article, I tried a method that keeps the phone reception built with ConversationRelay as it is, receives the audio of just the name part separately and passes it to Gemini, and obtains the reading in katakana.

Test environment

• Twilio ConversationRelay (speech recognition: Deepgram nova-3, language: ja-JP)
• Twilio Media Streams (unidirectional stream)
• Gemini on Vertex AI (gemini-2.5-flash, asia-northeast1)
• A test server running as a single Node.js process (http and ws)

Intended audience

• Those who delegate phone reception to AI and support workflows that match reservations or records by name
• Those building Japanese phone AI with Twilio ConversationRelay

References

• ConversationRelay (Twilio Docs)
• Stream resource (Twilio Docs)
• Understanding audio (Gemini API)

Background

The main speech recognition settings of ConversationRelay are provider, model, language, and hints (as of October 2026). hints make the specified words easier to recognize, but they do not change the notation. It is also impossible to handle this by listing every existing name in hints. Even if you change the model from Deepgram's nova-3 to flux, names still arrive in kanji.

There is also the option of abandoning ConversationRelay and building the entire audio processing yourself, but that would mean substantially rebuilding something that is already working.

So I used Gemini for the transcription. I used Gemini because it can receive audio and instructions together. You can ask it to "write it in katakana exactly as it sounds."

Architecture

The ConversationRelay WebSocket connection is kept as it is, and just before asking for the name, a Media Streams unidirectional stream is started via the REST API. When the caller finishes giving their name, the stream is stopped, the accumulated audio is converted to WAV and sent to Gemini. The returned katakana is sent as text to the ConversationRelay connection that has been connected from the beginning.

The test server is a single always-running Node.js process that keeps accepting WebSockets, with separate endpoints by path.

• /ws: ConversationRelay WebSocket (for the conversation). Stays connected from the start to the end of the call
• /ms: Media Streams WebSocket (for the name audio). Connects each time a stream is started and closes when it is stopped

Implementation

I'll explain only the key points.

Starting and stopping the stream

A stream can be started and stopped during a call with the Streams resource of the REST API. It does not touch the ConversationRelay connection, so the conversation continues as it is.

const res = await fetch(`https://api.twilio.com/2010-04-01/Accounts/${accountSid}/Calls/${callSid}/Streams.json`, {
  method: 'POST',
  headers: { Authorization: `Basic ${auth}`, 'Content-Type': 'application/x-www-form-urlencoded' },
  body: new URLSearchParams({
    Url: `wss://${host}/ms`,
    Track: 'inbound_track',
    'Parameter1.Name': 'k',
    'Parameter1.Value': sharedKey,
    'Parameter2.Name': 'label',
    'Parameter2.Value': streamName,
  }),
});
const { sid } = await res.json();

// Stop it once the caller has finished giving their name
await fetch(`https://api.twilio.com/2010-04-01/Accounts/${accountSid}/Calls/${callSid}/Streams/${sid}.json`, {
  method: 'POST',
  headers: { Authorization: `Basic ${auth}`, 'Content-Type': 'application/x-www-form-urlencoded' },
  body: new URLSearchParams({ Status: 'stopped' }),
});

I set Track to inbound_track. Only the caller's voice arrives, and the AI's speech is not mixed in. In /ms, the media payloads (base64 mu-law) are accumulated, and the audio is finalized on stop.

Don't use name as a custom parameter name

At first, I passed which name was being asked for using a custom parameter called name. Then stream creation failed with HTTP 500 (error code 20500).

To isolate the cause, I tried different combinations of parameters in calls that did not use ConversationRelay.

The cause was not the combination with ConversationRelay but the name of the custom parameter.

Don't add a query string to the stream URL

The server that receives /ms is exposed externally, so you will want to verify that a connection is from Twilio. At first I added a verification value to the URL's query string, but the stream was not established, with errors 31920 / 31922.

Pass the verification value as a custom parameter (k in the code above). The value is included in customParameters of the start event that arrives at /ms, so check it there and close the connection if it doesn't match.

Confirm readiness to receive with start

The time from creating the stream until the first audio (media event) arrived varied. In some calls it arrived about 1.3 seconds after creation, while in others it took around 10 seconds. In the slow calls, it seemed to start arriving at the point when the caller began speaking.

If you wait for the first media before asking for the name, the question can be delayed by nearly 10 seconds. So I made receipt of the start event the signal of readiness, and asked for the name after receiving it. The beginning of the spoken name was never cut off.

A filler phrase to cover the wait

When the final prompt (last=true) of the spoken name arrives, I immediately send "Let me confirm." as text. After that, I stop the stream and call Gemini. While waiting for Gemini, the caller still hears the AI's voice.

// Once the final prompt of the spoken name is received, first read out a single phrase
send({ type: 'text', token: '確認いたします。', last: true, lang: 'ja-JP' });
await stopStream(sid);
const wav = wavFromPcm(mulawToPcm(Buffer.concat(chunks)));
const kana = await geminiKana(wav);
send({ type: 'text', token: `${kana}様でよろしいでしょうか。`, last: true, lang: 'ja-JP' });

Converting the audio to WAV

The Media Streams audio is mu-law, 8kHz, mono. I converted it back to 16-bit PCM and added a 44-byte WAV header.

function mulawToPcm(u8) {
  const out = Buffer.alloc(u8.length * 2);
  for (let i = 0; i < u8.length; i++) {
    const u = ~u8[i] & 0xff;
    const sign = u & 0x80;
    const exponent = (u >> 4) & 0x07;
    const mantissa = u & 0x0f;
    let sample = ((mantissa << 3) + 0x84) << exponent;
    sample -= 0x84;
    out.writeInt16LE(sign ? -sample : sample, i * 2);
  }
  return out;
}

function wavFromPcm(pcm, rate = 8000) {
  const h = Buffer.alloc(44);
  h.write('RIFF', 0);
  h.writeUInt32LE(36 + pcm.length, 4);
  h.write('WAVE', 8);
  h.write('fmt ', 12);
  h.writeUInt32LE(16, 16);
  h.writeUInt16LE(1, 20);
  h.writeUInt16LE(1, 22);
  h.writeUInt32LE(rate, 24);
  h.writeUInt32LE(rate * 2, 28);
  h.writeUInt16LE(2, 32);
  h.writeUInt16LE(16, 34);
  h.write('data', 36);
  h.writeUInt32LE(pcm.length, 40);
  return Buffer.concat([h, pcm]);
}

Request to Gemini

Instead of uploading a file separately, I base64-encoded the WAV and embedded it in the request body. It is sent in the same user turn as the instruction text. A WAV of about 4 seconds of spoken name is about 65KB, which fits within the 20MB limit for inline data.

{
  "contents": [{
    "role": "user",
    "parts": [
      { "inlineData": { "mimeType": "audio/wav", "data": "<WAV の base64>" } },
      { "text": "<下の指示文>" }
    ]
  }],
  "generationConfig": { "temperature": 0, "thinkingConfig": { "thinkingBudget": 0 } }
}

The instruction text is as follows. (※No system prompt was used.)

この音声は、電話でお客様が自分の名前を名乗った部分です。聞こえた音のとおりに、姓と名をカタカナで書き起こしてください。漢字やローマ字に変換しないでください。アルファベットを1文字ずつ読み上げている場合は、その綴りを日本語の読みとしてカタカナにしてください。出力はカタカナの名前だけを1行で、姓と名の間に半角スペースを1つ入れてください。

Results

The same spoken name came out as follows with ConversationRelay and with Gemini. Because I tested with the names of real people, I withhold the names themselves and show only their form.

The ConversationRelay conversation was not interrupted before or after starting and stopping the stream. In calls where the caller spoke quickly, Gemini also sometimes got the reading wrong.

The breakdown of time is as follows.

The roughly 1.3 seconds spent waiting for Gemini could be covered in the background by reading out "Let me confirm."

Discussion

I think this method can be used for utterances where it is known what was asked. Besides names, if you ask for a city name and then instruct "what is being said as a city name", place names could also be transcribed with context. Another advantage is that it can be introduced simply by adding processing before and after the name segment, without touching ConversationRelay's conversation handling.

On the other hand, the accuracy of the readings cannot be guaranteed. Gemini also makes mistakes with rare names or fast speech, so I think it presupposes a flow in which the AI repeats the reading back and the caller confirms it. Also, the roughly 3.2 seconds from the end of speech until the transcription is finalized cannot be shortened.

Another challenge is that even if you build the ConversationRelay endpoint serverless, you need to run a separate server that keeps accepting Media Streams WebSockets, like /ms.

As for cost, I expect the additional cost to be small. About 4 seconds of spoken name is about 130 tokens (32 tokens per second), and even with the instruction text and output it amounts to a few hundred tokens, so the Gemini cost per call would likely be less than 1 yen. Media Streams is also billed per minute, and I think it is small compared with the ConversationRelay call charges, but please check the unit prices on each company's official pricing page before adoption.

Conclusion

With Twilio ConversationRelay's Japanese speech recognition, you cannot control how names are written. So, without interrupting the ConversationRelay conversation, I received only the audio of the segment where the name is asked via Media Streams and passed it to Gemini, and was able to obtain the reading in katakana and return it to the same conversation. I hope this article helps those who handle name readings in phone AI.