I tried building a minimal setup for AI Japanese conversation over the phone using Twilio and OpenAI Realtime API

Introduction

When building phone AI with Twilio in the past, the standard approach was to manually chain together a multi-stage pipeline: stream audio received via Media Streams to a cloud speech recognition service, feed the results to an LLM, then convert the response text back to audio via TTS and return it. Since latency accumulates at each stage, considerable effort went into selecting and tuning the services in between to achieve a natural conversational tempo.

What changed this dramatically was OpenAI's gpt-realtime model, which went GA in August 2025. It provides a single-pass interface where you stream audio directly over WebSocket and receive audio responses back, with VAD (end-of-speech detection) and interruption handling managed server-side.

This time, I built a minimal configuration combining gpt-realtime with Twilio MediaStreams to allow callers to an American phone number to have a two-way Japanese conversation with an AI. The goal is to achieve a state where "a caller to a US Twilio number can have a two-way voice conversation in Japanese with an AI." I chose Fly.io for hosting, as it works well with WebSocket.

What is Twilio Media Streams

Twilio Media Streams is a Twilio Voice feature that enables real-time sending and receiving of call audio over WebSocket. By default, G.711 μ-law (8 kHz) base64 payloads are streamed, with <Connect><Stream> in TwiML for bidirectional operation and <Start><Stream> for receive-only operation.

Verification Environment

• Runtime: Node.js 24.x
• Language: TypeScript 5.x
• Framework: Fastify 5.x + @fastify/websocket
• OpenAI model: gpt-realtime (GA August 2025)
• Phone: Twilio Voice + Media Streams (US number)
• Hosting: Fly.io (iad region / shared-cpu-1x / 256 MB)
• Verification date: May 2026

Target Audience

• Those who are building or considering building phone AI / IVR with Twilio
• Those looking for samples that follow the OpenAI Realtime API GA specification (gpt-realtime)
• Those unsure about where to host a WebSocket relay server
• Those who want concrete numbers on response latency and actual costs for phone AI

References

• Realtime conversations | OpenAI
• Realtime API over WebSocket | OpenAI
• Media Streams | Twilio
• TwiML Voice: <Stream> | Twilio
• Fly.io Docs

Why a Relay Server is Needed

Twilio MediaStreams cannot be connected directly to the OpenAI Realtime API. There are three reasons.

1. Cannot inject authentication headers
   Connecting to the OpenAI Realtime API requires the Authorization: Bearer <api_key> header, but WebSocket connections established from Twilio cannot have arbitrary headers added.
2. Event schemas are different
   The Twilio side uses the format {event: "media", media: {payload: ...}}, while the OpenAI side uses {type: "input_audio_buffer.append", audio: ...} — structures unknown to each other.
3. Session initialization is required
   Right after connecting, a session.update must be sent to configure audio format and other settings, which Twilio knows nothing about.

Therefore, at a minimum, a WebSocket relay that simply converts between Twilio and OpenAI event formats must be hosted somewhere. This time I wrote it as a Node.js process on Fly.io.

The audio format is aligned to audio/pcmu (G.711 μ-law / 8 kHz) on both Twilio and OpenAI sides. This means the relay side performs absolutely no binary conversion — it simply passes the base64 payload arriving from Twilio directly to OpenAI, and passes audio chunks arriving from OpenAI (response.output_audio.delta) directly to Twilio.

Prerequisites and Deploying to Fly.io

First, here is what you need to prepare.

• A purchased Twilio number
• An OpenAI API key and credits
• A Fly.io account and the flyctl CLI

The reason I chose Fly.io for hosting is that it can be used as-is for use cases that maintain WebSocket connections for extended periods.

The main parts of fly.toml are as follows.

primary_region = "iad"

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = "off"
  auto_start_machines = true
  min_machines_running = 1

[[vm]]
  size = "shared-cpu-1x"
  memory = "256mb"

Deployment is complete after creating the app with flyctl launch --no-deploy, injecting OPENAI_API_KEY / TWILIO_AUTH_TOKEN / PUBLIC_BASE_URL with flyctl secrets set, then running flyctl deploy. Finally, switch the Voice Configuration for the target number in the Twilio Console to Webhook and set the URL to https://<your-app-name>.fly.dev/twilio/voice to complete the wiring.

Implementation Excerpts

Focusing on the key points to understand in the GA specification, here are excerpts from 4 locations.

1. Having TwiML Open a WebSocket

Return TwiML instructing Twilio to "stream audio bidirectionally to this WebSocket." The only thing to note is using <Connect><Stream> (bidirectional) rather than <Start><Stream> (unidirectional).

app.post("/twilio/voice", async (request, reply) => {
  const wsUrl = `${config.PUBLIC_BASE_URL.replace(/^https/, "wss")}/twilio/stream`;

  const response = new twilio.twiml.VoiceResponse();
  response.connect().stream({ url: wsUrl });

  return reply
    .header("Content-Type", "text/xml; charset=utf-8")
    .send(response.toString());
});

2. Connecting via WebSocket to OpenAI Realtime

The Authorization header must always be sent. Since browser WebSocket APIs cannot send headers, the premise is a configuration where the server side holds this connection in both production and verification environments.

const url = `wss://api.openai.com/v1/realtime?model=${encodeURIComponent(model)}`;
this.ws = new WebSocket(url, {
  headers: {
    Authorization: `Bearer ${apiKey}`,
  },
});

3. session.update Payload

This section will not work if written with the mindset of the old beta.

{
  type: "session.update",
  session: {
    type: "realtime",
    instructions: "You are a verification AI assistant. Please respond concisely in Japanese.",
    output_modalities: ["audio"],
    audio: {
      input: {
        format: { type: "audio/pcmu" },
        transcription: { model: "whisper-1" },
        turn_detection: { type: "server_vad" },
      },
      output: {
        format: { type: "audio/pcmu" },
        voice: "alloy",
      },
    },
  },
}

• The old beta's input_audio_format: "g711_ulaw" cannot be used in GA. It has changed to the nested format audio.input.format: { type: "audio/pcmu" }.
• audio/pcmu is fixed at 8 kHz, so adding rate: 8000 with a PCM mindset will be rejected with Unknown parameter: 'session.audio.input.format.rate'.
• Without explicitly specifying output_modalities: ["audio"], the AI may return only text responses, causing a situation where no audio reaches the phone side.
• The output audio event name has also changed from response.audio.delta → response.output_audio.delta (the relay described later accepts both as a precaution).
• Adding transcription: { model: "whisper-1" } automatically transcribes the user's speech to text, delivered via the conversation.item.input_audio_transcription.completed event. This does not affect the AI's response content and is included for observation purposes.

4. The Bidirectional Audio Relay Core

Thanks to aligning on audio/pcmu, this can be written concisely. It simply transfers the base64 payload as-is.

// Twilio → OpenAI
case "media":
  this.openai?.send({
    type: "input_audio_buffer.append",
    audio: parsed.media.payload,
  });
  break;

// OpenAI → Twilio (compatible with both GA / beta)
case "response.output_audio.delta":
case "response.audio.delta":
  this.twilioWs.send(JSON.stringify({
    event: "media",
    streamSid: this.streamSid,
    media: { payload: event.delta },
  }));
  break;

Adding barge-in processing that detects when the user starts speaking and cancels the AI-side response (response.cancel + sending a clear message to Twilio) to this is essentially the entirety of the relay.

Setting Up the Twilio Webhook and Making an Actual Call

Once configuration is complete, call the target number and try having a conversation. This time I built it to output structured logs to stdout, so I was able to fully reconstruct the call content after the fact using flyctl logs. The transcript from one actual call (approximately 78 seconds) is as follows.

Subjectively, it felt natural enough that "if you didn't know you were talking to an AI, you'd have to listen carefully to notice."

Latency

The time elapsed from the end of user speech (input_audio_buffer.speech_stopped) to when the AI begins speaking (first arrival of response.output_audio.delta), extracted from flyctl logs.

Since natural back-and-forth between humans is said to be on the order of several hundred milliseconds, this is within a range that produces no perceptible awkwardness.

Observations from Running It

Although this was a brief verification, here are some things I was able to observe.

• Audio frames may drop slightly right after a call begins
  Since Twilio starting to stream audio after receiving a call and the server finishing its WebSocket connection to the OpenAI Realtime API happen in parallel, if the Fly.io side is in a cold start state, the first few hundred milliseconds of Twilio frames may be discarded. Testing with a hot machine state reduced this to zero. In today's flow where "the AI speaks first right after the call is answered," there was no practical impact, but for use cases where you need to capture user audio in the first few hundred milliseconds, workarounds like setting min_machines_running = 1 to ensure a hot state may be needed.
• Barge-in is handled automatically by server VAD, making double-firing with custom implementations easy
  In server_vad mode, AI responses are automatically canceled the moment user speech is detected by default. Not knowing this and naively sending response.cancel from the client side caused the error response_cancel_not_active (no cancellation target) to flood the logs every time. This time I added a guard to "only cancel when the AI is currently responding."
• Hallucinations happen normally
  As the transcript above shows, the AI gave an immediate answer with weather forecast numbers, but this implementation passes no function calling or external tools whatsoever, so these values are entirely constructed within the model. Since communication verification was the main purpose this time, I won't go deeper here, but when putting this into production, it will be a prerequisite to build in a design of "don't answer what can't be answered" and real data references via function calling.

Cost

The actual OpenAI Realtime API consumption for the 2 verification calls in this session (approximately 160 seconds total) was as follows. Actual amounts were retrieved using the Admin Usage API.

Approximately $0.10 (about ¥15) per call, roughly ¥11 per minute. Output audio tokens ($64/1 million tokens) are dominant, with greater impact the longer the AI speaks. On the Twilio side, it's approximately ¥2 per minute (calls $0.0085/min + MediaStreams $0.004/min), and Fly.io is negligible if only running during testing, so even combined, phone AI works out to roughly ¥13 per minute.

Summary

Simply bridging Twilio MediaStreams and the OpenAI Realtime API with a thin WebSocket relay, AI phone conversation was achievable with a surprisingly small amount of code. Response latency stays within a few hundred milliseconds, with virtually no perceptible awkwardness as natural conversation. Costs are on the order of a few dozen yen per call, making it easy to try out during the PoC and prototyping stage. Please try it with your own number as an option that lowers the implementation barrier for phone AI by one notch.