How to Add Voice Input to an AI-Built App

To add voice input to an AI-built app you need four things: a way to capture microphone audio in the browser, a decision about whether to transcribe in batches or in a live stream, a speech-to-text provider, and a plan for what happens when the transcript is wrong. The capture is the easy part and takes about twenty lines. The other three are where voice features quietly fail.

This walks through all four, in the order you will hit them.

Decide batch or streaming before you write anything

This single choice determines your architecture, your cost, and most of your UI work. Get it wrong and you rebuild.

**Batch** means the user records, stops, and then you send the whole clip for transcription. One request, one response. The user waits a second or two after they finish speaking.

**Streaming** means audio flows to the provider while the user is still talking, and partial transcripts come back live. The words appear as they speak.

The accuracy gap is real and providers publish it. Google, for instance, reports 2.6% average word error rate for pre-recorded audio against 4.0% for live streams on its newest speech model, a difference we broke down in our look at Gemini 3.5 Transcribe.

Default to batch. Choose streaming only if the user is dictating something long enough that watching a blank box for three seconds would feel broken, or if you need to interrupt them mid-sentence. Most forms, search boxes, and note fields are batch.

Capture the audio

Browsers give you this natively through the MediaStream Recording API. No library required.

async function record() {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
  const rec = new MediaRecorder(stream, { mimeType: 'audio/webm;codecs=opus' })
  const chunks = []

  rec.ondataavailable = (e) => chunks.push(e.data)
  rec.onstop = async () => {
    const blob = new Blob(chunks, { type: 'audio/webm' })
    stream.getTracks().forEach((t) => t.stop())   // releases the mic indicator
    await sendForTranscription(blob)
  }

  rec.start()
  return rec   // caller holds this to call rec.stop()
}

Three details in there that people miss.

`stream.getTracks().forEach(t => t.stop())` is not optional. Without it the browser keeps the recording indicator lit after the user has finished, which reads as "this site is still listening to me" and is the fastest way to lose someone's trust.

The `mimeType` needs a fallback. Safari has historically not supported `audio/webm`, and if you pass an unsupported type the constructor throws. Feature-detect with `MediaRecorder.isTypeSupported()` and fall back to `audio/mp4`.

`getUserMedia` requires a secure context. It works on `localhost` and on HTTPS, and nowhere else. If your staging environment is plain HTTP, voice input will appear broken there and fine in production, which wastes a day.

Handle the permission prompt like a real state

The microphone permission has three states and your UI needs all three: not yet asked, granted, denied. Most implementations handle the first two and leave the third as a button that silently does nothing.

Denied is permanent from your code's perspective. You cannot re-prompt. The only fix is the user changing it in browser settings, so the honest UI is a message saying exactly that, ideally naming the browser.

const status = await navigator.permissions.query({ name: 'microphone' })
// status.state is 'granted' | 'denied' | 'prompt'

Firefox does not support querying `microphone` this way, so wrap it in a try/catch and treat a throw as `prompt`. Ask for permission on a click, never on page load. A site that asks for your microphone before you have done anything gets denied, and that denial is sticky.

Send it to a provider

Do not call the speech API from the browser. That would put your provider key in client-side JavaScript where anyone can read it. Post the audio to your own endpoint and let the server hold the key.

async function sendForTranscription(blob) {
  const form = new FormData()
  form.append('audio', blob, 'clip.webm')
  const res = await fetch('/api/transcribe', { method: 'POST', body: form })
  if (!res.ok) throw new Error(`transcribe failed: ${res.status}`)
  const { text } = await res.json()
  return text
}

On the server side, three guards before you forward anything:

1. **Cap the duration.** Audio is billed per second and a stuck recorder can send a very long clip. Reject anything over your longest legitimate use case, which for a form field is usually 60 seconds.

2. **Cap the file size independently.** Duration and size are not the same check, and a corrupt file can be large without being long.

3. **Rate limit per user.** Voice endpoints are unusually expensive per call, so they are worth protecting properly. The same approach as adding rate limiting to an AI-built app applies here.

If you already handle uploads, the storage and validation path is the same one described in adding file uploads to an AI-built app, and you can usually reuse it rather than building a second pipeline.

Design for the transcript being wrong

This is the part that gets skipped, and it is the part users notice.

Speech-to-text is not a solved problem. It is very good on clear speech in a common language and noticeably worse on names, jargon, accents outside the training distribution, and anything said in a room with background noise. Your app will get transcripts that are almost right.

Four things that make almost-right acceptable:

• **Always show the transcript before acting on it.** Never send an email, book a slot, or submit a form directly from voice. Put the text in an editable field first. This one rule prevents most voice-feature complaints.

• **Make editing the obvious next step, not a recovery path.** The field should already have focus with the cursor at the end.

• **Give the user a re-record button that does not lose what they had.** Failed re-records that wipe the first attempt are infuriating.

• **Feed domain vocabulary to the provider if it supports it.** Most speech APIs accept a hint list of terms. Your product names, your customers' industry jargon, and any acronym you use internally belong in that list.

If the transcript then feeds a language model, remember it is user input from an unreliable channel and deserves the same scepticism as any other. Our notes on telling when an AI answer is hallucinated apply doubly when the model's input was itself a guess.

Cost, roughly

Speech-to-text is billed per unit of audio, typically per second or per minute, and it is cheap enough that most people never think about it until they have a stuck client sending silence in a loop.

The maths worth doing once: take your expected clips per user per day, multiply by average length, multiply by your provider's per-minute rate, multiply by your user count. If that number is small, stop thinking about it. If it is not, the fix is almost always a client-side silence check that refuses to send a clip with no speech in it, which costs you nothing and kills the most common waste.

A sensible order to add voice input

1. Capture and play back locally. No provider, no network. Prove the microphone works and the indicator turns off.

2. Add the server endpoint with a hardcoded fake transcript. Prove the round trip.

3. Wire the real provider. Now you are testing accuracy, not plumbing.

4. Add the editable-transcript UI.

5. Add limits and rate limiting.

6. Only now consider streaming, if the batch version genuinely feels slow to real users.

Doing it in this order means every step has one thing that can be wrong. Doing it all at once means a silent failure could be permissions, codec, network, key, or provider, and you will spend the afternoon finding out which.

The general pattern here, building the boring version first and adding the impressive version only when the boring one proves insufficient, is the same one that makes building an app with AI work at all.

FAQ

Can I use the browser's built-in speech recognition instead?

The Web Speech API exists and is free, but support is inconsistent, several implementations send audio to the browser vendor's servers anyway, and you get no control over the model or vocabulary. It is fine for a prototype and a poor foundation for a product.

Do I need WebSockets for voice input?

Only for streaming transcription. Batch transcription is an ordinary POST with an audio file, which is why it is the right default for almost every form field and search box.

How do I handle voice input on mobile Safari?

Feature-detect the recording MIME type and fall back to `audio/mp4`, request permission on an explicit tap, and test on a real device. Mobile Safari has historically been the strictest environment for microphone access, and desktop testing will not surface its quirks.

What audio quality do I need?

16 kHz mono is enough for speech recognition and is what most providers downsample to anyway. Recording at higher rates makes your uploads bigger without making the transcript better.