Your speech-to-text sends the recording. It only needed the words.

Start with a school and a cupboard full of cassettes. Students have answered questions out loud, a teacher has recorded them, and now somebody has to type out what each student said so it can be marked.

There are two ways to get that typing done.

You can put the cassettes in an envelope and post them to a typing bureau in another city. They are good at typing. They are fast and they are cheap. A week later the typed pages come back, and they are correct. But the bureau has the cassettes. They have a shelf, and the shelf has your students’ voices on it, and what happens to that shelf is now a question about the bureau’s policies rather than yours.

Or you can put a typist in the school office. The cassettes never leave the building. The typed pages come out of the same room they went into. The typist is slower than the bureau and you had to find a desk for them, and that is the whole of the trade.

Both ways give you the same page of text, and the page of text is the only thing the marking ever reads. This piece is about building the second one in software, in a platform where the students are real and the recordings are of children and young adults taking a practice exam.

Two ways to get the same page of text

In a browser, a spoken answer becomes an audio file the moment the student stops talking. From that point it has to reach a model that can read it. There are exactly two places that model can be.

Nobody chooses the first route because they want to share the audio. They choose it because it is four lines of code and the other one is an afternoon of work. So it is worth being precise about what the four lines actually do.

Why everyone calls the API

A hosted speech API is genuinely excellent. It is more accurate than a small model on your own hardware, it handles accents better, it costs a fraction of a cent a minute, and it scales to a thousand students without you thinking about it once. If you are building a voice note feature for a to-do app, call the API. That is the right answer and this piece is not arguing with it.

It stops being the right answer when three things are true at the same time. The audio is of a person who did not choose the vendor. The audio is of them being tested, which means it captures them at their least polished. And the organisation running the test has to be able to answer, in writing, what happens to it.

An exam platform hits all three. A student sits down to practise, speaks for two minutes about their home town, and has no idea that a copy of their voice is now on a server in a country they have never visited, under an agreement they never read. The institute cannot tell them what happens to it either, because the institute does not know.

What the model actually needs

Here is the part that makes the local route obvious once you see it. The model does not want the file the browser produced. It wants something much plainer, and much bigger.

A browser records compressed audio, because compressed audio is small and small is good on a network. The model cannot read compression. It wants raw samples: the height of the sound wave measured sixteen thousand times a second, two bytes per measurement, one channel. Multiply that out for a minute of speech and you get 16,000 × 2 × 60, which is 1,920,000 bytes. Nearly two megabytes for one minute of a person talking.

And what comes back out of the model is a paragraph.

Look at that gap for a moment. The thing you would be uploading is roughly two thousand times larger than the thing you actually wanted, and it is the only one of the three that can be played back to identify a person. The text cannot be played. Nobody can listen to a paragraph and recognise a voice in it.

Which means the audio is not the output. It is scaffolding. It exists for the length of one function call and it has no reason to travel.

The seven steps, in order

This is the path a single spoken answer takes in the platform. Click through the steps rather than reading them as a wall.

The first step is the cheapest and the one most people skip. A browser will sometimes hand over a recording that contains no sound at all, only the wrapper around it. Feed that to ffmpeg and it fails with a parsing error, and you have spent a process spawn and a log line on nothing.

1// A container stub with no audio in it. ffmpeg cannot parse these
2// ("EBML header parsing failed"), so there is no point spawning it.
3const MIN_AUDIO_BYTES = 1024;
4if (audioBuffer.length < MIN_AUDIO_BYTES) {
5  slog("STT", "Skipping empty audio buffer", { bufferBytes: audioBuffer.length });
6  return "";
7}

A thousand and twenty-four bytes is not a clever threshold. It is simply smaller than any real recording and larger than any empty one, and it was picked after watching what actually arrived.

Then the conversion. This is the whole of it:

1// WebM/Opus from the browser → the only shape the model will read.
2await execFileAsync(ffmpegPath, [
3  "-i", inputPath,
4  "-ar", "16000",      // 16,000 samples a second
5  "-ac", "1",          // one channel, not two
6  "-c:a", "pcm_s16le", // plain 16-bit samples, no compression
7  "-y",
8  outputPath,
9]);

Three flags carry all the meaning. -ar 16000 sets the sample rate, -ac 1 collapses stereo to one channel because a microphone in a laptop was never really recording two, and -c:a pcm_s16le says store the raw numbers with no compression at all. The output is the two-megabyte file from the chart above, sitting on local disk, going nowhere.

Then the model reads it, in the same process that wrote the file:

1// Loaded once, kept in memory for the life of the process.
2const { pipeline } = await import("@huggingface/transformers");
3whisperPipeline = await pipeline("automatic-speech-recognition", env.WHISPER_MODEL, {
4  // Use fp32 for CPU compatibility
5  dtype: "fp32",
6});
7
8// And the call itself. No network in this line.
9const result = await whisper(wavPath, {
10  language: "english",
11  return_timestamps: false,
12  chunk_length_s: 30,
13});

Two details worth naming. fp32 means full-precision numbers, which is slower than the half-precision a graphics card would use, and it is chosen because this runs on an ordinary CPU with no graphics card at all. chunk_length_s: 30 means the model reads the audio in thirty-second pieces, which is what lets a two-minute answer work at all rather than running out of memory.

And then the temporary files go, in a finally, so they go even when the step above them threw:

1} catch (err) {
2  slogError("STT", "Transcription failed", err, { provider: env.STT_ROUTING });
3  return ""; // Return empty rather than crashing session
4} finally {
5  try { if (fs.existsSync(webmPath)) fs.unlinkSync(webmPath); } catch { /* ignore */ }
6  try { if (wavPath && fs.existsSync(wavPath)) fs.unlinkSync(wavPath); } catch { /* ignore */ }
7}

Notice the catch. A failed transcription returns an empty string rather than an error. A student in the middle of a spoken exam should not see their session die because a model had a bad minute. We will come back to what happens to that empty string, because it is the reason for one of the least comfortable parts of this design.

There is a switch, and you hold it

Honesty first: this system can also send the audio away. There is one environment variable, and it has two branches.

1// The whole routing decision. Two branches, one env var.
2if (env.STT_ROUTING === "local") {
3  text = await transcribeLocal(wavPath);   // Whisper, in this process
4} else {
5  text = await transcribeViaRemote(wavPath); // Groq or OpenRouter
6}

Set STT_ROUTING to local and the model runs in the API process. Set it to anything else and the WAV goes to Groq or OpenRouter, whichever is currently selected. That branch is real, it works, and it is faster than the local one.

A design that pretended otherwise would be a worse design. The switch exists because an institute running a thousand mocks a week on a small server has a genuine reason to want the remote path, and because a platform with no escape hatch is a platform that fails on the day the CPU cannot keep up.

What matters is where the switch lives. It is a server-side setting in the institute’s own deployment. It is not a default, it is not buried in a vendor’s dashboard, and it does not change by itself when somebody upgrades a library. If the recordings are going out, somebody made that decision and can say so.

What this costs you

Now the honest bill. Running the model yourself is slower, and it is slower in a specific way that will surprise you if nobody says it out loud.

1// One transcription at a time. Every call waits for the last one.
2let transcriptionQueue: Promise<void> = Promise.resolve();
3
4async function runExclusive<T>(task: () => Promise<T>): Promise<T> {
5  const previous = transcriptionQueue;
6  let release: () => void = () => {};
7  transcriptionQueue = new Promise<void>((resolve) => { release = resolve; });
8
9  await previous.catch(() => {});
10  try {
11    return await task();
12  } finally {
13    release();
14  }
15}

That is a queue of one. Every transcription waits for the one before it. It is there because a Whisper model on a CPU will happily use every core it can reach, and four of them running at once do not go four times faster, they go slower and take the rest of the API down with them.

So when twenty students finish a spoken section at the same time, the twentieth transcript is produced after nineteen others. On this platform that is acceptable, and the reason is worth understanding rather than assuming. Transcription does not happen while a student is waiting. It happens after a turn ends, in the background, while the student is already being asked the next question. The number that has to stay small is the time before the student sees a band and written feedback at the end, and that has minutes of room in it, not seconds.

If your product needs live captions on a call, this design is wrong for you and no amount of tuning will fix it. The queue of one is not a limitation to work around. It is the shape of the decision.

The recording is still kept

Here is the part that is easy to oversell and should not be. Running the model locally does not mean the audio disappears. The original recording is uploaded to the institute’s own object storage with a private access rule, and it stays there.

It is kept on purpose, and this is why:

1// Why the original recording is kept. Scoring runs this first.
2const transcript = await transcribeAudio(Buffer.from(bytes));
3if (transcript.trim().length === 0) {
4  // transcribeAudio swallows provider errors and returns "" — still failing,
5  // leave the turn empty; the next scoring attempt / manual rescore retries.
6  continue;
7}

Remember the empty string from earlier. Before a session is scored, the system looks for turns that have a recording but no transcript, fetches the recording back out of storage, and tries again. A student who lost a transcript to a bad minute gets their answer marked anyway, and never finds out anything went wrong. That recovery is only possible because the audio was kept.

So the claim this architecture earns is narrow and worth stating exactly. The recording is stored in storage the institute controls, under an access rule the institute sets, and it is never handed to a third-party model vendor. It is not “the audio is deleted”. Anyone telling you a speech feature keeps no audio at all should be asked how it retries.

Where this breaks

Accuracy. A small model on a CPU is less accurate than the best hosted one, particularly on strong accents, and on an exam platform the accent is the whole point. This is the real cost and it is not fully solved. A larger model is one setting away and considerably slower.

The scoring step still leaves. The transcript is sent to a language model for a band and written feedback. That is a genuine external call and the piece would be dishonest without saying so. What crosses is text, which cannot be played back and does not carry a voice, but it is not nothing.

The first request is slow. The model loads once and is held in memory. The request that triggers the load waits for it. After a deploy, somebody pays that cost.

It consumes the server you already have. Transcription competes with everything else the API is doing. On a small box under real load, that is felt.

ffmpeg is a dependency with opinions. It ships as a binary, it has to be executable in your container, and the day it is not, every spoken answer silently produces an empty transcript.

Check it on your own machine

You do not have to take the size argument on trust. Record anything in a browser, save the file, and run the same conversion this platform runs:

1ffmpeg -i clip.webm -ar 16000 -ac 1 -c:a pcm_s16le clip.wav
2ls -l clip.*

The WAV will be roughly fifteen times the size of the WebM, and it will be about 1.9 megabytes for every minute. That is the file a hosted API receives. Then look at the transcript it gives back and count the characters.

And if you are choosing a speech vendor rather than building one, there is a shorter version of the same test. Ask them one question: after the transcript is returned, is the audio still on your systems, and for how long? A vendor who can answer that in a sentence is fine. A vendor who sends you a policy page is telling you the honest answer without meaning to.

The uncomfortable truth in all of this is that the local route is not better engineering. It is slower, less accurate and more work, and a good engineer looking only at the code would pick the API. It wins on the one axis that does not appear in the code at all, which is that the person whose voice it is never got a say. That is not a technical argument, and it is still the right one.

Related on this site: why an AI should draft but never send, what your Android app actually ships, AI automation and web application development.