Our interpretation was 18 seconds behind, but our logs said seven

During an English-to-Spanish test on 16 October, the Spanish interpretation settled about 18 to 20 seconds behind the English audio. Our logs told a different story:

target es-ES is 6.9s behind, skipping a 3.3s sentence

We were watching that number to understand how the interpreter was doing. It left out the time spent waiting for a sentence to finish, so it understated what a listener heard by about eleven seconds.

Once we compared the actual broadcast tracks, we had two jobs: reduce the delay and fix the number we were using to track it.

We shipped changes on 17 October. In a follow-up production test on 5 November, Spanish started about 6.5 seconds behind on individual sentences and finished 8.4 seconds behind on continuous speech. That's roughly ten seconds better than the first test, though the tests used different speech and measurement methods. It isn't an exact before-and-after benchmark.

What we were timing

Our interpreter listens to a broadcast, recognises speech, translates it, synthesises the translated voice, and publishes that voice as another audio track. Outside AI services handle recognition, translation and voice synthesis. Our engine decides how to divide the incoming speech, runs those steps, and plays the results out in order.

For this investigation, we compared the original and interpreted tracks available to the viewer's player. That tells us how far Spanish is behind English in the broadcast. It doesn't tell us how far either track is behind the person speaking in the room. Delivery delay shared by both tracks cancels out in the comparison.

The first sentence wasn't the whole story

The October test ran for about three minutes on an unlisted production broadcast from our own account. We looped a 33-second recording of continuous English. With no natural pauses, it gave the interpreter little opportunity to catch up.

Afterwards, we fetched both HLS audio renditions and transcribed them locally with an open-source speech model. Their starting presentation timestamps were 0.000 and -0.021 seconds, close enough to compare the sentence timings directly.

Spanish was about eight seconds behind on the first sentence. Then the delay grew to 18 to 20 seconds and stayed there.

One passage took 12.2 seconds to say in English and 18.4 seconds in Spanish. At that pace, translated audio would keep accumulating while the English continued. Our engine already sped the voice up when audio was waiting, up to 1.4 times its normal speed. In steady state, it produced 31.9 seconds of Spanish for each 33.1-second English loop. That stopped the backlog growing, but it didn't clear the delay we'd already accumulated.

We also checked a few possible explanations. CPU wasn't busy: the task had two vCPUs and used about two percent. The initial five-second burst of input was absorbed by our pacing. And the interpreter read directly from ingest over RTMP, so it wasn't waiting for the viewer's HLS playback to deliver its input. Our muxer's track alignment contributed at most 1.6 seconds in this investigation, as covered in the input-order post.

A relay was adding 4.8 seconds

Each interpreted language left the engine over SRT and passed through a relay before reaching ingest. That relay also receives streams from venues with unreliable uplinks, so its buffering was set for those conditions.

SRT negotiates latency using the larger setting at either end. Our interpreter requested one second. The relay's listener requested four, so the connection used four. The relay also probed each stream for two seconds before forwarding it.

We measured the route by passing music through unchanged and comparing the original and Spanish tracks. Cross-correlation gave a match of 1.00 and a delay of 4.8 seconds. The configured four-second buffer and two-second probe don't simply add up to that measured value; we didn't isolate their individual contributions.

The interpreter was publishing from inside our own network. It didn't need the same route as a venue uplink. We changed it to publish directly to ingest over RTMP and kept the old SRT route available behind a setting.

// outputURL is where the engine publishes a language. Straight to ingest's
// RTMP server by default: over SRT, ingest's receive buffer (sized for venue
// uplinks) and the relay behind it put every sentence about 5 s later.
func (j *job) outputURL(t Target) string {
	if j.m.cfg.Output == "srt" {
		return fmt.Sprintf("srt://%s:%s?streamid=%s,key=%s&latency=%s", /* ... */)
	}
	return fmt.Sprintf("rtmp://%s:%s/%s/%s?key=%s", /* ... */)
}

Lowering the relay's buffer for everyone would also affect the venue streams it was configured to receive. Bypassing it for the interpreter let us leave those publishers alone.

Short speech chunks were cutting words in half

The engine used to cut incoming speech every 3.5 seconds, regardless of what the speaker was saying. In the test sample, 48 of 54 pieces hit that forced cut, about 89 percent. Some cuts landed mid-word.

This affected meaning as well as timing. In the test, “welcome anyone who is new” became “¿Quién es nuevo?”, and “we will read together” became “Leímos”. Feeding recognition and translation fragments of a sentence was losing information before we even voiced the result.

We changed where the cuts happen. After three seconds, a piece can end at the first pause lasting 140 milliseconds. If speech continues to 4.5 seconds, we cut at the quietest 60-millisecond window in the last second and carry the remainder into the next piece.

That still puts a limit on waiting. It also gives the recogniser a better chance of receiving whole words. We kept sentence-level translation rather than trying to reduce latency with even shorter fixed cuts.

In a local run on 198 seconds of dense speech, the engine's median delay fell from 8.7 to 6.6 seconds. Its p90 fell from 10.2 to 7.6 seconds, and its maximum from 11.3 to 7.8 seconds. On a three-minute sermon with natural pauses, the median didn't improve: it went from 6.1 to 6.4 seconds. The number of forced mid-speech cuts in that sample did fall, from 24 to two.

A few smaller changes mattered too

We run two recognisers concurrently. A short piece could finish before the longer piece preceding it, then get translated and voiced first. In one case, “Come near.” played before the sentence it was supposed to finish. We added an ordering stage that holds completed pieces until the earlier ones are ready.

We also made pacing respond sooner. Previously, five seconds of queued audio was needed to reach the maximum 1.4-times voice speed. We lowered that threshold to 2.5 seconds. The code comment attributes about a second of improvement on continuous Spanish speech to this change; we haven't measured it separately here.

Finally, we changed the logs. Each language now reports once a minute how far behind the speaker its sentences went out, including the wait the old metric omitted. It records the median, the longest delay, and how many sentences were skipped.

Listening to the output again

On 5 November, we started another unlisted broadcast through the normal creator flow, streaming from software with interpretation already enabled.

This time the input was synthesised English: six individual sentences, each followed by about eighteen seconds of silence, then roughly two and a half minutes of continuous speech. We recorded the original and Spanish tracks from the same HLS stream viewers receive.

Rather than transcribing and matching words across languages, we timed when sound started and stopped in each track. We corrected their 0.02-second starting offset and used 50-millisecond analysis windows.

Here are two excerpts, each taken from the same moments of the two tracks. English is in the left channel and Spanish in the right. Headphones make it easier to hear the gap. Both come from that production test; the English source is synthetic and the Spanish is our AI interpretation.

The median starting delay for the individual sentences was 6.5 seconds. Nothing was skipped in this test. Timing sound onset and ending doesn't tell us the delay throughout the middle of continuous speech; for that portion, the engine's per-minute medians were 8.1 to 9.9 seconds.

The logs were much closer to the output this time. For individual sentences, they reported medians of 5.2, 5.5 and 5.8 seconds, while the recorded tracks showed starting delays of 5.7 to 6.7 seconds. For continuous speech, the reported medians were 9.9, 8.1 and 8.5 seconds, with a longest delay of 10.4 seconds and nothing skipped. Spanish finished 8.4 seconds after English in the recording.

These are different measurements, so they won't match exactly. But the metric we use to monitor the service was now much closer to what a listener heard.

Keeping the measurements separate

We collected several kinds of timing during this work. They're useful for different questions, and mixing them would make the result look more precise than it is.

We also had 71 one-minute production reports from 17 October through 1 November, almost all from internal tests. The median of the Spanish minute-medians was 7.2 seconds across 31 reports; French was 6.5 seconds across 24. Chinese, Japanese, Italian and German each had four reports, with figures between 4.7 and 5.8 seconds. Six of 464 Spanish sentences were skipped, about 1.3 percent.

Those are engine reports. We only measured Spanish against English at the viewer end, so we can't use them to claim the same viewer-side result for every language.

What a listener still gives up

The voice can run at up to 1.4 times normal speed when audio is queued. That's audible. The engine can also skip a whole sentence if playing it would leave the language more than ten seconds behind.

In a three-minute October production test, one 6.1-second sentence was skipped when the reported delay was 4.2 seconds. Those values describe the sentence length and the delay at that point, rather than a rule that skipping only begins once the current logged delay exceeds ten seconds.

The direct RTMP route uses TCP rather than SRT's loss recovery, so we keep it inside our own network. And the local sermon test is worth repeating: its median latency didn't improve, even though the cuts were cleaner. Most of the timing benefit in our engine tests came on dense speech.

Continuous speech is still the case we're trying to improve. Translation needs enough speech to work with, and a longer spoken translation can create a queue even after the processing is done. The November test put that much closer to the individual-sentence result, but there's still a gap.

We should have compared the broadcast tracks earlier. Our internal delay number was convenient, and it described less of the listener's experience than we thought. Recording both tracks gave us a way to check the metric as well as the changes.

The October viewer measurements used local transcription; November used sound timing. Local engine comparisons ran on the same machine, and the later production figures came from the per-minute reports. None of these track-to-track measurements includes delivery delay shared by the original and interpreted audio. Stream keys, internal addresses, account details and broadcast identifiers are omitted. The test broadcasts and their files were deleted after the tests.