Why adding a language moved our interpretation further ahead of the video

On a broadcast with five interpreted languages, the fifth language played about 23 seconds ahead of the video. The second was about seven seconds ahead. Viewers heard it live, and the same offsets were there in the replay.

An interpretation has to wait for the speaker to say something before it can translate it. Hearing it that far ahead of the picture meant we'd moved the tracks relative to each other somewhere along the way.

The clue was in the language list. Each successive language played about another five and a half seconds early. We eventually traced that pattern to the order ffmpeg opened its inputs.

One broadcast, several inputs

Our ingest receives the video and each interpreted language as separate RTMP streams. A process we call the mux runs one ffmpeg command to read them all. It produces the video with one audio track per language, which we send to the media server for viewers and also record.

That command has to bring independently arriving streams onto one timeline. With the command we were using, each input's clock started at its first packet. ffmpeg opened the inputs one after another, so the time spent opening one input affected the next.

The language order gave it away

We investigated using August recordings that still had audible room audio. Speech-recognition word timings let us compare what the speaker said with what each track played, and separate the interpretation delay from the offset between tracks.

Our usual order was Arabic, German, French, Hindi, then Chinese. The further down that list a language appeared, the earlier it played. In a session with only French and Chinese, they were one step apart. In the full list, they were two steps apart.

That suggested the position in the input list mattered more than the language.

Every input had the same analysis options:

-analyzeduration 5000000 -probesize 5000000 -i rtmp://.../<stream>

The five-second analysis window was there for the video. We'd given it to the audio-only inputs too. Each audio input spent that time being analysed before ffmpeg moved to the next one.

Because a later input's first sample was placed at zero on its own clock, it played earlier when the tracks were put together. The opening times accounted for the pattern we were seeing: approximately 1.6 + 5.5 × (k − 1) seconds early for language input k.

language  position  predicted offset
ar        1         -1.6 s
de        2         -7.1 s
fr        3         -12.6 s
hi        4         -18.1 s
zh        5         -23.6 s

BeforeAfter

These are offsets introduced by our mux command. They don't mean the interpretation service was translating future speech. We'd put its output ahead of the picture during playback.

We couldn't just shorten every probe

Opening all the inputs quickly was the obvious thing to try. But when we shortened the video's analysis window to one second or less, ffmpeg didn't detect the H.264 stream. We'd lose the video entirely.

We also considered -use_wallclock_as_timestamps or -copyts as ways to align timestamps. That would mean changing more of the command's timestamp handling. We started with a smaller change we could test directly: open the audio inputs quickly, then open the video last with its existing analysis settings.

The audio inputs now use a 0.1-second analysis window. We also specify FLV for these RTMP inputs, which avoids reading the first 2 KB just to identify the format.

Audio first, video last

Previously, the command opened the video first and gave every input a five-second probe:

// Input 0: video stream (contains embedded original audio)
args = append(args,
	"-thread_queue_size", "4096",
	"-analyzeduration", "5000000", // 5s analysis window for jittery inputs
	"-probesize", "5000000",
	"-i", fmt.Sprintf("%s/%s", b.IngestURL, video.StreamName),
)

// Inputs 1..N: audio-only streams
for _, audio := range audios {
	args = append(args,
		"-thread_queue_size", "4096",
		"-analyzeduration", "5000000",
		"-probesize", "5000000",
		"-i", fmt.Sprintf("%s/%s", b.IngestURL, audio.StreamName),
	)
}

The helper supplies the shorter audio probe:

func (b *Builder) interpretationInput(audio registry.StreamInfo) []string {
	args := []string{"-thread_queue_size", "4096"}
	if strings.HasPrefix(b.IngestURL, "rtmp") {
		args = append(args, "-f", "flv")
	}
	return append(args,
		"-analyzeduration", "100000",
		"-probesize", "5000000",
		"-i", fmt.Sprintf("%s/%s", b.IngestURL, audio.StreamName),
	)
}

The output track order and language tags stayed the same. For audio-only broadcasts, we made the equivalent change with the original audio opening last as the reference. A unit test in sync_test.go checks the input order and the short audio probe.

Running both commands on the same streams

We wanted to compare the commands without differences in the test broadcast getting in the way. We ran the old and new commands at the same time, reading the same streams on production ingest. The publisher, both commands and the recorder ran in one task inside our cloud network.

Each track carried a pseudo-random pattern of 250-millisecond tone bursts. Comparing that pattern in each recorded track against the picture side gave us the offsets below. Negative means an audio track plays early; positive means it plays late.

The measured offsets were close to the prediction. After the change, all five tracks were within two seconds of the picture. We deployed it on 6 October. The post-deployment recording had a correlation of 1.00 against the test pattern and no decode errors.

There's still a startup offset

Opening an input still takes about 0.4 seconds. With five interpreted tracks opening before the video, the first can end up around two seconds late relative to it. Reordering the inputs reduced the error substantially, but it didn't give them a shared clock.

We accepted that remaining offset on the late side. Interpretation already arrives after the original speech: in 4,784 utterances from the August recordings, the median was 2.2 seconds, with an interquartile range of 0 to 3.8 seconds. The remaining startup offset adds to that delay; this change doesn't remove it.

There's another consequence. Chat replay uses the moment ffmpeg launched as its timeline origin. With the new order, it runs about 0.4 seconds early per interpretation, or around two seconds for five languages. That's within our few-second tolerance for chat replay. Anything needing closer synchronisation would need a better reference.

We had checked the tracks separately

Each track decoded, played, and contained the right language. Those were the things our checks covered. They didn't compare an interpreted track with the video, so they missed a broadcast in which all the individual tracks worked but the combined result was wrong.

The tone-burst setup was a one-off measurement bench, using our production ffmpeg command and a local RTMP relay. The unit test now protects the input order, but it isn't a replacement for measuring synchronisation in the output.

The August analysis used recordings and speech timings; the controlled test used tone patterns inside our cloud network, without a home internet connection in the path. Broadcast identifiers, stream names, account details and server addresses are omitted here.