Interpretation tracks drifted up to 23 seconds early, and the fix was the order we open inputs

On a Reedapt Live broadcast with AI live interpretation into five languages, the fifth language played about 23 seconds ahead of the picture. The second played about 7 seconds ahead. This held for live viewers and for replays. The cause was the order in which ffmpeg opened its inputs, and the fix was to open the five language inputs first, with a short probe, and the video last. In a controlled test on production ingest, the offsets against the picture went from -2.0, -7.5, -12.9, -18.3 and -23.7 seconds to +1.6, +1.2, +0.8, +0.3 and -0.1 seconds. It was deployed on 6 October. Two seconds of residual error remains by design, and it falls on the side where interpretation is late anyway.

Context

A live broadcast has one video stream and, when interpretation is on, one audio stream per language. Each of these arrives at our ingest as its own RTMP stream. A process we call the mux runs a single ffmpeg command that reads all of them and writes one output with the picture and one audio track per language, which the media server packages for viewers and which we also record.

For ffmpeg to line the tracks up, it has to give them a common timeline. It does that by timestamp, and where an input's timestamps start is where that input's own clock starts.

What we saw

We found it using August recordings that still had audible room audio, and word timings from speech recognition to line up what was said with what each track played. When we corrected the lag for the offset between tracks, the offsets were large and had a pattern. In the usual language order (Arabic, German, French, Hindi, Chinese) the later the language in the list, the earlier it played. The offset depended on a track's position in the list. In a session with only French and Chinese, the two were one step apart, where the full list puts them two places apart.

For a listener this is not subtle. A Chinese listener hears the interpretation roughly 20 seconds before the speaker says the words on screen.

The mechanism

ffmpeg opens its inputs one after another. Each input's clock starts at that input's first packet, so an input that is opened later starts its clock later, and its first sample is placed at zero. Played back together, the later input is shifted earlier by however long the opening of the inputs before it took.

Each input was opened with a 5-second analysis window:

-analyzeduration 5000000 -probesize 5000000 -i rtmp://.../<stream>

A 5 second window is what the video needs. An audio-only input does not need it, but spent it anyway, waiting out the full window before moving on. So input k joined about 1.6 + 5.5 × (k − 1) seconds after the video and played that much ahead of it.

language  position  predicted offset
ar        1         -1.6 s
de        2         -7.1 s
fr        3         -12.6 s
hi        4         -18.1 s
zh        5         -23.6 s

What we tried and why it lost

A shorter probe on the video. If the video were opened fast too, none of the inputs would lose time. It does not work. At 1 second or less ffmpeg does not detect the H.264 stream at all, so the video would be dropped. The 5-second probe on the video is required.

Aligning inputs by arrival time, using -use_wallclock_as_timestamps or -copyts. It would not depend on open order, but it is a bigger change to the whole command. Opening the inputs in the right order was a small change we could test directly, so we started there.

What we built

The interpretations open first, with the format named and a 0.1-second probe. RTMP always carries FLV, so naming it means ffmpeg does not have to read the first 2 KB to guess. The video opens last, with its options unchanged. The output order of tracks and their tags did not change, so nothing downstream had to change.

func (b *Builder) interpretationInput(audio registry.StreamInfo) []string {
	args := []string{"-thread_queue_size", "4096"}
	if strings.HasPrefix(b.IngestURL, "rtmp") {
		args = append(args, "-f", "flv")
	}
	return append(args,
		"-analyzeduration", "100000",
		"-probesize", "5000000",
		"-i", fmt.Sprintf("%s/%s", b.IngestURL, audio.StreamName),
	)
}

Audio-only broadcasts got the same treatment, with the original track as the reference that opens last. A unit test (sync_test.go) pins the order: language inputs first with the short probe, video last.

The numbers

Offset of each interpretation against the picture, in seconds. Negative means the track plays early.

How the "before" and "after" rows were measured matters. We ran the test from inside the same network as the production ingest, with one publisher feeding every stream. The old and the new ffmpeg command ran at the same time on the same streams, so network conditions and start-up timing are the same for both. Each track carried a pseudo-random pattern of 250-millisecond tone bursts. We compared each recorded track with the picture side to find the lag. The correlation was 1.00 on the post-deploy recording and there were no decode errors.

What this costs us

The fix does not make the offsets zero. Opening each input still costs about 0.4 seconds, and an interpretation that opens before the video is shifted by that much per input opened after it. For five languages that is about 2 seconds, and it lands on the late side. We accepted that because interpretation is already late: speech reaches our server about 2.2 seconds after the speaker (median per utterance, interquartile range 0 to 3.8 seconds, 4,784 utterances in August recordings), so an interpretation that is about a second late is not the part of the delay a listener notices.

The timeline we use to line up chat in a replay takes its start from the moment ffmpeg launched. After this change it runs about 0.4 seconds early per interpretation, around 2 seconds for five. Chat replay matches to within a few seconds, so this is inside its tolerance, but nothing that needs better accuracy should use that timeline.

What we got wrong

Our checks looked at each track on its own. Each one decoded, played and was in the right language, and none compared a track with the picture. The bench we built to measure this (the production ffmpeg command, a local RTMP relay and the tone-burst analysis) was a one-off. A unit test now pins the input order, so a reordering fails before it ships.

How we measured, and what we left out

The test publisher, both ffmpeg commands and the recorder ran in one task inside our cloud network, so the home internet connection of whoever ran the test was not in the path. The August figures use the same corrected method on older recordings. Identifiers of broadcasts, accounts and streams are left out of this post, and so are our server addresses.