What Packet Loss Concealment Synthesizes During Dropouts

By Nadia Ferrante

When an audio packet fails to arrive before its playout deadline, a real-time decoder cannot pause execution. The digital-to-analog converter (DAC) requires a continuous stream of pulse-code modulation (PCM) samples at a fixed clock rate—typically 8 kHz, 16 kHz, or 48 kHz. If the driver runs dry, the hardware buffer underruns, producing an audible electrical transient: a click or pop caused by an instantaneous step change in voltage toward zero.

Packet loss concealment (PLC) exists solely to prevent this step discontinuity. It does not recover lost data. It does not infer what the speaker meant to say. Instead, PLC invents audio out of past decoder state to bridge the temporal gap until the network delivers valid payload again. What it actually invents depends strictly on the domain in which the codec operates: time-domain waveform history, linear predictive filter parameters, or transform-domain spectral bins.

What does 1. Playout Deadlines and the Origin of Concealment actually mean?

Real-time audio transport operates under hard temporal bounds. In systems governed by RFC 3550, packets carry a monotonically increasing sequence number and a timestamp tied directly to sample intervals.

"The RTP timestamp reflects the sampling instant of the first octet in the RTP data packet." — RFC 3550, Section 5.1

The jitter buffer delays incoming packets to absorb transmission variance. If packet $N$ arrives at time $T$, but its playout deadline was $T - 5\text{ ms}$, that packet is logically lost. The jitter buffer must discard it or register a late arrival, and the decoder must execute an erasure frame.

Playout Deadlines and Erasure Flags

The interface between the jitter buffer and the decoder core is binary: the buffer either supplies an array of compressed bytes for packet $N$, or it asserts an erasure condition. In the reference implementation of the Opus codec, standardized in RFC 6716, an erasure is signaled by passing a null pointer or a zero byte-length to the decode function.

At that exact moment, the decoder ceases to be a passive decompressor. It becomes an autonomous synthesizer. It must generate an entire frame's worth of PCM samples—typically 10 ms, 20 ms, or 40 ms—relying entirely on the memory buffers populated by packets $N-1$, $N-2$, and older history.

State Decay and Plausibility Windows

Human speech exhibits short-term stationarity. Over intervals of 10 ms to 30 ms, vocal tract geometry changes slowly. Vowels maintain a relatively stable fundamental frequency ($F_0$) and formant structure.

Concealment algorithms exploit this physical inertia. They operate on the assumption that the unobserved signal is statistically identical to the signal observed 10 ms ago. However, this assumption collapses rapidly. After approximately 40 ms to 60 ms of continuous loss, the probability that the vocal tract has shifted approach certainty. Consequently, all standard PLC algorithms enforce state decay: they systematically attenuate the synthesized signal toward silence, usually cutting amplitude to zero within 60 ms to 100 ms to avoid generating unending, autonomous tones.

What does 2. Core Synthesis Mechanisms by Domain actually mean?

The mathematical machinery used to invent missing frames differs depending on whether the codec operates directly on audio waveforms, on source-filter representations, or in the frequency domain.

Time-Domain Waveform Extrapolation

For raw waveform codecs like G.711 (running at 64 kbps with 8-bit $\mu$-law or A-law PCM at an 8 kHz sampling rate), the decoder retains no internal model of the vocal tract. The standard reference for this architecture is ITU-T Recommendation G.711 Appendix I.

When an erasure occurs in G.711 Appendix I, the algorithm executes the following sequence:

1. Pitch Detection: The decoder takes the preceding 20 ms to 30 ms of decoded PCM history and performs a normalized cross-correlation to estimate the pitch period ($P$). The search bounds are constrained to typical human voice ranges—roughly 66 Hz to 400 Hz (which translates to pitch lags between 20 and 120 samples at 8 kHz). 2. Periodic Replication: Once $P$ is determined, the algorithm extracts a segment of length $P$ from the most recent historical pitch cycle and copies it into the missing frame boundary. 3. Pitch-Synchronous Overlap-Add (PSOLA): Simply pasting a past waveform block induces phase misalignment at the boundary. The algorithm uses a raised-cosine or triangular window to overlap and blend the synthesized period with the edge of the previous frame across a 5 ms transition window. 4. Decay Factor: If subsequent packets are also lost, the cloned pitch period is repeated, but scaled by a multiplicative attenuation coefficient (typically 0.8 per frame). By the third consecutive missing frame (60 ms total), the amplitude is forced to zero.

If the lost segment was an unvoiced fricative (such as an "s" or "f" sound), pitch correlation produces a low coefficient. In those cases, the algorithm skips periodic replication and substitutes attenuated, low-pass-filtered past noise.

Linear Predictive Coding and Filter Extrapolation

Codecs that utilize Linear Predictive Coding (LPC), such as the SILK mode in Opus, model speech as an excitation signal passed through an all-pole autoregressive filter:

$$H(z) = \frac{1}{1 - \sum_{i=1}^{p} a_i z^{-i}}$$

In these architectures, the bitstream does not transmit raw audio samples; it transmits quantized filter coefficients (represented as Line Spectral Pairs or Line Spectral Frequencies) and an encoded residual excitation signal.

When packet $N$ is lost, the LPC decoder invents the signal by separating the vocal tract envelope from the excitation:

1. Envelope Freezing: The decoder holds the autoregressive filter coefficients $a_i$ constant from frame $N-1$. It assumes the vocal tract shape has not moved. 2. Excitation Extrapolation: - If frame $N-1$ was classified as voiced, the decoder measures the pitch lag and repeats the past residual excitation buffer periodically. - If frame $N-1$ was unvoiced, the decoder substitutes a pseudo-random number generator scaled to match the root-mean-square (RMS) energy of the previous frame's residual. 3. Synthesis: The newly invented excitation signal is fed into the frozen LPC filter. The filter provides the spectral shape (formants), while the periodic or noisy excitation drives it.

Because autoregressive filters contain memory (the filter registers retain past output values), running the filter forward on synthetic excitation naturally preserves boundary continuity, eliminating the need for crude time-domain cross-fades. However, the excitation gain must be dampened aggressively. The reference implementation in RFC 6716 scales down the excitation gain over consecutive lost frames, preventing the infinite impulse response (IIR) filter from entering runaway resonance.

Transform-Domain Phase and Energy Extrapolation

Transform-based codecs, including the CELT layer of Opus, divide the audio spectrum into frequency bands using the Modified Discrete Cosine Transform (MDCT), typically operating over frame sizes of 2.5 ms, 5 ms, 10 ms, or 20 ms. CELT normalizes the energy in each band, transmitting the band energies (the spectral envelope) and algebraic vector-quantized pulses representing the normalized spectrum.

When CELT encounters a lost packet, time-domain pitch replication cannot be directly inserted into the MDCT overlap-add buffer without causing time-domain aliasing cancellation (TDAC) failure. Instead, CELT performs loss concealment using an internal pitch predictor:

` [Decoded History Buffer] │ ▼ [Cross-Correlation Lag Estimation] ──► [Frequency-Domain Pitch Filter] │ │ ▼ ▼ [Past MDCT Band Energies] ────────────► [Attenuated Band Synthesis] │ ▼ [Inverse MDCT + Overlap-Add] `

The decoder identifies the pitch lag in the decoded time-domain buffer, extracts that history, and transforms it into the frequency domain. For high-frequency bands where phase alignment cannot be maintained reliably, the decoder preserves the coarse band energy but randomizes the transform bin phases. This converts coherent periodic energy into band-limited noise, preventing the metallic "chirping" artifacts that occur when phase-shifted high frequencies are summed across block boundaries.

3. Pathological Failures Under Jitter and Erasure Bursts

Packet loss concealment handles isolated, uncorrelated packet dropouts with high fidelity. Under severe network impairment, however, the assumptions underpinning PLC fail completely. When jitter increases and packet arrival delays cross the buffer margin, concealment mechanisms degrade speech intelligibility through specific, predictable failure modes.

If the network experiences burst loss exceeding 3 to 4 consecutive frames (60 ms to 80 ms), the decoder exhausts its plausibility window. In an LPC-based system, holding filter coefficients static while driving them with repeated periodic excitation creates an undamped oscillator. The listener hears a distinct, artificial tone at the fundamental frequency—often called a "metallic robot" artifact. To mitigate this, codecs force muting, causing the speech to abruptly drop out.

More damaging is the state divergence that occurs when packets arrive out of order or after a late-burst recovery. Codec decoders maintain internal prediction memory: long-term pitch buffers, LPC filter taps, and spectral envelope smoothers. When packet $N$ is concealed, the decoder advances this internal state using its *invented* data. If packet $N+1$ subsequently arrives intact, its bitstream instructions are decoded against an internal state that no longer matches the encoder's state at the time of transmission.

This state desynchronization produces audible distortion, clicks, or transient smearing that can persist across several subsequent valid frames—long after the network loss event has passed. If high network jitter causes the jitter buffer to continually alternate between underflow (forcing concealment) and sudden frame-skipping (to dump delayed accumulation), the reconstruction engine rapidly alternates between synthetic pitch-looping and realigned real frames. The result is severe comb-filtering and unnatural pitch modulations that destroy intelligibility far more than an absolute, clean mute would.

4. The Concrete Trade-Off in Network-Assisted Recovery

Because PLC algorithms are fundamentally mathematical guesses based on backward history, they cannot reconstruct new phonemes. If a speaker transitions from a vowel to a plosive (such as from "ah" to "t") during an erased 20 ms window, PLC will invent more "ah". The "t" never existed in receiver memory and cannot be synthesized.

To bridge this gap without paying the latency penalty of full TCP-style retransmissions, protocols rely on codec-internal forward error correction (FEC) or redundant payloads.

In the Opus codec, the encoder analyzes the network's reported loss rate (via RTCP Receiver Reports). If packet loss reaches a threshold, such as 5% or 10%, the encoder enables inband FEC. When operating with FEC, packet $N$ carries its own primary payload along with a low-bitrate, heavily quantized description of frame $N-1$.

When the jitter buffer reports packet $N-1$ missing, the decoder does not execute blind PLC. It holds execution until packet $N$ arrives. If packet $N$ is delivered on time, the decoder extracts the FEC payload to reconstruct frame $N-1$. The audio produced is not an extrapolation; it contains the true LPC formants and transient markers transmitted by the sender, albeit at a lower spectral resolution.

The difference in output is absolute: pure PLC generates a mathematical extrapolation of the past, while FEC-assisted decoding delivers the actual observed past at reduced fidelity.

The Practical Rule

Do not attempt to tune a jitter buffer to zero loss if doing so increases end-to-end latency beyond the conversational threshold of 150 ms. Packet loss concealment algorithms in modern codecs are designed to make isolated 10 ms to 20 ms erasures acoustically imperceptible.

Your network budget should allow for a 1% to 3% random packet loss rate without alarming monitoring systems, provided that loss is distributed as single-packet drops. However, you must design your jitter buffer and queue management to prevent *consecutive* packet losses: two consecutive lost 20 ms packets will degrade audio quality noticeably, and three consecutive lost packets will result in audible muting or robotic distortion that no concealment algorithm can disguise.

Terms used in this note

A codec is an algorithm that compresses audio for transmission. A jitter buffer is a queue that absorbs variable arrival times before playback. Packet loss concealment is the substitution a decoder performs when a frame never arrives.

• Sample rate — how many times per second the signal is measured
• Frame size — the block of samples a codec encodes at once
• Lookahead — the delay a codec needs before it can encode
• Bitrate — bits per second the stream consumes
• Jitter — variation in packet arrival time
• Playout delay — total time from capture to speaker
• Concealment — what the decoder invents for a missing frame
• Transcoding — decoding and re-encoding between formats
• Comfort noise — synthetic background added during silence
• RTP — the transport that carries real-time media
• RTCP — the control protocol reporting on an RTP session
• Narrowband — speech-band audio, roughly 300 to 3400 Hz