How LTX-2.3 Actually Works

Two streams, one diffusion process

LTX-2.3 is a DiT-based audio-video foundation model. It generates synchronized video and audio in a single joint diffusion process. Not picture first and sound afterwards — both at once, inside the same denoising loop.

The paper describes the shape: an asymmetric dual-stream transformer, a 14B-parameter video stream alongside a 5B-parameter audio stream. Those figures describe LTX-2, which totals 19B. The LTX-2.3 checkpoints are 22B. The two streams meet in 48 shared transformer blocks carrying bidirectional audio-video cross-attention, temporal positional embeddings, and cross-modality AdaLN that hands both streams the same timestep conditioning.

Bidirectional is the load-bearing word. The audio stream can see what the video stream is doing and the video stream can see what the audio stream is doing, at every block, at every step. The asymmetry — 14B against 5B — tells you where the difficulty was judged to be. Picture is roughly three times the model that sound is.

One more design decision matters more than its one-line description suggests: the audio and video streams receive different context embeddings from the same prompt. You write one prompt. The model derives two conditioning signals from it. This is what makes lip-sync and modality-specific conditioning possible, and it is why lesson f5 spends half an hour on prompting. You are writing a shot description and a sound brief in the same paragraph, and they go to different places.

Why a single pass kills drift

Picture the obvious alternative: generate video, then generate audio conditioned on that video, then nudge them into alignment. Every stage in that chain is a place where error accumulates, and the last stage is a negotiation between two models that never agreed on what was happening. Sync drift in stitched pipelines is not a bug someone forgot to fix; it is what happens when alignment is a correction rather than a constraint.

Joint generation makes alignment structural. There is no drift to remove because there was never an offset to introduce.

That is the good news, and it also sets the shape of the bad news. Because the streams are entangled, changing one independently is not free. If you have a clip you like and want different audio over the same picture, you are not just asking for a re-render of one stream — you are asking to hold one output fixed while re-running a process that produced both. That is exactly why the API carries dedicated endpoints for those jobs: audio-to-video, and retake with the ability to target video, audio, or both. Learn the architecture and the endpoint list stops looking arbitrary.

The video VAE, and why frame counts look strange

The transformer does not work on pixels. A video VAE compresses [B, 3, F, H, W] into latents shaped [B, 128, F', H/32, W/32]: 32× spatial downsample per axis, 128 latent channels.

Two consequences worth internalising.

First, 32× per axis is aggressive. A 768×512 frame becomes a 24×16 latent grid. Whatever survives is what the model reasons about; the decoder is responsible for putting detail back. Expect fine, high-frequency content — small text, distant faces, hair against a busy background — to be where quality degrades first. That is not a prompting failure you can write your way out of.

Second, the temporal axis is packed too, and that produces the rule that catches everyone: frame counts must satisfy `(F-1) % 8 == 0`. Legal counts are 1, 9, 17, … 121, 201, 241. Illegal counts include every round number you would naturally reach for: 100, 120, 128, 240. This is why the official local example uses num_frames=121 at frame_rate=25.0 — 4.84 seconds, an odd-looking duration that is a legal frame count.

The audio VAE

The audio side is mel-spectrogram based: roughly one token per 1/25 s, 4× temporal downsampling, and a HiFi-GAN vocoder taking a 16 kHz mel representation to a 24 kHz waveform.

A token per 1/25 s means the audio stream's native rhythm lines up with a 25 fps frame grid — the same 25 fps that appears throughout the official local examples. And 24 kHz is your output ceiling. That is entirely adequate for dialogue and foley, which is what this model is for. It is not a music-mastering format, and if your delivery spec says otherwise, plan for the audio to be replaced rather than assuming you can push the model harder.

Distillation and the two-pass pipeline

Alongside the full dev checkpoint there are distilled checkpoints running at 8 steps with CFG=1, plus a distilled-1.1 that refines aesthetics and audio.

What the architecture predicts

Read the design and you can predict the product before you touch it.

Easy: short, self-contained audiovisual events where picture and sound are one thing — a door slamming, a person speaking a line, rain on a window. Joint generation is doing exactly what it was built for.

Harder: anything that needs one modality held constant while the other changes; anything depending on fine detail that a 32× spatial downsample discards; and anything long, for reasons that turn out to be published limits rather than architectural mysteries. That is the next lesson.