Long video generation is fundamentally a long context memory problem: models
must retain and retrieve salient events across a long range without collapsing
or drifting. However, scaling diffusion transformers to generate long-context
videos is fundamentally limited by the quadratic cost of self-attention, which
makes memory and computation intractable and difficult to optimize for long
sequences. We recast long-context video generation as an internal information
retrieval task and propose a simple, learnable sparse attention routing module,
Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.
In MoC, each query dynamically selects a few informative chunks plus mandatory
anchors (caption, local windows) to attend to, with causal routing that
prevents loop closures. As we scale the data and gradually sparsify the
routing, the model allocates compute to salient history, preserving identities,
actions, and scenes over minutes of content. Efficiency follows as a byproduct
of retrieval (near-linear scaling), which enables practical training and
synthesis, and the emergence of memory and consistency at the scale of minutes.