Current controllable diffusion models typically rely on fixed architectures
that modify intermediate activations to inject guidance conditioned on a new
modality. This approach uses a static conditioning strategy for a dynamic,
multi-stage denoising process, limiting the model’s ability to adapt its
response as the generation evolves from coarse structure to fine detail. We
introduce TC-LoRA (Temporally Modulated Conditional LoRA), a new paradigm that
enables dynamic, context-aware control by conditioning the model’s weights
directly. Our framework uses a hypernetwork to generate LoRA adapters
on-the-fly, tailoring weight modifications for the frozen backbone at each
diffusion step based on time and the user’s condition. This mechanism enables
the model to learn and execute an explicit, adaptive strategy for applying
conditional guidance throughout the entire generation process. Through
experiments on various data domains, we demonstrate that this dynamic,
parametric control significantly enhances generative fidelity and adherence to
spatial conditions compared to static, activation-based methods. TC-LoRA
establishes an alternative approach in which the model’s conditioning strategy
is modified through a deeper functional adaptation of its weights, allowing
control to align with the dynamic demands of the task and generative stage.