Equivariant neural network architectures are extended to handle time-parameterized transformations, improving performance in sequence models like RNNs for tasks involving moving stimuli.
Data arrives at our senses as a continuous stream, smoothly transforming from
one instant to the next. These smooth transformations can be viewed as
continuous symmetries of the environment that we inhabit, defining equivalence
relations between stimuli over time. In machine learning, neural network
architectures that respect symmetries of their data are called equivariant and
have provable benefits in terms of generalization ability and sample
efficiency. To date, however, equivariance has been considered only for static
transformations and feed-forward networks, limiting its applicability to
sequence models, such as recurrent neural networks (RNNs), and corresponding
time-parameterized sequence transformations. In this work, we extend
equivariant network theory to this regime of `flows’ — one-parameter Lie
subgroups capturing natural transformations over time, such as visual motion.
We begin by showing that standard RNNs are generally not flow equivariant:
their hidden states fail to transform in a geometrically structured manner for
moving stimuli. We then show how flow equivariance can be introduced, and
demonstrate that these models significantly outperform their non-equivariant
counterparts in terms of training speed, length generalization, and velocity
generalization, on both next step prediction and sequence classification. We
present this work as a first step towards building sequence models that respect
the time-parameterized symmetries which govern the world around us.