Runway launches new Gen-4 AI video generator

Runway AI Inc. today introduced Gen-4, a new artificial intelligence model that can generate videos based on natural language prompts.

New York-based Runway is backed by more than $230 million in funding from Nvidia Corp., Google LLC and other investors. The company launched its first AI video generator, Gen-1, in February 2023. The new Gen-4 model that debuted today marks the fourth iteration of the algorithm series.

Many video generation models are based on a neural network designed to generate images. The reason is that a video is a sequence of images, which means it can be generated one image at a time. This is usually done through a process called diffusion: A model starts with an image containing noise and gradually adds details over multiple steps.

What sets a video generator apart from an image generator is that it must ensure visuals are consistent across all the clip frames it produces. This requires extending the core diffusion-optimized artificial neurons with additional components, which adds complexity. Even with the additional components, ensuring consistently across a clip’s frames is often a challenge for video generators.

Runway says that its new Gen-4 model addresses that limitation. It allows users to upload a reference image of an object that a video should include along with a prompt containing design instructions. From there, Gen-4 ensures that the object retains a consistent look throughout the entire clip. 

“Whether you’re crafting scenes for long form narrative content or generating product photography, Runway Gen-4 makes it simple to generate consistently across environments,” Runway staffers wrote in a blog post.

The company says that Gen-4 can keep an object consistent even if users modify other details. A designer could, for example, change a clip’s camera angle or lighting conditions. It’s also possible to place the object in an entirely new environment.

Gen-4 doubles as an image editing tool. Users can, for example, upload two illustrations and ask the algorithm to combine them into a new drawing. Gen-4 generates multiple variations of each output image to reduce the need for revisions.

Initially, Runway will enable users of the model to generate five- and 10-second clips. The startup released several demo videos that are nearly two minutes long, which hints it could update Gen-4 in the future to let customers generate more complex clips.

Gen-4’s launch comes shortly after OpenAI temporarily disabled Sora’s video generation features for new users. The ChatGPT developer cited “heavy traffic” as the reason. OpenAI Chief Executive Officer Sam Altman stated earlier today that the company had added 1 million new users in under an hour. 

Image: Runway