Stability AI turns photos into 3D models with camera movement

Stability AI speaks of a “multi-view video generation with 3D camera control”. This refers to an AI model that can convert photos into 3D models. These in turn can be viewed from all sides, enabling an immersive view. Stable Virtual Camera is still a research preview.

Stability AI uses a diffusion model to create the 3D videos. These are AI models in which images are generated point by point using noise. A single photo or up to 32 images can be used as input. The generated videos are available with different camera paths, such as “dynamic”, “spiral”, “dolly zoom”, “pan” and more. In the blog post, the provider speaks of “realistic depth and perspective – without complex reconstructions and scene-dependent optimizations”.

The concept behind the model is based on the needs of digital cameras in filmmaking and 3D animations. However, thanks to AI, significantly less input and work is required.

Stable Virtual Camera for researchers

Initially, Stable Virtual Camera is only available to researchers under a non-commercial license on Higging Face or via Github. The weights have also been published. According to Stability AI, the new model beats comparable models such as ViewCrafter and CAT3D in some benchmarks. Stability AI particularly emphasizes the new image synthesis. However, the provider also says that certain content could lead to poorer quality, for example when people, animals or dynamic textures, such as water, are to be shown. There are also occasional flickering artifacts, especially with irregularly shaped objects.

After several disputes at the top of Stability AI, the AI company should be back on a firm footing. The co-founder and investor Emad Mostaque is said to have financially damaged Stability. Some founders and employees left the company as a result. Now there are new investors, and James Cameron, someone who is familiar with the needs of the film business, sits on the supervisory board.

(emw)