Alibaba has unveiled Wan2.1-VACE, an open-source AI model designed to shake up how we create and edit videos.
VACE isn’t appearing out of thin air; it’s part of Alibaba’s broader Wan2.1 family of video AI models. And they’re making a rather bold claim for it, stating it’s the “first open-source model in the industry to provide a unified solution for various video generation and editing tasks.”
If Alibaba can succeed in shifting users away from having to juggle multiple, separate tools towards one streamlined hub—it could be a true game-changer.
So, what can this thing actually do? Well, for starters, it can whip up videos using all sorts of prompts, including text commands, still pictures, and even snippets of other video clips.
But it’s not just about making videos from scratch. The editing toolkit supports referencing images or specific frames to guide the AI, advanced video “repainting” (more on that in a sec), tweaking just selected bits of your existing video, and even stretching out the video. Alibaba reckons these features “enable the flexible combination of various tasks to enhance creativity.”
Imagine you want to create a video with specific characters interacting, maybe based on some photos you have. VACE claims to be able to do that. Got a still image you wish was dynamic? Alibaba’s open-source AI model can add natural-looking movement to bring it to life.
For those who love to fine-tune, there are those advanced “video repainting” functions I mentioned earlier. This includes things like transferring poses from one subject to another, having precise control over motion, adjusting depth perception, and even changing the colours.
One feature that caught my eye is its ability to “supports adding, modification or deletion to selective specific areas of a video without affecting the surroundings.” That’s a massive plus for detailed edits – no more accidentally messing up the background when you’re just trying to tweak one small element. Plus, it can make your video canvas bigger and even fill in the new space with relevant content to make everything look richer and more expansive.
You could take a flat photograph, turn it into a video, and tell the objects in it exactly how to move by drawing out a path. Need to swap out a character or an object with something else you provide as a reference? No problem. Animate those referenced characters? Done. Control their pose precisely? You got it.
Alibaba even gives the example of its open-source AI model taking a tall, skinny vertical image and cleverly expanding it sideways into a widescreen video, automagically adding new bits and pieces by referencing other images or prompts. That’s pretty neat.
Of course, VACE isn’t just magic. There’s some clever tech involved, designed to handle the often-messy reality of video editing. A key piece is something Alibaba calls the Video Condition Unit (VCU), which “supports unified processing of multimodal inputs such as text, images, video, and masks.”
Then there’s what they term a “Context Adapter structure.” This clever bit of engineering “injects various task concepts using formalised representations of temporal and spatial dimensions.” Essentially, think of it as giving the AI a really good understanding of time and space within the video.
With all this clever tech, Alibaba reckons VACE will be a hit in quite a few areas. Think quick social media clips, eye-catching ads and marketing content, heavy-duty post-production special effects for film and TV, and even for generating custom educational and training videos.
Alibaba makes Wan2.1-VACE open-source to spread the AI love
Building AI models this powerful usually costs a fortune and needs massive computing power and tons of data. So, Alibaba making Wan2.1-VACE open source? That’s a big deal.
“Open access helps lower the barrier for more businesses to leverage AI, enabling them to create high-quality visual content tailored to their needs, quickly and cost-effectively,” Alibaba explains.
Basically, Alibaba is hoping to let more folks – especially smaller businesses and individual creators – get their hands on top-tier AI without breaking the bank. This democratisation of powerful tools is always a welcome sight.
And they’re not just dropping one version. There’s a hefty 14-billion parameter model for those with serious horsepower, and a more nimble 1.3-billion parameter one for lighter setups. You can grab them for free right now on Hugging Face and GitHub, or via Alibaba Cloud’s own open-source community, ModelScope.
(Image source: www.alibabagroup.com)
See also: US slams brakes on AI Diffusion Rule, hardens chip export curbs
Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is co-located with other leading events including Intelligent Automation Conference, BlockX, Digital Transformation Week, and Cyber Security & Cloud Expo.
Explore other upcoming enterprise technology events and webinars powered by TechForge here.