Alibaba's new open source video model may become a Chinese Adobe

Know the danger

follow with interest

2025-05-16

0 comments

920 views

1 Collection

27 minutes

  B-end products need to rely more on sales teams and channel cooperation to promote their products, while C-end products need to make more use of online marketing and word-of-mouth communication to promote their products .

Alibaba's open-source Wan2.1-VACE video editing model, with its powerful multitasking capabilities and editability, is expected to become the Chinese version of Adobe. This model not only supports basic cultural videos, but also has various functions such as image reference, video redrawing, and local editing, greatly improving the flexibility and controllability of video generation. This paper will discuss the technical highlights, application scenarios and potential impact of this model on the future video production industry.

Last night, Alibaba officially opened sourced the All in One video editing model, Wan2.1-VACE, which may make Alibaba the future Adobe of China in the field of video production.

Why do you say that? Before introducing VACE, let's first lay the foundation for the current status of video generation model products.

The most impressive impression of such products on the public is usually the stunning sensation brought by instant generation. Not only does it generate quality, but its card drawing feature makes the experience of different output results for the same input as a blind box interesting.

However, for professional groups who consider AI as productivity, card drawing is just the first step in their work, and in reality, they often collapse into the second and multiple editing stages.

Imagine a scenario where a startup company wants to post a 30 second promotional video for a new product on social media. The product of this company is a portable coffee machine, targeting urban white-collar workers and travel enthusiasts. Employees hope to have AI help complete the production of short films. In practice, relying solely on AI's "one-time output" of materials or "frequent card draws" will never work.

Because the design requirements themselves will constantly change from the first generation, for example, the highlights proposed by the marketing department ("fast extraction", "USB-C power supply", "lightweight") are often adjusted temporarily in subsequent meetings. In addition, good ideas need to be repeatedly polished, such as the rhythm of the visuals, the tone of the copy, and the switching of shots. Only after reading the initial draft can one know whether they are "right or not".

By taking on these three challenges, product managers will only continue to appreciate their value

Good product managers are scarce. Product managers who understand users, business and data are still in demand when they go out of the Internet. On the contrary, if a product manager only engages in simple communication, inefficient execution, and shallow thinking, they may not be able to survive the flood of the next 3-5 years.

View details>

If AI can only produce results at once, any later modifications will be difficult or even equivalent to starting over. So only with multiple rounds of interaction and editability can the creative cycle be greatly shortened, while maintaining creative flexibility, and AI can become a true productivity tool.

Therefore, close human-machine interaction is currently the most suitable path for the development of AI, but it is very difficult to achieve this.

Compared to text, the controllability of pixel based object generation is obviously more difficult. Without considering semantic or physical constraints, when comparing the number of states, for a sentence of 10 tokens, taking GPT-4 as an example, its vocabulary size is about 10 ^ 5, so the total number of states is (10 ^ 5) ^ 10=1.0 × 10 ^ 50. For a color RGB video (3 RGB channels per pixel, 256 values per RGB channel, for a total of 768 values), such as a 128 × 128 pixel, 3-second 10 frame video, with a total of 491520 pixels and a potential state count of 768 ^ 491520, the magnitude of the state count is much larger than that of text.

It is not difficult to understand why video generation products are currently generally slow and expensive, which actually reflects the efficiency and cost advantages of secondary editing compared to mindless card drawing.

At present, there has been good progress in the controllability of image and 3D generation, but the controllable generation of videos has only recently achieved visible results to the naked eye. Moreover, mainstream related products still have significant limitations, which actually greatly restrict the implementation of creativity.

The Tongyi Wanxiang team expressed to Zhiwei that video generation and editing face significant challenges

Fragmentation problem in video generation and editing: Traditional video generation or editing methods often focus on a single task (such as text video, reference generation, object replacement), lack a unified framework, resulting in independent models for different tasks, low link concatenation efficiency, and high inference costs.

Lack of controllability: Existing methods are difficult to support multi-dimensional or multi task editing simultaneously (such as referencing the subject, content, and structure simultaneously), and users cannot adjust videos as flexibly as editing text.

High quality content generation requirements: The short video and film industries require high fidelity and high consistency video generation, but existing models are prone to problems such as frame flickering and semantic inconsistency.

Taking professional P image software as an example, the key to why a design software can truly come in handy in tight production processes is that it provides a diverse and on-demand tool ecosystem: from repairing brushes and content aware fillers, to channel mixers, bitmap/vector masks, to action scripts and third-party plugins, almost every creative demand can find corresponding "tools".

This allows designers to flexibly switch their ideas and techniques at different stages of the project, without having to jump out of the work interface to complete it.

Last night, Alibaba officially opened sourced the Tongyi Wan2.1-VACE, which has achieved production level multitasking capabilities in the field of AI video.

The open source address is as follows:

GitHub： https://github.com/Wan-Video/Wan2.1

HuggingFace： https://huggingface.co/Wan-AI

Magic Community: https://www.modelscope.cn/organization/Wan-AI?tab=model

Wan2.1-VACE has two versions, 1.3B and 14B. The 1.3B version is suitable for local deployment and gameplay fine-tuning, and can run on consumer grade graphics cards (previously released as a Preview version). It supports 480P resolution, while the 14B version has higher generation quality and supports both 480P and 720P resolutions.

Now, developers can download and experience it on GitHub, Huggingface, and the Magic Community. The model will gradually be launched on the official website of Tongyi Wanxiang and Alibaba Cloud Bailian.

Wan2.1-VACE focuses on "the most comprehensive functions" and "editability". A single model not only supports the most basic liberal arts videos, but also supports multiple functions. There is no need to train a new expert model for a single function, and the cost of deploying multiple models is also eliminated. The Tongyi Wanxiang team stated that Wan2.1-VACE is the first integrated model based on video DiT architecture that supports such a wide range of tasks.

Text conditions have greatly improved the editability of video generation, but they are not sufficient to accurately control all details in the video (such as precise layout, object shape, etc.). Therefore, Wan2.1-VACE has extended multitasking capabilities to achieve more refined editability.

Overall, Wan2.1-VACE's multitasking capabilities include:

Image reference capability, given a reference subject (face or object) and background, generates video content with consistent elements.

Video re drawing capability, including pose transfer, motion control, structure control, re coloring, etc. (based on depth map, optical flow, layout, grayscale, line draft, and pose control);

Local editing ability, including subject reshaping, subject removal, background extension, duration extension, etc.

For example, image reference generation. In the example, Wan2.1-VACE generated a video based on reference images of a small snake and a girl, in which the girl gently touched the small snake. Image reference generation is important for adding new elements and ensuring element consistency in multi shot videos.

Tip: In a joyful and festive scene, a little girl wearing bright red spring clothes is playing with her cute cartoon snake. Her spring clothes were embroidered with golden auspicious patterns, emitting a festive atmosphere, and her face was filled with a bright smile. The snake's body presents a bright green color, with a rounded shape and wide eyes that make it appear both friendly and humorous. The little girl happily caressed the snake's head with her hand, enjoying this warm moment together. The colorful lanterns and ribbons around them decorate the environment, and the sunlight shines on them, creating a New Year atmosphere full of friendship and happiness.

Partial editing is both efficient and indispensable, as it enables the ability to delete, replace existing elements, and add new ones. In the following picture, Wan2.1-VACE uses video editing capabilities to remove the tablet computer from the woman's hand without leaving any trace.

Tip: Documentary photography style, the real estate self media blogger is located in the center of a modern living room. The blogger is wearing simple and fashionable clothes, with a smile on his face. He holds both hands in front of him, with nothing on his hands, and is introducing the situation of the house to the camera. The background is a spacious and bright living room, with simple and modern furniture and green garden outside the floor to ceiling window. The room is well lit and cozy. Mid shot full body portrait, from a head up perspective, with a slight sense of movement, like tapping a finger on the screen.

In addition, by further combining video re drawing, Wan2.1-VACE can control the presentation of new elements based on different motion control capabilities.

For example, sketches/edge maps are suitable for controlling the overall motion trajectory of objects. The following figure shows the fighter motion and lens motion effects generated by Wan2.1-VACE based on sketch motion trajectories and fighter reference maps.

Tip: From the perspective of a fighter jet, rapidly spinning, engaging in a dogfight with enemy planes in the clouds, suddenly rolling, rapidly rolling down, missiles brushing over the fuselage, and the tail flame tracing an arc in the clouds.

Grayscale videos provide information on the brightness and darkness of content, which can be used to guide models in coloring or reconstructing details. In the following image, Wan2.1-VACE also generated a video of a man riding a horse next to a running train based on grayscale images.

Tip: A foreign man is running on a brown horse beside the railway tracks. He was wearing a gray shirt and a black cowboy hat, with a background of a steam train in motion, consisting of multiple carriages and emitting smoke. The sky is an orange sunset scene.

The human pose map (skeleton keypoints) provides intuitive structural information and is very suitable for controlling the actions of characters in videos. In the following figure, Wan2.1-VACE generated a video of a boy practicing karate based on a human pose map.

Tip: Realistic style photography, a 10-year-old white boy wearing a white martial arts uniform and a yellow belt practicing karate in a spacious and bright room. He punches and poses with concentration and discipline, and his movements are smooth and effortless. The background is blurred, and the stacked mats and other gym equipment can be vaguely seen. The camera follows his movements, horizontally shaking left and right to capture mid to close range shots, showcasing his strong and coherent movements.

Optical flow describes the motion of pixels between frames and is an important modality for expressing fine-grained motion structures. In the following figure, Wan2.1-VACE generated a scene of plum blossoms falling into the water and causing splashes of water based on optical flow diagrams.

Tip: Documentary photography style, a deep purple plum blossom slowly falls into a transparent glass, splashing crystal clear water droplets. The scene captures this moment in slow motion, with water splashing in the air, forming a beautiful arc. The water in the glass is clear to the bottom, and the color of the plum blossom contrasts sharply with it. The background is concise and highlights the main subject. Close up close-up, vertical top-down perspective, showcasing the beauty of details.

Wan2.1-VACE also supports video background extension and video duration extension. In the following image, Wan2.1-VACE extends the video background to restore the close-up of a woman playing the violin to the large-scale performance scene already hinted at in the original content.

Tip: An elegant lady is enthusiastically playing the violin, with an entire symphony orchestra behind her.

By extending the video duration, Wan2.1-VACE presents the scene of an off-road rider running to a small slope ahead from behind the camera.

Tip: Off road motorcycle race scene, a fully equipped athlete rides a motorcycle onto a slope, with the wheels splashing high soil.

Overall, the above case demonstrates the organic relationship between Wan2.1-VACE multitasking abilities. Image reference and local editing provide the ability to delete, replace, and add new elements, while video redrawing controls the specific presentation of new elements. Different modalities have their own strengths, and video background extension and video duration extension provide more open imaginative space or restore complete scenes in space and time.

So to fully leverage the advantages of Wan2.1-VACE, we should explore the free combination of various atomic abilities, only in this way can we achieve production level scenario implementation, and Wan2.1-VACE can indeed support this very well.

For example, in the following multi shot promotional video, Wan2.1-VACE freely combines multiple abilities to meet the needs of each shot, while maintaining good character consistency between shots.

For example, this clip combines image extension, posture transfer, and image reference. The window is enlarged with image extension, the girl is encouraged to do stretching exercises with posture extension, and more birds are added with image reference.

This clip combines local editing and image reference, using image reference to instantly shuttle the little elephant doll through the "arbitrary door" to the marked local area of the park scene.

This clip combines motion control and image reference, allowing the elephant to float from the ground and soar into the sky.

This clip combines local editing, pose transfer, and image reference, using pose transfer to control the girl's gait and allowing her to quickly change clothes through local editing and image reference.

Finally, this clip combines posture transfer and image reference, endowing the girl with a professional skateboarding posture, combined with different landscape images, allowing the girl to ride a skateboard through the city, desert, and sea.

How can developers not love such a awesome production tool? This can be seen from the current achievements of Tongyi Wanxiang.

Since February of this year, Tongyi Wanxiang has successively opened sourced the Wensheng Video Model, Tusheng Video Model, and Front and Back Frame Video Model. Currently, the download volume in the open source community has exceeded 3.3 million, and it has won over 1.1 million stars on GitHub, making it the most popular video generation model of the same period. It is expected that Wan2.1-VACE will also bring a new wave of community activity.

How did Wanxiang achieve the integration of so many abilities into one model? To answer this question, Zhiwei communicated with the Tongyi Wanxiang team.

Tongyi Wanxiang expressed that to achieve this, there will actually be many challenges:

Multi task unified modeling: How to integrate multiple tasks such as generation and editing within a single architecture while maintaining high performance.

Fine grained control: How to decouple content (objects), motion (timing), style (appearance), and other attributes in videos to achieve independent editing.

Data and training complexity: Multi task data construction requires processing according to the characteristics of each task and constructing a high-quality training set.

In the modeling section, VCU (Video Condition Unit) is the core module for Wan2.1-VACE to achieve comprehensive and controllable editing. The use of VCU is the source of task unification and also the difference from other proprietary models that only support specific tasks. ”The criticality of VCU is reflected in:

Unified representation: Define the input for video generation and editing as input video, input mask, reference image, etc;

Multi task unification: VCU serves as an intermediate layer to isolate task differences (such as generation or editing), in order to inject representations of different tasks into the generation module.

Fine grained control: Through the decoupling design of VCU, task differentiation and fine-grained control can be achieved.

Briefly explain the composition of VCU. In fact, the multitasking capability of Wan2.1-VACE can be represented as a unified input interface for three modalities of data, namely text prompts, reference images, and masks.

According to the requirements of various video task capabilities for three types of multimodal inputs, they are divided into four categories:

Text to video generation (T2V);

Reference image generation (R2V);

Video to Video Editing (V2V), which refers to video redrawing;

Mask video to video editing (MV2V), which refers to partial video editing.

VCU uses a unified representation method to represent the inputs of the above four types of tasks in the same triplet form (T, F, M), where T is the text prompt, F is the reference image or context frame, and M is the mask:

In T2V, there is no need for context frames or masks. Each frame defaults to a 0 input value, and each mask defaults to a 1 input value, indicating that all pixels in these 0-value frames will be regenerated.

For R2V, additional reference frames (such as faces, objects, etc.) are inserted before the default 0-value frame sequence. In the mask sequence, the mask of the default frame is all 1s, and the mask of the reference frame is all 0s, which means that the default frame should be regenerated while the reference frame should remain unchanged.

In V2V, the context frame is the input video frame (such as depth, grayscale, pose, etc.), and the mask defaults to 1, indicating that all input video frames will be regenerated.

For MV2V, both context frames and masks are required, with mask parts being 0 and 1, and frames with mask 1 will be regenerated.

Thus, different tasks were unified into one model, as shown in the following figure.

VCU: A unified input representation for four types of video processing tasks.

Image source: https://arxiv.org/pdf/2503.07598

The structure of VCU is very simple and beautiful, but it is also based on the team's long-term technical accumulation and evolution. Tongyi Wanxiang said, "ACE and ACE++are our initial attempts at unified generation and editing in the field of images, and have achieved good results. And VACE is also a flexible application of ACE in the video field, where the construction idea of VCU evolved from a unified input module in images. ”

To implement VCU itself, there are also some challenges. Generally speaking, Wanxiang said, "VACE adopts a unified strategy (early fusion) on the input side, which is different from using additional encoding modules to process different input modalities. We follow the design principles of simplicity and uniformity. The core challenge is to use a single model to achieve results compared to proprietary models. ”

Building a multitasking model also requires higher data quality. The Tongyi Wanxiang team needs to mark the first frame of the video, such as which objects are in the image, and perform position box selection and localization to remove videos with target areas that are too small or too large. They also need to calculate whether the target appears in the video for a long time in the time dimension to avoid abnormal scenes caused by the target being too small or disappearing.

In order to adapt the model to flexible ability combinations, the Tongyi Wanxiang team randomly combined all tasks for training. For all operations involving masks, perform arbitrary granularity enhancement to meet the local generation requirements of various granularities.

The training process adopts a phased, easy to difficult approach. The Tongyi Wanxiang team first focuses on tasks such as mask repair and extension based on pre trained text to video models. Next, gradually transition from single input reference frames to multi input reference frames, and from single tasks to composite tasks. Finally, use higher quality data and longer sequences to fine tune the model quality. This allows the input for model training to adapt to any resolution, dynamic duration, and variable frame rate.

In recent years, video generation AI models have undergone rapid evolution, completing a leap from "being able to generate" to "being able to handle generation". The evolution of multimodal input reflects the shift from "one key opening one lock" to "multi clue collaborative command".

Different modalities have their own strengths: text provides abstract semantics, images provide appearance details, pose/sketch limits structure, optical flow constrains motion continuity, while reference frames ensure identity is constant, and so on. This process fully demonstrates the potential of AI video: by continuously introducing new control dimensions, humans will continuously enhance their ability to make AI create videos according to intention.

By integrating different control dimensions, video generation models have begun to possess the ability to comprehensively understand and make decisions, balancing the needs of all parties under complex conditions. This not only greatly improves the editable nature of the generation, but also makes the model more adaptable to the needs of mixing multiple materials in real creative scenarios.

It can be seen that Wan2.1-VACE is a key achievement in completing this transformation.

Looking ahead to the future, how to further improve the generated reality, extend the duration, enhance interactivity (such as real-time adjustment of generated videos), and combine physics and 3D knowledge to avoid distortion will be a continuous research focus. But it can be certain that the editable and multi conditional video generation paradigm has been basically established and will become a new paradigm for digital media production.

And this production paradigm may completely change the workflow of video post production in the future, overturning tools such as PR, AE, and Final Cut in the hands of video production workers.

Author: Liu Dagu Editor: Da Bing

This article is written by everyone who is a product manager [Zhiwei]. WeChat official account: [Zhiwei]. It is original/authorized to be published by everyone who is a product manager. Reproduction without permission is prohibited.

The title image is from Unsplash, based on the CC0 protocol.

阿里开源的新视频模型，没准会成为中国 Adobe