Over the past year, we have seen a huge amount of development in 2D image generation AI tools using platforms such as Stable Diffusion, Midjourney and Dall-e. These text-to-image AI tools turn text prompts into detailed imagery in a matter of seconds. As production companies look towards 2024, the film industry continues to undergo a transformative shift partly due to the integration of artificial intelligence (AI) and visual effects (VFX).
Tue 19th of Sep 2023
With its ability to learn from data and make predictions, AI is revolutionizing VFX, giving artists powerful tools to push the boundaries of creativity and efficiency. AI technology is advancing and is already being used in the visual effects industry.
While AI is not yet able to generate to create full 4K shots, it can be used for background replacements and rotoscoping, to name a few. Rotoscoping, such as removing stunt wires from a shot, can be a very time-consuming task which makes any VFX artist frown.
Autonomous rotoscoping is a game-changer
Weta Digital‘s use of AI in rotoscoping for “The Mandalorian“ is a perfect example of this technology’s potential. Their innovative AI-driven approach was able to handle the complex task of rotoscoping the entire film, effectively cutting down the time by an estimated 90%.
AI tools can also handle complex scenarios such as motion blur or overlapping objects, showing their versatility and adaptability. The development possibilities are virtually endless as well.
Realistic facial animation
Besides rotoscoping, creating realistic facial animations is one of the major points for the believability of the film. With AI animation companies can generate highly accurate and lifelike animations for digital characters by analyzing human facial movements and expressions.
For example, Disney used AI to create the facial animation of Sully in “Monsters University”. AI was able to analyze the movements and expressions of real-life actors and learn human facial expressions to make the performance more realistic and emotionally engaging.
These are just a few examples from the VFX industry. As being said, AI has potential in various aspects of the VFX industry, from autonomous rotoscoping and realistic facial animation to crowd simulation, set extensions, matte painting, and many more.
All about AI in VFX
According to Scott Squires, a veteran visual effects supervisor who worked for Industrial, Light & Magic (ILM) for 20 years, much of the very time-consuming and tedious hand work VFX done in the past has been at least partially automated. He agrees about new technology enabling VFX artists to make an even higher quality level of work and provide many more options for creativity.
Automating VFX tasks film studios increase the cost-effectiveness and reduce the need for large teams leading to lower productions costs. And last but not least, it enhances creativity by reducing the time spent on repetitive tasks.
Simplified example of NST in action adding style to Ember’s flames. © Pixar
Pixar released their latest animation film “Elemental” in 2023. The characters are more abstract than ever, and the prospect of making a character who is made of fire, water or air is no easy one. When hand sculpting and animating every detail is not an option, Pixar used “volumetric neural style transfer” (NST).
The creative team took the raw voxels of the “pyro simulation,” or generated flames, and passed it through a style transfer network trained on an artist’s expression of what they wanted the character’s flames to look like.
The resulting voxels have the natural, unpredictable look of a simulation but also the unmistakable cast of the artist’s choice. Pixar’s Paul Kanyuk points out that Pixar didn’t use AI to make Elemental. According to Kanyuk they used volumetric NST to shape her silhouette edges. Still, NST is a machine learning technique which generally speaking falls under the AI umbrella. AI tools were used also in DNEG’s “Nimona” and HBO’s “The Last of Us”.
The future of VFX
As AI technology continues to advance, we can expect AI’s role in visual effects to become even more significant. While we have already seen remarkable applications of AI, the journey has just begun. The future holds exciting possibilities, with AI set to revolutionize the VFX industry, leading to more immersive and visually stunning cinematic experiences.
AI’s role in VFX is not about replacing artists though, but empowering them with tools that augment their capabilities, bringing their creative visions to life in ways previously unimaginable.
The production incentive has played a major role in getting and keeping the productions in Finland. Read more here!
Kajawood Studios helps producing music videos from concept to edit. Read about our latest music video production here!
Kajawood Studios was a co-producer of the Finnish road movie Detour to father premiered in March 2023.
The international environmental sustainability project "albert" will also be introduced at Kajawood Studios in Sotkamo.