British filmmaker Scott Mann directed Robert De Niro in the 2015 action thriller “Heist.” The film was a critical and box office fop ($4 million at the box office on a $15 million budget), so no one could’ve predicted at the time the film’s greatest legacy would be an artificial intelligence that automatically dubs movies into any language. Mann and his business partner Nick Lynes unveiled their new dubbing company Flawless this week in a feature published by Input Magazine. The company uses AI to digitally recreate and edit an actor’s face so that his or her mouth movements match the dub, thus maintaining the authenticity of the performance.
Mann was inspired to create Flawless after he saw a foreign-language dub of “Heist.” The director remembered how obsessive De Niro was in the creation of the character, going so far as to analyze what cufflinks would be best for his character to wear. But all the specific choices De Niro made to create the character went out the window with a bad dub.
“The dubs effectively removed the way [De Niro] made character decisions, by changing dialogue to fit his facial movements, instead of adapting the script and staying true to his performance,” Mann said. “It became apparent that the process doesn’t work. At best, it’s a compromise.”
Per Input: “[Flawless’ technology] strips actors’ faces off, converting their visages into a 3D model, according to Lynes. ‘This creates millions of 3D models, which the AI uses as reference points,’ he says. ‘And then, using an existing foreign-language recording of the dialogue, it studies the actor and generates a new 3D model per frame.’ Finally, the imagery is converted back to 2D. Digital effects artists can then manually fix anything that seems off.”
The Flawless method moves the dubbing process out of post-production. Dubbing audio is recorded at any point in production and is used in tandem with the 3D models created by the AI to synch up the dub with the proper mouth movements. While the system recalls deepfake technology, Mann stressed it is a much different process.
“Deepfakes use two-dimensional face-replacement to hijack people’s identity,” Mann said, “which is the opposite of what we do, which is responsibly utilizing neural networks to enhance a performance. We also take into account the geometry and geography of the image, which is the biggest difference — this creates a fully three-dimensional performance.”
Mann told Input that Flawless is in talks with film studios and is currently working with one streamer. The first movie to be released using the Flawless technology should arrive sometime in the next 12 months.
“There will be people who will refuse to watch our dubs and say subtitles are the purest form, but this helps share great international stories with those who may have never otherwise seen them,” Mann added about Flawless. “We’ll soon reach the level where you won’t realize we’ve done it, and the debate will solve itself.”
Flawless is far from the only company looking to help Hollywood solve its subtitles problem. IndieWire reported in January on Deepdub, an Israel-based startup that claims that it can accurately dub an actor’s voice into other languages using just a few voice samples while retaining the actor’s tone and pitch.
Head over to Input Magazine’s website to read more on Flawless.
Source: Read Full Article
