Adobe’s generative artificial intelligence flagship is Firefly, a text-to-image tool that powers generative fill-in for Photoshop and text-to-vector in Illustrator. However, the 2024 Firefly model will power three AI video tools inside video editing software Premiere Pro: Generative Extend, Object Addition and Removal, and Text to Video.
Premiere already allows video editors to integrate Adobe creative cloud software like After Effects (motion graphics) and Audition (sound mixing). Today’s announcement makes clear that Adobe will take the same approach with AI. The company also announced that it is in “early explorations” to integrate video generation from OpenAI (Sora), Runway, and Pika Labs into Premiere.
Last week, Adobe demonstrated these new gen AI tools for IndieWire and shared examples (see video at the top of the page) of what will be available inside Premiere Pro before the end of the year.
Generative Extend
Every editor has wished for just a few more frames to allow for a smoother edit or transition. The Gen Extend tool would allow editors to use AI to create new frames of up to a “few seconds” that Adobe promises will “seamlessly add frames to make clips longer.”
In the preview video, the tool extends the actress’ stoic performance by a couple of seconds to fill a gap in the editor’s timeline. While extending performance introduces ethical and union issues, filmmakers have long used VFX tricks to blend footage to buy some extra frames. This tool could let editors create real-time solutions inside Premiere.
Object Addition & Removal
Adding and removing objects from frame often involves painstaking VFX work, but the Firefly tool would allow an editor to do it in real time. The key will be if gen AI can maintain photorealism. The examples offered in Adobe’s preview (adding a stack of diamonds to the briefcase and removing an electrical box from frame) appear photoreal, maintaining image and lighting consistency. Both in its press release and in a briefing with IndieWire, Adobe insisted that these pixels were generated by the upcoming Firefly video model without further manipulation.
Early results look promising, but until editors test drive it on a wide range of footage we won’t know how applicable these tools will be in the near future.
Text to Video
Firefly is not new in terms of the ability to use text-to-video tools to create shots. What’s fascinating is Adobe will give filmmakers the ability to access text to video from inside Premiere, and will include Firefly competitors Sora, Runway, and Pika Labs. It’s clear Adobe anticipates editors utilizing these tools in their daily workflows.
Copyright & Content Credentials
Many gen AI image and video models are built on copyrighted material. In an embarrassing interview, OpenAI Sora’s CTO couldn’t even answer straightforward questions about what trained its the generative video model.
However, filmmakers have praised Adobe for using only licensed images to train Firefly. Adobe executives confirmed to IndieWire that the soon-to-be-released Firefly video model will continue abiding by these same rules.
It’s curious, then, that Adobe will integrate third-party AI companies that are far less ethical regarding copyright. However, Adobe said it will continue its use of content credentials.
“As one of the founders of the Content Authenticity Initiative, Adobe pledges to attach
Content Credentials — free, open-source technology that serves as a nutrition label for online content — to assets produced within its applications so users can see how content was made and what AI models were used to generate the content created on Adobe platforms,” reads today’s press release.
The permanence of these nutrition labels — and how easily they can be peeled off the jar — remains to be seen.
AI Audio Tools
In May, Premiere’s audio workflow will get an upgraded set of tools, some of which are AI powered and have been available in beta for a while. These include:
Interactive fade handles: Editors can create custom audio transitions by dragging clip handles to create audio fades.
Essential Sound badge with audio category tagging: AI automatically tags audio clips as dialogue, music, sound effects, or ambience.
Effect badges: Visual indicators show which clips have effects, quickly adds new ones, and automatically opens effect parameters from the sequence.
Redesigned waveforms in the timeline: Waveforms intelligently resize as the track height changes on clips, while new colors make sequences easier to read.
AI-powered Enhance Speech tool: Removes unwanted noise and improves poorly recorded dialogue; this one has been generally available since February.
