Every day there seems to be a revolutionary advancement in Artificial Intelligence. With AI that creates artwork and writes essays in seconds, you might wonder if similar tech exists for video.
Well, there’s great news: Forward-thinking video production companies in NYC are taking full advantage of AI tools in order to save clients time and money – while INCREASING the overall quality.
In this blog, we’ll discuss what AI is, how it’s being leveraged in production and post-production, and what the future holds for video AI. We’ll also explain why humans remain an essential part of the process – at least for now!
What is AI and How Does it Work?
Computer programs have automated tasks for decades. What makes AI different?
In a traditional program, instructions are hard-coded with well-defined rules to produce consistent results. The program doesn’t get better unless you update the code.
AI programs are special because they continually learn and refine their effectiveness. Results are less predictable but incredibly accurate, and get more accurate over time. AI can handle a broader array of inputs, operating with more limited information than traditional programs, which don’t work unless you plug in every variable. Because of these advantages, AI is particularly suited for image-based tasks.
But how does AI learn? It might seem like a sci-fi concept, but the basics are simple. Think of AI as an interconnected series of weights. When an input enters the AI, it’s assigned a weight based on its properties. Different weights are applied as the input is processed until it receives a final score which determines the AI’s output. At first, weights are assigned without reason. An AI meant to identify an animal from an image might apply weights to a picture of a bird that leads to it being identified as an elephant. At this point, the results are random noise.
For AI to learn, humans tell the AI when it’s wrong and when it’s correct. If you’ve ever filled out a form proving you are human by identifying a car or airplane, you’ve helped train an AI. With this information, the AI adjusts the weights it assigns until pictures of birds are almost always identified as birds and elephants as elephants.
The actual mechanisms of an AI program are more nuanced than what we’ve described, but the key takeaway is that AI can generate near-instant results from complicated inputs and get better over time, accomplishing tasks faster and more accurately than a person. In fact, sometimes programmers don’t even understand how the weight assignments are achieving the desired results, only that they are. The nebulous nature of this process makes it challenging to adjust an AI’s output at a granular level, so human oversight is still required to achieve specific results.
How is AI Used During Pre-Production and Production?
AI has some uses during pre-production and production, but its role is currently limited.
For example, AI can be used to generate prompts to give you a start on your script. It can also create concept art or a mood board to inspire your visual language and set design. While some experimental projects have used AI to write the film’s scripts, the dialogue is unengaging, repetitive, and the narrative structure rote.
In production, AI tools are finding increased utility. While a talented and knowledgeable crew is required for a successful shoot, AI-powered equipment can reduce the margin of error. Sony’s new camera, the a7R5, has a built-in AI chip for subject detection and focus. It identifies people, birds, animals, trains, plains, and cars. It also understands different poses and where on the subject to focus, even when the subject is obscured. By taking a picture of your subject and identifying it as your target, the camera will keep focus on it even when other people are in frame.
As more film equipment integrates AI, crews will be able to work faster and create more dynamic shots.
How is AI used for Post-Production?
AI has the most utility during post-production. In fact, AI tools are built into most popular editing software, such as Adobe Premiere Pro and Davinci Resolve. Adobe Sensei is the AI technology that powers apps across the entire Adobe Creative Cloud, from Premiere to Photoshop. Additionally, standalone AI tools and plugins provide further capabilities.
AI for Audio Enhancement
Audio engineering is a time-consuming and complicated process. Thankfully, AI can increase an engineer’s efficiency or help automate acceptable results on a tight deadline. Here are some of the ways video and audio professionals use AI:
Denoise
Sometimes a door slams in the middle of the perfect take, the buzzing of a fridge overwhelms dialogue, or screaming children in a nearby playground distract from a key point. In the past, manual noise reduction techniques were required, such as lowering the volume of a specific audio frequency. Since sounds overlap in frequency, this could cause degradation of the overall audio quality.
Today, AI noise reduction tools can analyze an audio track and make frame-by-frame adjustments to the frequencies targeted. They can also reconstruct the voice portion of these frequencies so that dialogue sounds as crisp as prior to denoising.
While these tools often work well, they don’t always produce the intended results. Video and audio professionals can fix problems by adjusting the parameters, or making manual changes, to ensure the best-sounding audio.
Audio Levels
It’s common to adjust audio levels. For example, the volume of music is lowered or raised depending on if someone is speaking. Adjusting music audio levels for dialogue is tedious work, but this process, known as “audio ducking”, can now be AI automated. Editors specify how much the music should be lowered when it is ducked, and the program identifies the dialogue and music tracks, making changes in the proper places. The editor fine-tunes from there.
Level adjustments are also required when dialogue volume is inconsistent. An actor or presenter might start to mutter or move off angle from the mic. Davinci’s AI-based audio leveler will normalize audio to a consistent volume, reduce noise, and even recalculate the audio to compensate for off-angle distortions.
Audio Reconstruction
Sometimes audio gets damaged. A scream blows out the mic or something goes wrong with the audio recorder. AI tools, such as those developed by Izotope, analyze audio and generate new replacement sounds. This generated audio has the same characteristics as the audio that wasn’t usable but without the flaws. It feels like magic, but it’s just AI.
Music Remix
Have a music track that is just a bit too short or long? Or does your music have a great section you want to extend? Adobe Premiere’s auto-remix feature uses AI to allow editors to stretch the section of the track they like to any duration. Premiere will automatically recut the music so that it seamlessly transitions on beat. With this tool, editors spend less time fussing over the perfect track length or manually recutting the music and instead focus on finding the best way to tell the story.
Transcripts and Captions
Transcripts allow clients or editors to develop a paper edit, but manually creating transcripts is time-consuming. With AI transcription tools such as Temi, hours of conversation are accurately transcribed in minutes. With the click of a button, filler words such as “um” can be removed. And the transcript and video can be played back in sync. Finally, the transcript can be shared so that multiple collaborators can highlight what to keep, cross out what to eliminate, and make any corrections to the spelling of names.
Creating and timing captions is another laborious task devoid of creative input. This makes it perfect for AI. Premiere Pro has built-in AI transcription and caption features that understand what is being said, who is saying it, and the duration the text should appear for. Editors perform a simple corrective pass and spend more time making the video great.
AI for Visual Enhancement
AI video tools are used to clean up and enhance images. Here’s some of the most commonly used techniques:
Noise Reduction
When too little light hits a camera’s sensor, you get visual noise – black and white or colored dots over your image. Noise makes a video look muddy and unprofessional. In the past, noise reduction looked as bad, or worse than, the noise you were trying to reduce, resulting in a smeared and textureless image.
New AI noise reduction techniques require powerful computers and take time to process, but the results are worth it. AI analyzes previous and future frames to determine which pixels are noise and which pixels belong to the image. From there, it replaces the noise with clean imagery.
Upscale
Working with archival footage, or a poorly recorded Zoom call that doesn’t meet today’s HD standards? AI tools upscale your image, producing new pixels so that your picture looks crisp. Upscaling tools for static images have been rolled into Photoshop. Simply select your target resolution and choose between different algorithms, and the AI will draw upon its knowledge of similar images to upscale yours.
Increasing the resolution of a video is a more complicated process, and good solutions have only recently emerged. That’s because for every second, at least twenty-four images need to be processed. Additionally, each frame needs to have temporal consistency. In other words, the AI should upscale consecutive frames so that the resulting higher-resolution images flow together. If a building looks sharp in one frame, it needs to have the same sharpness in the next frame.
While video upscaling is still in its infancy, skilled production houses can do wonders with your low-resolution footage.
Content-Aware Fill
Did someone who didn’t sign a release get recorded? Or do you have a copyrighted logo you need removed? With AI-powered content aware fill in Adobe Photoshop and After Effects, you can make it as if the problem object never entered the frame.
When content-aware fill first released, it only worked for photos. By analyzing the patch of the image you wanted removed against the rest of the photo, it would generate a new patch that seamlessly fit. Now these capabilities are available for video. Simply track the object that needs to be removed and After Effects will generate a consistent “fill” for each frame. You might need to adjust parameters before achieving a perfect result.
Auto-Reframe
Need a video reframed vertically for social media? Davinci Resolve’s AI-based auto-reframe feature will crop your footage and automatically apply pans and zooms. This is a great tool in a pinch or under time constraints – but if you have the time and budget it’s better to manually reframe ensuring the most important parts of the image are kept.
Morph Cut
Sometimes you want to cut out an “um” or merge two different takes but you don’t have anything to cut to and a jump cut doesn’t fit your video’s tone. With a morph cut, an AI-powered Premiere tool, fake frames are generated to bridge the two shots together so it appears like one seamless take.
Slow Motion
Footage that wasn’t filmed in slow motion occasionally needs to be slowed down to extend a great moment or to better fit the pacing of the video. Past methods of slowing down footage came with compromises. Slow down the footage too much and it would look jittery or produce visual artifacts. AI methods generate new frames that allow you to slow your footage down further and more seamlessly than previously possible.
AI for Tracking and Identifying Video Content
AI is particularly skilled at understanding what an image contains. Filmmakers leverage this technology to organize their footage and improve their effect workflow:
Scene Edit Detection
If you need edits made to a completed work and don’t have the original project file available, scene edit detection is a useful feature that automatically detects a cut in the footage and breaks it up into individual clips. It works by leveraging its knowledge of film cuts and applying it to your footage. It even lets you know how much certainty it has that there is a cut, so you can double-check the decisions in which it has the least confidence.
Organizing and Searching Footage
A well-organized project makes for an efficient workflow. But getting organized takes hours. You have to review all the footage, manually tag it, and create subfolders for easy access. AI can identify the subject of your clips as well as metadata. With a simple search, you see all clips containing a certain actor or shooting location and can generate a folder containing them.
Masking and Tracking
One of the most common tasks in effects work is tracking an object. Once an object is tracked, a mask can be applied to create a localized effect. While simple movements can be tracked manually, AI-based tracking is accurate and efficient for complex movements, allowing creative work to begin sooner.
While a rough mask works for most tasks, sometimes a mask needs to be pixel accurate. This is the case when a subject is being separated from the background, such as for green screen footage and rotoscoping. In the past, you would have to go frame by frame adjusting your mask. With AI masking tools, you can generate and refine an accurate mask for your whole clip by just drawing a line over your subject during a single frame.
Auto-background removal tools have also become incredibly accurate, allowing you to remove a background from a subject in real time during a live recording.
What is the Future of AI in Video Production?
Indigo Productions is at the forefront of cutting-edge AI video production trends & technology. Our partnership with Synthesia provides a glance at some of the future uses of video AI.
Synthesia produces AI-generated videos featuring real people. They create a photorealistic AI avatar that emotes naturally and can read any script. Your AI-generated video can be updated at any time, and new virtual videos can be created instantly. You can also select any background.
The creation process is simple: record the person you want to transform into an AI avatar while they read a provided script against a greenscreen. Synthesia’s AI then processes the footage and audio.
To get quality results, the AI needs great footage and pristine sound. That’s where Indigo Productions comes in. We’ll record your reading of the script in our studio space and ensure that you are properly lit so that you look your best both to humans and the AI.
Synthesia is just the tip of the iceberg. In the future, AI will be capable of generating completely life-like virtual video and audio recordings, bringing any character you can imagine to life. Make a big mistake during your shoot? No problem, AI will be able to fix complex audio and visual errors. And if you can’t afford high-end equipment, AI will approximate feature film-level images from phone and webcam footage. The creativity of filmmakers will be completely unleashed and your best ideas will find themselves on screen regardless of your technical expertise. Creating great videos will be cheaper and faster than ever. And marketing departments will benefit as well, with AI tools that can perform comprehensive analysis on who to target and how.
Will AI replace skilled filmmakers?
Artists around the world are wondering if their jobs will be taken by AI. The truth is, professionals who don’t keep up with the times will lose opportunities. But if you want a great video that stands out from the competition, anexperienced production company – and preferably one that uses the very real advantages of AI – is still your best bet.
Production companies provide clients with control over their message, original creative ideas, an aesthetic that matches their brand, and distribution guidance. Viewers want to feel a personal connection with the content they consume, and the human element is irreplaceable. While some companies and videos will get by with AI-generated content, those that truly excel will present a curated and unique brand identity in carefully considered videos.
And even with all these new AI video tools, you’ll still need a team that knows what is available and how to best use the tech.
Contact Indigo Productions
Indigo Productions is a leader in artificial intelligence video production. We use AI to deliver better results faster and with less expense. When we use AI, we don’t let it make creative decisions for us. Instead, we use it to tackle time-consuming tasks and provide a creative jumping-off point. Don’t hire a company that’s behind the times and wastes your resources. If you have video needs or are curious about how AI can improve your video marketing pipeline, contact us today. Visit our website, www.indigoprod.com, or call us at (212) 765-5224.