Using content-aware fill in Adobe After Effects

I previously discussed the latest video editing tools and features from the team at Adobe as it continues to leverage the power of artificial intelligence (AI).

AI was leveraged in photo editing where repeatable tasks, like selecting a subject in an image, could be handled at the click of a button.

Now, After Effects has the ability to leverage one of those key AI-driven features in video with content-aware fill. It’s not a one-click option, but it is still much faster than the manual way of doing things.

How to use content-aware fill in video projects

Let’s dive into After Effects, also known as the “Photoshop of video,” and take a look at this new feature.

Create a mask

In your video footage, locate the item you want to remove from the scene. This can be a static object or something moving. Create the mask by selecting the pen tool and drawing out the mask around your object. You could use other selection tools such as the ellipse, but the pen tool gives you more flexibility with difficult items to mask.

By default, when you create a mask in After Effects, the mask type is an “add.” This means that everything outside of the mask boundary will be “masked” and covered up. Notice how my screen is now filled with empty pixels because of this type of mask (Figure A).

When using the content-aware fill, you’ll want to consider the process as “filling in a hole.” So instead of having your mask set to “add” in mask properties, change it to “subtract” (Figure B). This will remove the object you selected from the scene. You can also adjust the mask feather if you want a softer edge around the area that you just removed. This makes blending new pixels look better in most cases.

Prepping the AI

Now that you created the mask to remove your object of interest, you need to allow Adobe Sensei, Adobe’s branded AI engine, to analyze the footage. You must address the following questions:

1. The object you’re wanting to move, is it stationary?
2. Is it stationary, but the camera moves?
3. Is it a moving object?

If there’s any movement regarding the object that’s masked out, you will need to add motion tracking to the mask. This is as easy as a right-click in After Effects, or just go to the Tracking panel, click one time on your video layer, then click on the play button to begin the mask tracking.

The mask will change position in your footage as it tries to stay locked in the object you’re attempting to remove. If the mask loses a good lock during the tracking, just stop the tracker, move the mask with your mouse, and then resume tracking. After Effects does a pretty good job of keeping the tracking smooth and locked on. While tracking, After Effects creates keyframes behind the scenes on the layer’s mask properties (Figure C).

Once the tracking is completed, access the new Content-Aware Fill panel. If you don’t see this panel on your screen, simply go to the Window menu and select it from there. It will then dock itself into a panel of your After Effects interface.

When you access this panel, notice the different options. The Alpha Expansion gives you a preview of the masked area. (Personally, I don’t find it very useful as I get a larger view of the masked area in the actual program monitor.)

Next, there are a couple of drop-down selections to consider. First, the fill method has options you can choose. The Object and Surface options are really similar. The Object option (Figure D) is more for removing…objects. Stationary objects are ideal, but it works for moving objects, however, render times can be slower. It even works for removing logos on items such as clothing.

The Surface option is ideal for removing objects from surfaces that are moving. Think about a vehicle going down a street. You can remove the vehicle, but the camera will continue to pan. This will also encompass everything the Object option handles.

The Edge Blend option is ideal for textured surfaces that may need to be filled in such as removing a soccer ball from a lush, grassy field. I don’t recommend using this option in the same way that you’d use the Object or Surface options.

Fill the gap

Finally, fill in the footage hole with some AI-generated footage. When you click the Generate Fill Layer button (Figure E) in the Content-Aware Fill panel After Effects will analyze the pixels around the mask. It will even analyze the pixels as the camera or object moves in space to figure out what the masked area should look like.

Depending on your footage details (SD versus HD versus UHD versus 4k), duration, and computer hardware specs, the processing times will vary. Sometimes it’s under 60 seconds. Sometimes a render can take a couple of minutes.

Upon completion of the render, After Effects will create an additional layer in the layer panel (Figure F), which is placed above your footage layer. This will fill in the missing pixels created by your mask. It does a really nice job, but sometimes it may miss on fine details. Especially in frames with movement.

Using the Create Reference Frame option can assist with any inconsistency. This option opens up Photoshop and allows you to use the clone stamp tool to fill in the empty pixels of your mask. Once filled in, save the file in Photoshop, and then close Photoshop. This will take you back to After Effects with the reference framed placed at the bottom of your layer stack. After Effects’s AI will use that reference frame to assist with generating the missing pixels (Figure G).

It’s magical

The content-aware fill feature in After Effects is pretty impressive. It’s not perfect yet, but it’s pretty close. Some other items to consider when using this application, and its new feature includes:

• System hardware and RAM. My main rig has 32GB of RAM, and the content-aware fill option pushed my RAM usage to 28GB each time. The more RAM you install, the better the performance
• Your footage will also play a factor in render speeds. High-resolution files such as UHD or 4K take longer to render versus HD or SD. If your clip is longer than ten to 15 seconds, I recommend splitting the clip to smaller chunks and then adding the content-aware fill.