Inpainting

When an area of an image is missing, inpainting techniques can be used to fill in the gaps. The term "inpainting" comes from art restorers, who often need to fill in parts of a painting that have cracked or flaked off over time. Digital inpainting techniques can be used for all kinds of repairs, such as removing text from an image, erasing powerlines from a scenic view, or repairing cracks and scratches.

There are a wide variety of inpainting techniques, all developed in the last few years. One of the simplest is called diffusion-based inpainting. "Diffusion" is the process by which gas spreads out to fill any volume. For example, when a bottle of perfume is opened, even in the absence of any wind, the smell soon spreads through the whole room, though it is most intense near the bottle. To inpaint, then, we can allow the colors to diffuse into the missing areas of the image.

Another way of putting it is that we want to blur the colors out into the missing areas. Mathematically, repeated blurring and diffusion are identical. The technique is known as "convolution." When an image is blurred, the colors of each pixel are averaged with a small portion of the color from neighboring pixels. That pixel, in turn, contributes a small part of its color to each of its neighbors. The image to the left shows color that has diffused from pink and blue spots.

I created an action (a macro, or small program within another program) in Photoshop to perform diffusion-based inpainting. This is essentially the same as programs others have already written, but to my knowledge mine is the only inpainting tool that can be used directly in Photoshop. It starts by deleting the region to be inpainted, creating a transparent area. Then it blurs this region repeatedly, bringing in a small amount of opaque color from around the edges. Periodically it creates a new layer and gradually shrinks the area to be inpainted until it has shrunk away to nothing. The results can be seen in the image at the top of this page. When the cursor rolls over the image, the red areas are replaced with their inpainted versions. You can note minor errors where the inpainted region crosses an edge, or where the red area is particularly wide.

To the left is a more extreme example of the problems with inpainting an extremely wide area. There are several ways these problems could be improved. The finished product could be sharpened or deconvoluted (de-blurred). Care can be taken to extend the curvature of lines across the inpainted area. The technique can be combined with statistical texture generating tools.

Another idea I am exploring is using this technique to create natural media watercolor painting tools. You could paint an area of an image with "water" so that when it was painted with pigment the pigment would gradually diffuse throughout the wet area as you worked. It could interact with the paper texture or the brush shape to more accurately simulate painting.

If you have Adobe Photoshop, you can try this inpainting macro out on your own images. Save the File below in your Photoshop actions directory, Adobe/Photoshop 7.0/Presets/Photoshop Actions/. Open Photoshop, and in the Actions pallete, choose Load Actions. The file is called diffusion.atn. Then select the area of your image you want to be inpainted (there is no need to color it red like I did in these images). Make sure that "feather" in the selection tool is set to zero pixels. This ensures that each pixel is either fully selected or not selected. The area can be no more than about fifteen pixels across in any direction, and it really shouldn't be more than about four pixels across for good results. When the area is selected, choose the action diffuse2 and press play. Continue pressing play repeatedly until the selection disappears, and an error message comes up saying "no pixels were selected." Then you will need to flatten the many layers created by this process.

diffusion.atn