Using Channels to separate an object from its background

[Midgardsormr]

The topic of channels came up recently, and I alluded to the use of the color channels to cut an object off its background. Since I have to do that very thing today, I thought I might post a tutorial on the process.

I intend to use this rocky arch in a matte painting I am working on:
Arch.jpg
This image comes from CGTextures.com, and I am working with the highest resolution file they have there.

My painting is a mountain scene at night, so I want to get rid of the water and sky. While I could attempt to paint a mask around the object, there's an easier and more accurate way to get it done using Photoshop's Channels palette.

As I look at this picture, I can see that most of the information I want to get rid of is blue. I can easily verify that by looking at the blue channel by itself. Sure enough, in this channel, the sky is entirely white, and the water is pretty light, as well. The rock, on the other hand, is quite a bit darker. The rock is lightest in the red channel, which may be useful later on, but for now we'll stay in the blue and see what we can do with it.
1.jpg

The first step is to make a copy of the blue channel. We don't want to manipulate the original blue channel because that will change the way the image looks. All we want is to create a useful matte. Drag the blue channel to the "new" icon at the bottom of the palette
2.jpg

Now you can do whatever you want to the new channel without fear that you'll mess up your image.

An ideal mask is mostly black and white—the white areas are the parts of the image you want to see, and the black areas will be hidden. Obviously, then, this channel is reversed. Image > Adjustments > Invert (Ctrl-I) will fix that. It will also probably make it easier to see the small amount of detail in the sky. That's going to be the first thing to go.

While you're in this channel view, all of the image adjustments and filters you can do to a greyscale layer you can also do to a channel. Unfortunately, you can't apply them non-destructively with an adjustment layer, so any time you want to make sure you can get back to a particular look, you'll want to make another copy of the channel. That being the case, it may not be a bad idea to rename the channel. I'm going to call mine "matte." Simply double-click on the channel name, and you'll be able to type in a new one.

Now, the first thing to do is to try to exaggerate the difference between the black and the white areas. Levels, Curves, or Brightness/Contrast will all work for this. I like Image > Adjustments > Levels (Ctrl-L) in this case because it doesn't require as many button-pushes as Curves, but it allows control over the mid-range values, unlike Brightness/Contrast.

The first thing you want to do is to get rid of that detail in the sky. Pull the left-hand control of the Input Levels up until the detail just goes away. Don't go any further than that, or you'll lose the tops of the rock.

Then play with the left-hand and middle controls to refine the edges a bit. You want to increase the contrast between the rock and the sky as much as you can without losing any detail. Don't worry about the transition to the water at this point; we'll address that later. I wound up with my levels at 26, 1.66, and 90:
3.jpg

That gives me a pretty good matte for the sky, but there's still a lot of detail in the rock. If I leave it as it is, parts of my arch will be translucent or invisible. Fixing that is a simple matter of painting white across the arch. Use the magic wand tool to select the sky, and then shift-click the portion of the sky under the arch (holding shift tells Photoshop that you want to add to your selection instead of replacing it.) Now Select > Inverse (Ctrl-Shift-I) to invert the selection so that the sky area is protected. Use the Brush tool (B) to paint white over all the rock areas. (To quickly get to the default colors, hit D. While you are in the channels or a mask, white is the default foreground color.)

It looks like my Levels adjustment wasn't as good as I thought it was. I lost some of my clifftop because it was very brightly lit.
4.jpg

Stay tuned for the next post, and I'll show you how to fix that, and pull the matte for the water.