Using Channels to separate an object from its background

[Midgardsormr]

The topic of channels came up recently, and I alluded to the use of the color channels to cut an object off its background. Since I have to do that very thing today, I thought I might post a tutorial on the process.

I intend to use this rocky arch in a matte painting I am working on:
Arch.jpg
This image comes from CGTextures.com, and I am working with the highest resolution file they have there.

My painting is a mountain scene at night, so I want to get rid of the water and sky. While I could attempt to paint a mask around the object, there's an easier and more accurate way to get it done using Photoshop's Channels palette.

As I look at this picture, I can see that most of the information I want to get rid of is blue. I can easily verify that by looking at the blue channel by itself. Sure enough, in this channel, the sky is entirely white, and the water is pretty light, as well. The rock, on the other hand, is quite a bit darker. The rock is lightest in the red channel, which may be useful later on, but for now we'll stay in the blue and see what we can do with it.
1.jpg

The first step is to make a copy of the blue channel. We don't want to manipulate the original blue channel because that will change the way the image looks. All we want is to create a useful matte. Drag the blue channel to the "new" icon at the bottom of the palette
2.jpg

Now you can do whatever you want to the new channel without fear that you'll mess up your image.

An ideal mask is mostly black and white—the white areas are the parts of the image you want to see, and the black areas will be hidden. Obviously, then, this channel is reversed. Image > Adjustments > Invert (Ctrl-I) will fix that. It will also probably make it easier to see the small amount of detail in the sky. That's going to be the first thing to go.

While you're in this channel view, all of the image adjustments and filters you can do to a greyscale layer you can also do to a channel. Unfortunately, you can't apply them non-destructively with an adjustment layer, so any time you want to make sure you can get back to a particular look, you'll want to make another copy of the channel. That being the case, it may not be a bad idea to rename the channel. I'm going to call mine "matte." Simply double-click on the channel name, and you'll be able to type in a new one.

Now, the first thing to do is to try to exaggerate the difference between the black and the white areas. Levels, Curves, or Brightness/Contrast will all work for this. I like Image > Adjustments > Levels (Ctrl-L) in this case because it doesn't require as many button-pushes as Curves, but it allows control over the mid-range values, unlike Brightness/Contrast.

The first thing you want to do is to get rid of that detail in the sky. Pull the left-hand control of the Input Levels up until the detail just goes away. Don't go any further than that, or you'll lose the tops of the rock.

Then play with the left-hand and middle controls to refine the edges a bit. You want to increase the contrast between the rock and the sky as much as you can without losing any detail. Don't worry about the transition to the water at this point; we'll address that later. I wound up with my levels at 26, 1.66, and 90:
3.jpg

That gives me a pretty good matte for the sky, but there's still a lot of detail in the rock. If I leave it as it is, parts of my arch will be translucent or invisible. Fixing that is a simple matter of painting white across the arch. Use the magic wand tool to select the sky, and then shift-click the portion of the sky under the arch (holding shift tells Photoshop that you want to add to your selection instead of replacing it.) Now Select > Inverse (Ctrl-Shift-I) to invert the selection so that the sky area is protected. Use the Brush tool (B) to paint white over all the rock areas. (To quickly get to the default colors, hit D. While you are in the channels or a mask, white is the default foreground color.)

It looks like my Levels adjustment wasn't as good as I thought it was. I lost some of my clifftop because it was very brightly lit.
4.jpg

Stay tuned for the next post, and I'll show you how to fix that, and pull the matte for the water.

[Ascension]

I've always hated hand-drawing a mask around an image pixel by pixel and the other masking techniques have mixed results for me. You can expect some reppage from me once I see how this works Thanks very much.

[Midgardsormr]

I started in on using the channels to fix this, but ultimately it's just easier to use the brush tool to clean up that edge. A better Levels adjustment might have made it unnecessary, though.

Now, on to the water. The difference between the water and the rock is not as pronounced in the water as it is in the sky, so we're going to explore a new function: Image > Calculations.

This brings up a new dialog where two channels can be combined arithmetically to produce a third with different properties. As noted before, the rocks have a lot of information in the red channel, and the water has a lot in the blue, so those are the two channels you should experiment with.

Spend some time choosing different blending modes and turning the Invert box on and off in each one. You're looking for a combination that accentuates the contrast between water and rock without destroying the detail of the edge.

In this image, you can see that I've chosen to combine the channels with the Overlay mode, and the blue channel inverted. It looks like I may have done well to try this before working on the sky matte—I might have been able to do the entire matte in one go. You'll notice, though, that using the red channel has added a lot of noise to the sky, so that matte might not have been as clean as I could have gotten had I worked the sky matte a bit more.
7.jpg

Now you can play with Levels and make the water matte the same as you did the sky matte. Here's my finished product:
8.jpg

Now make a Marquee selection (M, then draw a box) around the bottom part of the sea matte channel. Don't let the top of the marquee include any uncertain parts of the rock. Copy (Ctrl-C), switch to the matte channel, and Paste (Ctrl-V). You have finished pulling your matte.

Ctrl-click the little thumbnail in the channels list, switch back to Layers, click on your original layer to get the RGB image back, and Layer > Layer Mask > Reveal Selection. Or just click the "Add Layer Mask" button at the bottom of the Layers palette.

All done! If you now save as a PNG, you'll have an object with transparency. The mask acts as a fourth channel, which indicates which pixels should be transparent, like so:
Arch.png

[Midgardsormr]

This was a fairly simple image to manipulate, but the technique works quite well for images that have very fine detail on the edges, especially hair. It takes some work and experimentation, but it can be done. If you do have very fine detail, it may be better not to clamp the levels quite so hard as I did here, as a bit of translucency is desired in that case.

One other technique you can try is to use the dodge and burn tools on the edges of the matte. Set the Dodge tool's range to "highlights" and the Burn tool's range to "shadows", reduce the Exposure to around 20%, and brush each tool lightly over the edge. Since you've restricted the range, the dodge will act only to lighten the lights, and the burn to darken the darks. You can get a more subtle edge with this method, though if you apply it several times, you'll wind up with the same kind of 1-bit (two values: black or white) matte I just demonstrated.

[ravells]

Excellent tut, Mid. If you do a lot of extraction work, I cannot recommend 'Fluid Mask' enough. It makes the whole process a breeze and plugs into PS. It's not cheap at £100 though.

It took less than two mins to do the cut out of your cliff image. It's also brilliant with hard to do stuff like hair and fine lattices.

Cheers

Ravs

[Midgardsormr]

Excellent tut, Mid. If you do a lot of extraction work, I cannot recommend 'Fluid Mask' enough. It makes the whole process a breeze and plugs into PS. It's not cheap at £100 though.

Yikes. I'd have to be doing a heck of a lot of extraction to make that worthwhile. And getting paid for it, which I'm not at the moment.

Incidentally, users of Photoshop CS4 may have noticed the absence of the Extract filter, which is also useful in performing this kind of work. Apparently Adobe is under the (erroneous) impression that nobody uses it, so they removed it from the standard installation. It can still be found in an extras folder on the program disc, though.

[Gandwarf]

Man, you are all masters of Photoshop

I am still struggling with The Gimp... and seeing the stuff you guys are using makes me jealous
Cutting out cliffs like that in 2 minutes and brilliantly at that... ack!

[RobA]

Man, you are all masters of Photoshop

I am still struggling with The Gimp... and seeing the stuff you guys are using makes me jealous
Cutting out cliffs like that in 2 minutes and brilliantly at that... ack!

Using channel separations to set up masks works almost exactly the same way in Gimp.

Speaking of which, I found a handy script Lasm's Channel Extract that will extract any/all of the channels (RGBA, HSV, CYMK, LAB) into separate B&W layers. I find it handy to quickly eyeball the best channel for separation. Here is a mosaic of the channels:
all_channels.jpg

-Rob A>

[Redrobes]

Yup, I use channel separation all the time to generate alpha masks and I was going to point out CMYK too as alternatives to get the best contrast so kudos again to Rob for pointing out a nifty way to see them all at once. After a while you just instinctively know whats going to be the best based on the colors in an image tho.

Another tip I could just shove in at the last minute which you might have mentioned cos I didn't read it all that thoroughly.... When you have your greyscale image from the separation, I used to use brightness and contrast to do the black and white range expansion but I find now that I get a dropper out and check the brightness just at the point where I want to separate it and then use a color curve and draw a box filter right at that point. Save faffing about trying to get the value right.

I'll dish some rep in a mo...

[Steel General]

Great stuff Midgard, have to give this a whirl at some point.

Might be worthwhile to combine the steps into a .pdf and add it to the first post.