Selections & Masks in Photoshop-- Part II

Article and Photography by Ron Bigelow

www.ronbigelow.com

Photoshop CS or Photoshop CS2 Used in this Tutorial

Selection: Color Range Tool

Another popular selection tool is the Color Range tool. The Color Range tool works on images where the selection is based on color.

Figure 1: Color Range Dialogue Box

Figure 1 shows the Color Range tool (Choose Selections/Color Range). On the right side of the dialogue box are three eyedroppers:

The Invert box option allows the selection to be inverted (i.e., the colors that are clicked are not selected). The Fuzziness slider determines the range of colors that will be selected. A low Fuzziness setting will cause the Color Range tool to select only colors that are very similar to the colors that were clicked. A high Fuzziness setting will cause the Color Range tool to select more colors. The Fuzziness box allows a value for the Fuzziness setting to be typed. Below the image box are two options: Selection and Image. The Selection option (the option that will be used for most photographic work) displays the selection in the image box. The Image option displays the image in the image box.

Figure 2: Color Range Select Option
The Select Option (see Figure 2) allows colors to be selected based on different criteria. For example, the tool can be used to select the Reds or the Out of Gamut colors. However, for most photographic work, the Sampled Colors option will be used the most. This option allows colors to be selected by clicking on the desired colors in the image.
Figure 3: Color Range Selection Preview Option

The Selection Preview Option (see Figure 3) determines how the image will appear.

Figure 4: Image that Needs Color Adjustments to the Grass
Figures 4 shows an image with rolling hills covered with green grass. It is desired to edit the color of the grass. A selection is needed that can be added to a Color Balance layer.
Figure 5: Image with Color Range Tool Selection
The selection is made by choosing the eyedropper tool and clicking on the grass (see Figure 5). Depending on the image, it may be necessary to use the Add to Sample and Subtract from Sample eyedroppers to modify the colors that are selected. The Fuzziness slider is also used to adjust the colors that are selected.
Figure 6: Layers Palette
Figure 6 shows the Layers palette after the selection was blurred and added to the Color Balance layer. Figure 7 shows the final mask.
Figure 7: Final Mask
Like the Magic Wand tool, the Color Range tool works well with images, such as this one, where the color of the object to be selected contrasts with the surrounding colors. If this is not the case, another selection tool should be utilized.

Messy, Messy, Messy

Don't you wish that everything in Photoshop was easy? For instance, we could tell Photoshop, "I want this color, and this color, and this color. However, I don't want any other colors". Then, we would get exactly what we wanted! Well, perhaps this will be in a future version, but it isn't that easy today. With any complicated selection, it is almost a guarantee that the initial selection will not be perfect. Instead, some areas that are desired will not get selected, some areas that are desired will only get partially selected, and some areas that are not desired will get selected anyway (at least partially). Figure 7 is a perfect example. It was desired to select all of the grass and none of the other areas, but the mask tells a different story. Some of the grass, which should be completely selected, is only partially selected (indicated by the fact that these areas are gray instead of pure white). Some of the non-grass areas, which shouldn't be selected at all, were partially selected (indicated by the fact that these areas are gray instead of pure black).

In order to improve the selection, some clean-up work is usually necessary. For this, clean-up tools are required. This is an appropriate place to introduce these tools. While the Marquee tools, the Lasso tool, and the Polygonal Lasso tool are often used on very simple selections that do not need much, if any, clean-up, the Magnetic Lasso, Magic Wand, and Color Range tools are often used for less simple selections and result in selections that need further work before they are useable. Furthermore, the tools that have yet to be introduced in this series are more advanced selection tools that are usually used with more challenging selections. As a result, clean-up becomes more important with these advanced tools.

Selection Clean-Up: Lasso Tool

Figure 8: Image with a tough Selection

Figure 8 shows an image that requires a selection that is challenging. It is desired to edit the sky but not any of the rest of the image. The problem is that the sky and the waves are not easily separated. Any tool that will select at least part of the sky will, likely, also select some of the wave area (at least partially). This is especially true on the right side of the image where the sky and ocean become increasingly close in both color and tone. In addition, in many places, the edge of the wave is actually defined by spray from the wave. These areas have a very poorly delineated edge. This makes the selection even more challenging.

Figure 9: Initial Selection Made with the Color Range Tool
Figure 9 shows a selection that was made with the Color Range tool. Figure 10 shows how a mask made from this selection would appear.
Figure 10: How a Mask Made from the Initial Selection Would Appear
There are many problems with this selection: the area around the sun has not been selected, part of the ocean and much of the beach has been selected, and the sky area just above the waves on the right side of the image has only been partially selected. Considering that a selection is needed that cleanly separates the sky from the rest of the image, this is not usable in its current state. Essentially, what the Color Range tool has done is provide the beginnings of a selection along the edge of the wave. However, some clean-up is needed to make this selection usable.
Figure 11: New Sky Selection with Lasso Tool

One of the easiest ways to clean-up large areas that need to be either completely selected or completely non-selected is to use the Lasso tool. Figure 11 demonstrates the results of this approach. To capture most of the sky in the selection, the Lasso tool was used to draw around the sky area with the Selection option set to Add To. This ensured that most of the sky was selected. The selection was made as close to the edge of the wave as possible without actually touching the wave.

Figure 12: What the Mask Would Look Like after Editing the Sky
Figure 12 shows what a mask would look like after the Lasso Tool was used to clean-up part of the sky. The sky now looks much improved. However, the non-sky areas still need to be edited. This can be done by using the Lasso tool again.
Figure 13: New Selection after the Non-Sky Areas were Removed with the Lasso Tool
Figure 13 demonstrates the next step. To remove most of the ocean and the beach from the selection, the Lasso tool was used to draw around the non-sky area with the Selection option set to Subtract From. This ensured that most of the non-sky area was deselected. This time, the selection was not made as close to the edge of the wave as possible. It is desirable to leave a little bit of room along the edge of the wave so that a transition region can be painted in by hand.
Figure 14: What the Mask Would Look Like after Removing the non-Sky Areas

Figure 14 shows what a mask would look like after the Lasso Tool was used to clean-up part of the non-sky areas. The mask is much improved compared to the original mask shown in Figure 10.

It is important to analyze the mask at this point. It can be seen that most of the sky is totally white, and most of the ocean and beach is totally black. This is as desired. However, along the edge of the wave, some more work needs to be done. There is a gray area just along the edge (this area was selected by the Color Range tool). This gray area is bordered on the ocean side by the rather harsh edge of the black area. This transition is too harsh. In addition, some of the sky just above the wave is partially selected (especially on the right side). This needs to be removed. Lastly, much of this gray area needs to be darkened. For this clean-up, other tools are needed.

Selection Clean-Up: Brush Tool in Quick Mask Mode

Figure 15: Selection in Quick Mask Mode

In order to better examine the selection, the image is moved into the Quick Mask Mode (click on the Edit in Quick Mask Mode on the Tools palette). Figure 15 shows the image in Quick Mask Mode. To make it easier to see which colors are selected and which are not, the Quick Mask Mode color was changed to yellow by double clicking on the Edit in Quick Mask Mode icon on the Tools palette and clicking on the Color box.

Figure 16 shows a close up of a section of the mask in Quick Mask Mode. It can be seen that part of the wave has been only partly selected. This can be edited in the Quick Mask Mode with the Brush tool.

Figure 16: Close up of Selection in Quick Mask Mode
Figure 17: Tools Palette
The Brush tool is found on the Tools palette (see Figure 17). Figure 18 shows the Options bar for the Brush tool.
Figure 18: Options Bar for Brush Tool
The Master Diameter option sets the width of the brush. The hardness setting sets the hardness of the edge of the selection. A high hardness setting creates a selection with a hard edge. A low hardness setting creates a selection with a soft edge. The Mode option determines the Blend mode for the selection (the Normal Blend mode will be used for most photographic work). The Opacity setting determines how much "paint" the brush lays down. The larger the Opacity, the more paint that is put down. The Flow setting determines how fast the paint is put down. The larger the Flow, the faster the paint is put down.
To help clean-up the mask, the brush is used to paint white on the selection where areas need to be selected (i.e., the sky) and black (it will show up as yellow on this Quick Mask) where areas should not be selected (i.e., the non sky areas). To do this, the foreground color needs to be set to white and the background color set to black (click the Default Foreground and Background Colors icon on the Layers palette). Now, it is simply a matter of painting on the mask. It is better to use a soft edged brush when painting on the ocean side of the wave. This will make the transition from the edge of the black area to the edge of the wave more gradual. A harder edge brush can be used on the sky side of the edge. For this mask, the Opacity and Flow both are set to 100% (other photographers may prefer to set a lower Opacity and make multiple passes). The Brush tool can be switched from white to black (and visa-versa) by clicking on the Switch Foreground and Background Colors icon on the Tools palette.
Figure 19: Selection in Quick Mask Mode after Painting in Quick Mask Mode
Figure 19 shows the mask after the painting in Quick Mask Mode. Figure 20 shows a close up of the edited mask in Quick Mask Mode.
Figure 20: Close up of Selection in Quick Mask Mode after Painting in Quick Mask Mode
The selection in Figure 20 is improved over that in Figure 16. However, as can be seen, the mask is not yet perfect. There are still issues along the edge of the wave.

Mask Clean-Up: Brush Tool (Overlay Mode)

Figure 21: The Mask

As seen in Figure 20, the edge of the wave is still a major problem. Any mask made from this selection would, likely, be very noticeable in the image. So, more clean-up is necessary. To proceed, the image is moved out of the Quick Mask Mode (click on the Edit in Standard Mode on the Tools palette), and the selection is turned into a mask (select the layer that is to receive the mask and choose Layer/Layer Mask/Reveal Selection).

Clicking on the mask thumbnail while holding down the Alt key brings up the mask on the screen. Figure 21 shows the mask. Figure 22 shows a close up of a section of the mask.

Figure 22: Close up of the Mask

Now, the mask can be edited directly. For this edit, the Brush tool is selected, the Opacity is set to an appropriate value (this will vary with the image and some experimentation will be necessary to determine the best value), and the Blend Mode is set to Overlay. With the foreground color set to white and the background color set to black, the edge of the wave is painted over with the brush. The edge is painted multiple times alternating the Brush between the background color (black) and the foreground color (white). I have found that it is usually best to start with black. In other words, one pass with a black Brush, one pass with a white Brush, another pass with a black Brush, another pass with a white Brush, and so on.

The secret here has to do with the Overlay Blend Mode. The Overlay Blend mode screens (lightens) the light areas and multiplies (darkens) the dark areas. So, when the brush is white, it will make the light areas even lighter, but it will have much less of an affect on the dark areas. When the brush is black, it will make the dark areas even darker, but it will have much less of an affect on the light areas. Thus, with each successive pair of black/white Brush passes, the contrast increases as the lighter shades progress toward pure white and the darker shades progress toward pure black. For this mask, it is important not to go too far. The edge of the wave should not be too sharp.

Figure 23: The Mask after the Overlay Brush
Figure 23 shows the mask after the Overlay Brush. Figure 24 shows a close up of the mask at this point.
Figure 24: Close up of Mask after the Overlay Brush

Mask Clean-Up: Brush Tool (Normal Mode)

If any undesired artifacts remain in the mask, they can simply be painted over with the Brush set to the Normal Blend Mode. A white brush is used to eliminate undesired dark areas and a black brush is used to eliminate undesired white areas.

However, at this point, this mask does not need any touch up with the Brush Tool in the Normal Blend Mode.

Mask Clean-Up: Blur Tool

Figure 25: Close up of Image before Blurring the Mask

Often, no matter how carefully a mask is made, the edges in the mask will be too sharp. This will often be noticeable in an image. Thus, for many masks, the final step is to blur the edges in the mask.

Currently, the mask is open on the screen. While it is possible to directly blur the mask. It is usually better to move back to viewing the image (click on the mask thumbnail while holding down the Alt key). This approach allows the photographer to view how the blurring of the mask is affecting the image.

A close up of the current image (see Figure 25) shows an edge that could be improved by blurring the edge in the mask to produce a more gentle transition from the masked to the unmasked areas (please note that Figures 25 and 28 show close ups of the image, not the mask, and that an extreme Curves layer was added in these figures to brighten the sky in order to make the effects more noticeable). The Blur tool will solve this problem.

Figure 26: Tools Palette
The Blur Tool is found on the Tools palette (see Figure 26). Figure 27 shows the Options bar for the Blur tool.
Figure 27: Options Bar for Blur Tool
The Master Diameter option sets the width of the brush. The hardness setting sets the hardness of the edge of the blur. A high hardness setting creates a blur that doesn't taper off at the edge of the tool. A low hardness setting creates a blur that tapers off at the edge of the tool. The Mode option determines the Blend mode for the selection (the Normal Blend mode will be used for most photographic work). The Strength setting determines how much the tool blurs an image. The Sample All Layers option determines which layers are blurred. When the option is checked, the Blur tool will base the blur on all of the layers. Otherwise, the blur will be based on the active layer only.

To blur the edges in the mask, the Blur Tool is applied to the appropriate area in the image. In other words, the blur is simply painted on (if the Blur tool acts on the image instead of the mask, undo the blur, click the mask thumbnail, and reapply the Blur tool).

Figure 28: Image after the Blurring of the Mask

Figure 28 shows the image after the blurring of the mask.

Mask Clean-Up: Final Touch-Up

At this point, the mask should be closely examined one more time. If any problems still exist, they can be touched up by painting directly on the mask with the Brush. However, this mask does not need any further work.

Articles

Selections -- Part I     Selections -- Part III