If you’ve been following Creative Curio for the past few weeks, we’ve been discussing masking in Photoshop. Here is the series so far:

• Mastering Photoshop Masks: The Basics
• Mastering Photoshop Masks: Layer Masks
• Mastering Photoshop Masks: Expert Techniques
• Mastering Photoshop Masks: Quick Masks

One thing not mentioned yet, though, is Vector Masks. I’m a little more selective when it comes to using these because you can’t control the hardness of the mask edge like you can with a pixel-based Layer Mask.

When are Vector Masks Preferred?

Vector masks are best used to cut out an object from its background when the object is in clear focus, with hard edge lines. It would be a good solution for a model in front of a white background, for example (you might be thinking you could just use the Magic Wand selection tool for this, but that tool usually doesn’t produce clean results, especially for print work).

How Do You Use a Vector Mask?

If you know how to use the Pen Tool, you know everything you need to create a Vector Mask. If you struggle with the Pen Tool, Veerle produced some excellent practice exercises for working with the Pen Tool a while back. It’s the Illustrator Pen Tool that she talks about, but the Photoshop version works the same.

There are a couple of ways to get a Vector Mask on a layer. One way is to click the New Layer Mask icon twice, this will give you first a Layer Mask and then a Vector Mask. Select the Vector Mask thumbnail from the Layer palette and then the Pen Tool (shortcut P) and set the Pen Tool to Path in the context sensitive toolbar at the top of the Photoshop window. Now you can draw in your vector path.

I find that way can sometimes be difficult (as shown!), so I like to first draw with the Pen Tool set to Path (but not in the Vector Mask). Then with the Select Path Tool (black arrow, shortcut A), select the whole path and cut it (Ctrl/Cmd+X). Then select the Vector Mask thumbnail and paste (Ctrl/Cmd+V). If it doesn’t show up…

Troubleshooting Vector Masks

If your mask isn’t hiding what you want it to, it’s probably because the tool mode is not set properly. Make sure you have the path selected on the layer mask and then click one of the following buttons in the context sensitive toolbar.

Add to Path Area is probably the most common mode, and the one you want if you just have one path in the Vector Mask and you want to cut the object from the background.

Subtract from Path Area will allow you to cut “holes” in the main Vector Mask (here I’ve subtracted a star shape; the green is the background showing through).

Intersect Path Area will only show the area covered by both masks.

Exclude Overlapping Path Area is the opposite of Intersect; it hides the area covered by both masks and shows the area covered by only one of the masks.

If you can’t select the path, try first selecting the Pen Tool (P) or the Path Selection Tool (A). Be sure to also select the proper layer and click in the Vector Mask thumbnail. Now try clicking on the path.

What is the Difference and the Benefit to a Vector Mask vs. a Layer Mask?

A Vector Mask is made with a vector path (infinitely scalable), where as a Layer Mask is pixel-based and cannot be scaled larger without loss of quality. You can also use many different art tools and filters, and have varying levels of transparency within the same Layer Mask, where you do not have these options for a Vector Mask.

In my experience, Vector Masks are faster to create and manipulate than Layer Masks. You can also use these two different types of masks in tandem; if you wanted to quickly cut out an object from its background, but then soften some of the edges, for example.

Photoshop Masking Series So Far

• Mastering Photoshop Masks: The Basics
• Mastering Photoshop Masks: Layer Masks
• Mastering Photoshop Masks: Expert Techniques
• Mastering Photoshop Masks: Quick Masks
• Mastering Photoshop Masks: Vector Masks (current post)

Do you ever use Vector Masks? How? When do you think is the best time to use them?

As always, if there’s something I missed or didn’t explain clearly enough, please ask about it! I’d be happy to clarify things in the comments.

Have you subscribed to Creative Curio yet?