Today’s Question: What’s your opinion on the use of vector versus layer masks in general photo editing [in Photoshop], and also the potential utility of combining both types of masks on a single layer?
Tim’s Quick Answer: In most cases I would say that only layer masks (and not a vector mask) should be used in the context of a photographic image. However, there are certainly situations where a vector shape can serve as a good starting point for a layer mask or a vector mask.
More Detail: In a very general way, you can think of a layer mask as being composed of pixels (just like a photographic image) and a vector mask as being composed of a shape (such as a font or a geometric shape). One of the things that makes vector graphics unique and useful is that they can be scaled infinitely without reducing the quality of the shape.
That is because a vector is defined by angles and curves rather than specific pixels. A good example of a vector shape would be the letters in a font. You can resize actual text without reducing the quality of that text, whereas resizing a raster image (such as a photo) will cause some reduction in quality.
The reason layer masks (rather than vector masks) are generally best for photographic images is that the shapes found within photographic images are generally not as smooth an precise as a vector shape. Thus, a bit more flexibility is often necessary.
That said, vector masks can be feathered just like layer masks, so it is not unreasonable to use a vector as the basis of a mask. In some cases, such as when you want to use a mask in the form of text, a vector mask makes perfect sense. In other cases you may find that a vector (such as those you can create with the Pen tool in Photoshop) provide an excellent starting point for a vector mask or even a layer mask.
But again, in general I would suggest that layer masks are most valuable in the context of photographic images, and that vector masks are helpful in a relatively small percentage of situations.