Answered my own question. Fill the layer mask with black and all is well.