There is a way, I have forgotten how, when you have your all black landmass on one layer and then a full white layer to make the coastlines jagged. I vaguely remember it has something to do with Posterize or something.

Anybody have any idea what I am talking about?

Cheers!