This PSD has a few layers that, together, let the user paint in a simple Black/White brush, and the jaggedness is applied as you go. Enjoy.

Attached image shows how 3 simple brush strokes can make what looks like a coastline.
psd-jagged.jpg

CG.com isn't letting me upload the ZIP so I put it on my Dropbox. Here's the link:
https://www.dropbox.com/s/c8h2r4ba0c...ronox.zip?dl=0