Well, I have generated coastlines before using clouds and such. But in this case I was trying to replicate a small section of a larger published map from the Tal'Dorei campaign setting. I needed a zoomed-in map of one region from the larger continent. So to generate the coastline, I did a bunch of math to figure out the scale of the original so I could make the scale match on the smaller area, then used that to crop the larger map down to the region I wanted.

Then I traced the edges of the coastline, roads, and rivers in the cropped area in Inkscape -- manually placing nodes and adjusting bezier curves and such -- in order to produce vectorized shapes for the coast, river, and roads. That's why it's so smooth. Once I had the vector graphics worked out, I exported them to black-and-white PNGs, cranking up the size to the desired output resolution. Those black and white images became masks in Photoshop.

I'll experiment a bit and see what I can do about the coasts. Maybe I could play with the smudge tool and spatter filters on the mask? Or maybe I should bite the bullet and generate the coastline all over again using generated clouds and the existing mask as a rough guide.

I am going to continue working on this -- it's going to be used several times in an adventure that I'm running at some conventions this year. I've got a beta playthrough of the scenario planned for this Saturday, then I'll be running it three times at NorWesCon at the end of this month, and 3 times at GenCon next August. So I've got a lot to sort out before Saturday, but there'll be some breathing room after that. It doesn't need to be perfect for the beta test, obviously.