You may have to fiddle with the settings for both, the Gaussian blur and Threshold to get the desired results. It's very dependent of the size/resolution of your map. It also helps to repeat the process with different settings. I took your picture, loaded it into GIMP and applied the following:
HSV noise (1, 0, 0, 255)
Fade HSV noise (darken only)
Gaussian Blur (5)
Threshold (168 )
and this is the result, a simplified, less crumbly and therefor more believable coastline, for continents at least.
Less Crumbly Coasts.jpg
If this was going to be the land shape for a regional map or an island, I would run the process again to get a more distinctive coastline, but for a world map I'd say this is fine.