Hand brainstorming, really, all of it. It helps to revisit the coastlines on and again instead of trying to accomplish a finished look at the first attempt. It also helps to work with a big file originally. Knowing the tectonics of your continent is definitely a good idea to make the coastline more realistic, as tectonics tells you natural places for peninsulas, smooth coasts, islands chains, inland seas, etc.
And finally, but probably more important than everything else, it helps to know Earth. I mean, staring at maps and google maps for ages seems like a pointless thing, but the back of your mind is storing references it will fetch later when is being creative.