One way to avoid recognizable coastlines when using real-life maps is to overlay one on top of another, and where the coastlines intersect randomly choose which one to follow. That way your "fictional" coast includes random features from two different real world ones.