The only advice that I can give you about which parts should be smoother and which crinklier is that (here on Earth, at least) it's entirely a product of natural forces such as tectonics, erosion, and sea level. How things appear on your map is further complicated by scale because very crinkly details in areas may disappear, while what would be relatively smooth areas locally might have significant detail when zoomed out.