Good start on this one.

What program are you using to do this? Photoshop or Gimp or some other program?

If your land part is its own layer I can give you a trick to make the coast line(the land part of it) look quite nice.

So for example, how I would start the map is, having a Sea layer(Bottom) a land layer(2nd) Color layer (3d) River layer(4th) A terrain layer-mountains and icons (5th) and a Trees layer (top)

(You can add a line art layer somewhere above the land layer or all the way at the top if you want.)

Then I would go to the land layer, SELECT the landmass.(Alpha to selection on Gimp.) Add a nice texture to the land, like a paper texture or something which give nice detail throw it into your land selection.
Now the trick, keep the land selection, choose the color layer and take a cloud or very random brush and choose black or some sort of dark brown. Lower the opacity waaaay down and start randomly clicking all the way around your coastline. This will only affect the land and not the sea.
Now, if you want it to look more cliff-like shore, then push the opacity just a bit up and paint that section of the coast just a little darker with your random brush.


1 Select land layer (Alpha to selection)
2 Throw in nice texture
3 Keep selection
4 click on layer above land layer and pick RANDOM brush, not as smooth brush.
5 Pick black or dark brown. Change opacity to something low.
6 with the land selection still on start clicking along the coast. lighter for a gentler slope. Darker for a more cliff like appearance.

(The reason why you do it on another layer is if you want to change the land texture it is easier and if you want to correct a mistake it is easier.)

If you understood that, you can try it.
Also, here are some terrain brushes and icons for you.