Oh right, well I made a seamless pattern first using this tutorial www.cartographersguild.com/showthread.php by Vhey.
I'm not entirely sure what size I went with but I think something like 100x100 pixels.
Then I drew a couple of horizontal lines at at regular intervals and drew the waves along those lines.
Then I deleted the horizontal lines leaving only he waves.
Then I just followed the tutorial to create a seamless pattern.
After that I select my land layer and expand the selection with an x amount of pixels (depending on the size of the map) then feathering the selection and filling it with the made pattern.
Sometimes I fill the layer with white and set the layer fill to 0%.
Then I add the pattern as a layer style > pattern overlay. This allows me to change the size of the pattern just by changing it in the layer styles.

I'm not sure if the is an official term for it, but I always call them coastal waves (: