The colored sea contours are typically done manually. I think the CC3 tutorials show this, but it's been a while since I've done much with CC3, I will admit. If you can afford the annuals for CC3, they have a great deal of information on various styles, including one similar to this (again, it's been a while).

I suposed I could code up a filter that would do these sorts of things automatically (sort of like the inverse of the lighted bevel mountains trick), but I have such a long list of things that I'm behind on for ProFantasy that I'm not sure another one would help.