It's a lighted distance field from the wavy edges of the mountains. Your software might call it any number of things (Bevel is a popular term), but the idea is that the software calculates the distance from the edges of the mountain area, treats that distance as a height value, calculates an illumination result based on the slope of the local height data and the lighting direction, and then modulates that illumination by any existing colors to get the final result.

If you're using a height field editor, then the distance field may be referred to by terms like SDF (signed distance field), EDM (Euclidean Distance Metric), or Mound. You'd blend that result to your existing height field and shade the result in whatever way your software allows.