Agreed, but that wouldn't be shown with a drop shadow. Rather it should be shown with a highlight on the earth within the radius of the dirt of the hole. Here, I've cropped out the hole and done a quick light and shade modification. Simon, I really hope you don't mind, but it's easier to show than to describe.

Here's the modified hole with an overlay layer:
Hole.jpg

And here's the overlay I used to create it:
HoleOverlay.jpg

Here you can see the berm and the edges still blend into the texture and tone of the grass around. So the hole looks more integrated into the map.