The lazy solution is to extend your mask coast contour down by about two color levels on your map (a continental shelf is approximately the edge of the continent that just happens to be submerged). The next level below that would be the abyssal plain. Make sure that the edge of your selection hits on the boundary between continental shelf and abyssal plain. You'll need to reapply the main coastline clipping to restore your original coastlines.
Another way is to make a height map from the sea levels of your map and erode those. Then add your land height map on top of that eroded ocean height map.