i am betting this is what happened
you had a 16 bit ( 0 to 65535 ) dem
and converted it to a 8 bit ( 0 to 255 ) image
there are a few ways to SYNTHETICALLY move the 8 bit data into 16 bit
but NONE are good
wilbur uses a blur
the best option really is using a PDE to inpaint the missing data on a massive enlargement and then reduce the size
there is a tool ( terminal based ) called Gmic it uses cimg.h
example "image.pgm" ( a 1024x1024 8 bit rgb )
use non repeating numbers like 3.1415 & 1.41421
Code:
gmic image.pgm -upscale_smart 314.141%,314.141%,100%,2,0.5,1 -resize 1024,1024 -to_colormode 1 -n 0,65535 -type ushort -o image.16bit.tiff
or something like that
you can also use a vector image tool like inkscape to convert it to a SVG
import it into blender as a plain
and bake a height map
that might give the best results but it is not easy for the new to blender user
there is a LOT to learn for that