I agree with Turambar's recommendations. I will also suggest that the better the quality of the input, the better than quality of the results because most software can elaborate small details better than it can generate big-picture stuff. Also, expect that the results of most sims are unlikely to be "realistic", but are definitely likely to be "plausible" (that is, features will likely be self-consistent, but may bear only a passing resemblance to real-world places because sims tend to use too many of one kind of feature and too few of another).

https://www.cartographersguild.com/s...ad.php?t=30645 is a similar question from ten years ago and I expect that the state of the art has no doubt advanced well beyond what was discussed there.

Some examples of what one particular elderly piece of software (Wilbur) can do with sufficient fiddling around can be found by searching for "johnsondale" here at the guild. https://www.cartographersguild.com/a...chmentid=43440 is the original CSU Johnsondale map result for reference. Another example can be found at https://www.cartographersguild.com/s...ad.php?t=28052 with the high-resolution result at http://fracterra.com/wilburiax.jpg if you want to skip past the thread.

Another common solution is to use collaging (stitching together real-world heightmap pieces). You can do this manually common image editing tools by slicing up existing DEMs and rotating/stitching them together (https://www.cartographersguild.com/s...ad.php?t=51301 has a discussion of this technique), but there are undoubtedly many software packages these days that can do a semi-automated result. About 20 years ago, Howard Zhou showed a result that used flow analysis and automated collaging to match the flow constraints, but I never saw a commercial result from it ( https://www.howardzzh.com/research/terrain/ is his project page ). It's been a while since I've looked too much at these kinds of techniques and it always required way too much talent and technique for my tastes.

As what you want broadly comes under the heading of image synthesis and/or style transfer, I would expect that many of the "AI" image synthesis tools can get you started. This is especially true if the backing model will take an input image and let you say things like "convert this sketch of a map into a heightmap and replace the brown areas with plains, light blue areas with rivers, dark blue areas with ocean, yellow areas with hills, and gray areas with high mountains." Or something like that. I'm not what the kids are calling a "prompt engineer" (similar to a "sanitation engineer", I suppose) these days.