Maya is a vector mesh app and you would need a LOT of mesh triangles to do that job. You could use a mesh of 1x1 and then use a displacement map. Now sometimes the displacement map affects just the vertices of the mesh and other times on higher end stuff it does a dynamic mesh subdivision. Given that it dont come much more high end than Maya then I would guess that it would be able to do it. Check if it has the subdivide option. If it cant then you are always going get stuffed by res with a 3D app.

Next up. Tiff is a very standard file format for GIS data but no so much in CGI. I'm not sure what a normal format would be but I have to use IFF which is pretty uncommon. Personally I think a 16bit PNG should be ok for terrain height.

For getting resolution in X-Y you can chop down the terrain height field and use that with Maya so that it diverts more of the image to vertices or use a DEM processing tool. I suggest latter. Now which one ? There are many but if you have never used a paint package would you buy PS ? I think you should try and get into Wilbur for a bit at a min cos its free and very capable. Once you know its limitations then you can decide what might be needed to cover for that. Thats like any bit of software. I think Wilbur is very capable of doing what your asking for.

Wilbur can add noise and resample height fields. Now whether by adding noise and blowing up a section would produce a realistic DEM of that area is another thing. That's where all the programming effort has been going for some time. Su-Liam has some experience of adding noise to DEMs and knows about different noise functions and whatever to get what he needs. Personally I only deal with Perlin noise and whittle DEMs away with fluid flows. I don't think you can get that all that much mileage out of *just* using noise but I repeat that's a very humble personal opinion. If you have started with a real DEM and want a little extra from it then you will be able to go there ok I think.