Well, I happen to live just outside of that Bohemian "diamond"... perhaps I could have a bigger beef with Ortelius: he left out all of the German low mountain ranges.
You are right with the "realism" here: he got it about right. He described what was there. We know how it is and that it is real... that is why we don't question it. But you also said: "Mountains (almost) never ... describe semi circular shapes."
Well, here they do! And look at the Dinaric Alps and the greek and balkan highlands... mountains that branch off at right angles! And, these are depicted quite unrealistically.
(While you are at it: check out his rivers. Especially what he did to the eastern european rivers. Here on this forum, the River Police would go into riot mode!)
All these are "real" in the sense that Greece is not really flatland, the Alps are not lined up north-south and Bohemia is not situated on top of a mountain chain.
And here I disagree: if we didn't have the direct comparison between Ortelius' map and "the real world"... his depiction would be "not realistic". Only because we can compare these two can we say: ah, this is what he ment... even if it is highly abstracted. (Or, in case of his messed up rivers: ah, he just didn't have the info... he simply guessed and didn't care about the physics.)What you talked about is consistency and not realismus.
A map or a picture can be consistent but utterly irrealistic.
Realistic means "that what represents things as they really are". And as we know only one reality, there is no ambiguity about what that means.
Representing a square moon or an upward flowing river can be done but is definitely not representing the things as they really are.
Mountains can be whatever one wants when one puts irrealistic magical forces in play and despite being consistent with those magical forces, they will still stay irrealistic.
That's why I added in my comment "even in a slightly realistic setting" to make sure that my comment is understood as applying only to realistic settings.
If the setting was not meant to be realistic then the comment despite being correct would be obviously irrelevant.
Now even if we assume that the same geological forces that formed our earth made this "fantasy world"... we can see that these forces can produce a lot of formations that some people might depict in a way that seems to be "unrealistic". But without being able to compare the map and the underlying "reality", you just cannot tell what is "realistic" and what is not. You simply do not have the basis to do so.