I often have the impression that the "fantasy" side of a castle is found a lot in its towers: higher, with additions on the sides... imagine elements of architecture impossible to realize (at least for the time)... overhangs, footbridges on arches, breaking repetitions with non-symmetrical elements. It's probably silly to say, but the Disney castle can give you some ideas?
But it'won't be easy to give this "fantasy" feel with a map from above, I must say...
edit : I'm not sure I'm helping... but here's a castle I'm currently working on. I try to give it a "fantasy feel" too... I think, it's mainly in the dimensions. Of course, it's a lot more easy, because it's in 3D, but maybe it can help you ?
Donjon01.jpg