Don't forget that there should be a road going into the castle somewhere and that would also mean a gatehouse with strong towers holding the portcullis and other defenses.

Its unlikely that the road would go into the keep side because of the ditch there, otherwise the keep would have more likely have been on that right side. The keep is always the last stand - the most protected part of the castle and therefore behind all the other defenses like the ditch and draw bridge.

The road does fix down some of the castle architecture so its a good idea to get it in on the earlier side. You tend to find that the poorer parts of a town are at the entrance to the town along the road and it gets more upmarket as you get closer to the keep. Usually, houses inside the walls were much better than outside.