I know exactly of the problem you're talking about.
Years ago, I had worked with a LARP to help them develop their maps and used Adobe Illustrator (versions 88 and 3.X). At the time, my fantasy gaming was done using canned milieus (Greyhawk, et. al) and I never needed to worry about mapping my own world.
About 20 years ago, I've been working on developing a notion for my own gaming world. I began with sketches which I traced in MS Paint. To say they were bad would be an understatement. However, since I refuse to throw out most of my gaming stuff, they were shoved into a folder called "Gaming World" buried in the recesses of my hard drive. Fast forward to 6, maybe 7, years ago, I decided it was time to move forward with all of the ideas I had.
The first thing I decided to do was investigate software to do this all electronically. Ha! I was going to get on top of this and do it right so I looked into some of the GIS packages which were available at the time. Didn't take me long to realize that I didn't have the time to learn a new technology so I opted to go with what I knew, which was the vector desktop graphics route as well. The problem was that Adobe Illustrator was still ridiculously expensive for my purposes.
I looked at what my options were and selected Inkscape. It's not the most robust program, but it was the first one I found which met my requirements and my price point (in this case, free). While I wouldn't recommend that you plan to use Inkscape for a major gaming world mapping project, its strength lies in the fact that it allowed me to autotrace the scans of the maps I had made over the years, saving me HOURS of manual tracing coastlines. It also allowed me to edit and save all of my work in SVG files.
After the project became too cumbersome for Inkscape, I found that Autodesk had released its iDraw product into the public domain and was being sold as a new package called Graphic. I picked up Graphic and it's been a much easier project since. I've figured out the beauty of using layers to store content and it's allowed me to make some very rich maps without too much effort.
There are other packages out there, Affinity Designer is one that comes to mind, which probably do as much (maybe more) than Graphic.