[WIP] Theia Project: The Duchy of Vistenia (using Wilbur and Grand Designer)

[nwisth]

Hi again, map affectionados!

I know I ought to start writing the next part of my novel, but I'm in the middle of a bit of a map-making spree at the moment. While refining Theia's climates to fit the Köppen model, and later reworking my inhabited continent map to match, I had tweaked my height map to give me a much better-looking system of rivers and peaks.

The area most impacted by this was my Eastern Highland region - a temperate eucalypt rainforest dominated by the mighty Vaerenbergh river and its many tributaries. The Vaerenbergh was still there - but this time, it meandered a lot more satisfyingly across the terrain, instead of simply being a great cluster of V-shapes caused by a too featureless heightmap. This forced me to re-draw the political borders of the region, however - and made it neccessary to re-create the local map of the upcoming setting of my next novel part: The Duchy of Vistenia.

Previously, Vistenia had been but a Countship, with but a single large town as its capital, but with adjusted borders, climate, and population numbers, the new version had grown to support a larger urban population, making it qualify as a Duchy.

My first map of the region was this one:

Vistenia-old-version-compressed.jpg

In this series of country maps, I simply zoomed in on my continent map in Photoshop, traced the water contours by hand with my Wacom, then used StarRaven's Sketchy Cartography Brushes to fill out the details. It gave me a nice hand-drawn look, which I imagined to fit the setting's technology level. There were some challenges, however. As you can see, the rivers are way too wide for scale. With the whole map showing an area of 450 x 510 km, the main tributaries are like great fjords, ranging from 2 to a whopping 9 kilometers from bank to bank, while the "tiny streams" are five hundred meters wide. Even the Amazon is rarely wider than 5 kilometers near its delta, and it has a far larger watershed than this upstream portion of the Vaerenbergh. Some of this could easily be attributed to cartographic exaggeration, but the fjord-rivers are just too immense to make any sense. I should have made them into fertile valleys instead, and re-drawn the rivers on a lot smaller scale.

I didn't feel like starting over and stamping everything back, one brush-click at a time. I had just gotten better at refining the height maps and using the various functions of Wilbur, so I decided to go that route, and see if I could make a more realistic, atlas-style map of the region.

Using BenVista's PhotoZoom Pro 6, I enlarged my continent height map to 2000%. Now, the resulting 60800 x 108480 pixel png file was a bit too large to start running through Wilbur - my old HP Z820 workstation still has a few muscles, but its starting to show its age - so I cut out a 10103 x 7213 pixel chunk of it and got to work.

Let me tell you - refining that height map took a while. I drew some extra height layers using Mbartelsm's wonderful mountain brush, to give it some added texture in the right places. I added the fault line going through the area with an extra Screen layer, then kept merging and refining, tossing it back and forth between Wilbur and Photoshop for countless iterations of Precipitation-based Erosion and various blur widths of Incise Flow.

As a side note: I kept an eye on Wilbur's resource use with Task Manager. Interestingly enough, when calculating the river systems, the program managed to utilize 96-98% of my CPU power, which is impressive, considering how it spread the load across all of my 16 Xeon cores. When doing precipitation erosion passes, however, it used the GPU, and here it trudged along, never using more than 14% of my single GTX 1080 Ti and about 4% CPU.

In the end, I had a height map I was quite satisfied with. Nice, wide valleys, with rivers meandering about along their bottoms. The fault line looked good. I had a cool, deep canyon in the northeastern quadrant, and mountains awash with ridges and valleys. For a region drenched in the evaporated waters of the Shimmersea both summer and winter, it looked good to me. :)

I was curious how Vistenia would look in three dimensions. Unfortunately, Wilbur crashed as soon as I tried to turn on the 3D Preview Window, but after scaling the map down a few notches, I managed to get a quick peek:

Vistenia-3D-exaggerater(800).JPG

The height level is obviously exaggerated to give me a more dramatic view. :)

I put the 16-bit height map into Photoshop and created a Gradient Map adjustment layer, trying to simulate the difference between fertile valley bottoms, forested slopes and ridges, and grass-and-moss-covered mountainsides. I exported a light map with an Azimuth of 279 and Elevation of 30 from Wilbur to give the gradient map some texture, as well as a few white-on-black River Flow maps, which I traced at various widths. That gave me this result:

Vistenia-satmap-gradient-and-rivers-compressed.jpg

It was a pretty good start, but still not very satellite-like. How could I refine the map to give me a better texture?

Turned out, 3D was the answer.

I was still curious about how to get my height map visualized in 3D better than what Wilbur could do. While googling, I came across a link which led me back here, to Naima's post from 2016 about a piece of software called Grand Designer. I had to try it out. At first, I had fun importing the Theia world heightmap, taking great pleasure in finally seing the ring shadow projected properly on the world. Unfortunately, I found no way of uploading my own texture; I had to try and recreate things using Grand Designers tools for temperature, moisture, sediment and vegetation - which didn't work on the same level of realism as Azélor and Charerg's climate models.

I just had to try one more thing before putting it away. I imported my Vistenia height map, and unfolded it back from a globe to a flat surface. Here, the sediment tool worked wonders. I could adjust the amount of sediment in cavities, sediment on slopes, and limit it by height, giving me a wonderful way of figuring out just what areas would be fertile ground for Vistenia's thick forest cover. With the Vegetation tool, I could differentiate between the lowland eucalypt forest and the high, cool rainforest dominated by moss-crusted pines, simply by adjusting the Growth by slopes and Growth by height sliders.

Here's a snapshot from Grand Designer, from when I was still toying with the settings (and exaggerating the Input Scale/Amplitude again for dramatic effect):

Render-Vistenia_003.png

When I had the result I wanted, I exported a high-res BaseColor map and Ambient Occlusion map from Grand Designer and put them into Photoshop. I found out that Grand Designer had kept the width of the files at their original 10103 pixels, but truncated the height to fit a 2:1 scale - to wrap it around a globe. I had to stretch the maps back to size, but at this resolution, I found the loss of quality to be negligible.

So, this is where I am at the moment, after blending all my various layers:

Vistenia-satmap-WIP1-compressed.jpg
(The attachment is exported at 6% quality to fit the forum size limit.)

As it is, Vistenia is divided into nine Baronies, grouped into three competing Countships, nominally united under a single Duke. It's got a population of 201747 adults and 118176 children, ruled (and fought over) by the 7842 adult members of the warrior caste, living in their Baron's castle. The farmer caste lives on collective farms, the laborer caste live in various labor camps near natural resources, the artisan, trader and a few of the priest castes live in the towns, while the rest of the priests, a few farmers and the elderly of all castes raise the next generation in creches, children's homes and youth homes.

I've calculated the number of 200-person collective farms that each barony needs to sustain itself, and placed them on the map. A little research gave me an average medieval farmland area of a square mile per 125 people, so tweaking it a bit to account for the terrain, I selected each farm and expanded it by roughly 25 pixels, using it to mask in a nice color-corrected farmland texture. I've also turned the rivers brown, to account for the sediment, and placed other settlements, such as castles, towns, creches and children's homes. I still have to place the 146 youth homes and draw roads, as well as create a legend before this is done. As it is, the scale of the map is approximately 570 x 412 km.

What more should a map like this include? I've got the barony borders down, but there are also countship borders, that group the baronies together. I also imagine that there would be a few more military installations for the warrior caste to control their land, in addition to their Baron's castles - things like watchtowers and such.

I also wonder about the glaciers. I've noted that Greenland glaciers can be as wide as 8 kilometers as they lie nested between the mountains of the coast. I've tried to fill my high mountain valleys with ice by using Levels and a sharp Gradient Map over my height map layer, moving the border down and filling in step by step, but it's been a time-consuming process, and somehow, I feel that my "rivers of ice" have ended up a lot narrower than I wanted them.

If anyone has any idea or suggestion on how I can improve the glaciers, or have any other input, comment or criticism, please let me know! :)

-Niels

[nwisth]

There hasn't been much time for cartography lately, but I've managed to squeeze in a little work on my Vistenia map. :)

For me, this map is mostly a worldbuilding tool - and it helps a lot to have a properly defined height field. In this version, I've added a series of beacon watchtowers, used by the various baronies to send news of rival troop movements or raids by casteless marauders to the castles, so that the barons and counts can decide whether to deploy their units to meet the threat head on, or pull back and get ready to withstand a siege.

I've tried to make the distance between each beacon tower no more than 200 pixels, which equals roughly 13 kilometers - and I've tried to make sure to maintain proper lines of sight, using the height map as a guide. With all the rain in this region, I imagine the system is far from infallible - and each tower would need a unit of ten warriors garrisoned there, to keep the wood dry and defend the place against casteless bandits and squatters. Of course, having warriors stationed all over the place comes at a cost, so I've tried to limit the towers to the areas with the largest chance of invasion. As it stands, the baronies tie up everything from ten to thirty percent of their warriors to man the towers. I guess the beacons could be manned by far fewer people - two or three - but then they'd be vulnerable to attack. After all, the casteless outnumber the warrior caste almost 2:1.

It would be cool with a semaphore system, in the tradition of Ank Morpork's clacks network, but that will have to wait a few centuries, I think, for when my world matures a bit more.

Here's the map, with watchtower lines-of-sight marked in pinkish lines, and the height indicated by a tiered gradient map. Hopefully, I haven't made too many mistakes. :)

Vistenia-height-and-tower-map-WIP2.jpg

-Niels

[Falconius]

This is very cool, I like the line of sight idea a lot. I did find the gradient confusing to read at first but then looked at the mountains and it snapped into place. Usually the convention is to make the lush green the low ground and the yellowish greens and off greens would be going up. You might want to include the way the gradient works in a legend which would also solve that misreading.

[nwisth]

Thanks, Falconius! :)

On this map I simply used the "Schwartzwald d050" gradient from cpt-city, which can be found here: http://soliton.vm.bytemark.co.uk/pub...ald/index.html - mainly because it had a farily large number of easily discernible steps. For a finished map, I will make sure to add a legend. :)

Maybe this version is a bit easier to make sense of right off the bat?

Vistenia-height-and-tower-map-WIP3.jpg

I've used the "NRWD" gradient, tweaked my levels and added a bit of white to the top - as well as turned on the light map and ambient occlusion layers to give it a bit more texture.

I've also turned on the Countship borders, to show where the internal lines of conflict are drawn through the Duchy. The counts of Silenia, Parvenia and Tusania all owe fealty to the Duke of Vistenia, but not to each other...

-Niels

[waldronate]

As a side note: I kept an eye on Wilbur's resource use with Task Manager. Interestingly enough, when calculating the river systems, the program managed to utilize 96-98% of my CPU power, which is impressive, considering how it spread the load across all of my 16 Xeon cores. When doing precipitation erosion passes, however, it used the GPU, and here it trudged along, never using more than 14% of my single GTX 1080 Ti and about 4% CPU.

This is a largely irrelevant aside, but the usage patterns that you describe are a little weird. Wilbur uses OpenMP for some of its operations and that would certainly explain the 100% CPU usage parts. Wilbur doesn't use the GPU for computations (it just copies the images to the frame buffer when done with the sole exception of the 3D window, which uses OpenGL 1.5), though, and I'm not sure why you'd ever get higher GPU loading than CPU loading.
I wonder if your system is arranged in a NUMA configuration and non-local accesses are killing performance.

[nwisth]

First off, I just want to thank you for creating such an amazing piece of software, Waldronate! Wilbur is letting me do things I could never, ever do by hand, and the more I play around with it, the better it gets.

I wonder if your system is arranged in a NUMA configuration and non-local accesses are killing performance.

I have no idea what you mean, but I will google it and try to find out. It would be awesome if I could manage to tweak my settings and increase performance a bit!

---EDIT---

I'm not at my workstation at the moment - having a nice, lazy saturday morning at home for a change - so I can't give you my exact processor name, but judging by their age and numer of cores I believe I have a pair of Sandy Bridge Xeons, which have a NUMA architecture. Is this something I can configure on or off in BIOS? And would it help Wilbur to use more of my 64 GB RAM to do its precipitation erosion passes?

-Niels

[waldronate]

Wilbur always allocates as much virtual memory as it needs. It will just crash outright if it runs out (yes, not the best memory management scheme).

My thought about the really odd performance was that accesses to memory across the bus from one socket to the other is much slower than through the local socket. Depending on how the scheduler allocates things, the threads can end up spending more time waiting on external bus transactions than they otherwise would. This usage might manifest as low apparent CPU usage, but I don't know for sure because I don't have a multiple-socket system (I have a single i7-3770 with 32GB in my main dev box). I was much more surprised to see the high GPU usage, though. Wilbur shouldn't be doing any GPU operations except during the lighting calculations and when calculating the crawling ants display of the selection, and even there it should be mostly just bus transfers.

Pulling on that thread, it looks like the selection display might be the culprit for high GPU usage when idle. When I wrote that code 20-odd years ago, I was optimizing for memory usage and not caching things (it was GDI and pure software rendering stacks). The behavior is pretty ugly, taking about 40% extra CPU performance on my machine when the selection is active (and showing between 2% and 20% constant GPU usage on my elderly GTX 460 for no apparent reason). Wilbur was retrofit to use OpenGL some years back and I didn't check for pathological interactions between the system and the GPU. I suppose that I should look at optimizing things one of these years.

Having said all of that about selections and performance, I do notice that the CPU usage goes to 100% and GPU usage to 0% on my machine while calculating precipiton erosion. I'm not sure what's causing the odd performance characteristics that you're seeing and I'm not sure how to rectify them, unfortunately.

[nwisth]

Hi again! I've started zooming in on the map yet again, to give me a nice closeup view of the area where the next installment of my novel will take place. This time, I've increased the scale to another 2000%, and cut out a nice 2:1 chunk of it that I hope will work well in Grand Designer.

That means another few days of churning huge png files through Wilbur, and this time I'm keeping track of the performance.

Checking on my CPUs, I found them to be a pair of E5-2670 Xeons, which had NUMA turned on. I went into BIOS and deactivated it, but did not see any change in speed or performance.

I did crash Wilbur a few times, trying to work on a too-big map size, but once I had the right crop, at 24256 x 12128 pixels, I could get to work.

While filling basins, CPU load rose to 4,7%, and GPU went up to 16,9% - until the progress bar was filled out. Then it changed to 98,9% CPU and 0% GPU for a little while, as it implemented its changes.

Using the Erosion Cycle, it kept using all the CPU cores at max for two and a half hours, forcing me to use Battle Encoder Shirasé to limit it by 20% or so, so I could get work done while waiting. Interestingly, Wilbur started out using 18 GB of RAM, but after half an hour memory usage had shrunk down to 1,6 GB. There was no GPU usage during this time.

Now I've started ten passes of Precipitation Erosion, and again, it's using around 4% of the available CPU power, running on 100% of one core until it's time to implement. Then it kicks in, using all the cores for a little while until the next "Eroding Surface" bar begins, whereupon it's back to only using 100% of Core 17. Here it also shows 2-3% GPU load.

I have not had anything selected during these processes, by the way, so there are no marching ants to take up GPU resources.

I hope this little test can be useful to you when you get around to optimizing things in a few years, Waldronate.

-Niels

[waldronate]

Thanks for the update! I see now why the CPU is showing about 5%: I hadn't done the simple arithmetic for number of cores correctly. Wilbur has parts of the precipiton operation running on a single thread, which is what you see with the 100% usage on one core and about 5% (a bit less than 1/16 total usage) overall. The high GPU usage was the concerning part for me, but I expect that it's spillover from a poor choice in UI update constant that is causing Wilbur to spend a lot of useless time checking if the user has pressed a key to cancel, and which gives the OS a chance to do some things that aren't helpful.
When I do a new build, I'll see if I can get that code to be a little nicer on useless work. Total CPU usage is a little harder to optimize for in this case, though, because the system basically walks all over the surface. Statistically, multithreading it wouldn't be likely to affect final results (two threads would have to be processing exactly the same pixel on the surface at the same time), but this particular case might be particularly bad because the flowlines tend to converge along major pathways and the individual threads would be racing each other to excavate the terrain. There's a chance that a simple version of multithreading might go slower than it already is (unlikely, but possible). Maybe I'll stick in a checkbox to let the user make decision about something that they can't predict the outcome of!

[nwisth]

It is now 25,5 hours since I started the ten passes of Precipitation Erosion, and Wilbur is still chugging along on trusty old Core 17. I really hope it's on the last pass now, but there's no way to tell. Maybe a 294 megapixel map was slightly too ambitious? Lol.

In the meantime, I've managed to do some more work on the Vistenia map. I've added mines, quarries and logging camps, and drawn in the major roads that the Trader Caste use to transport goods, food, and equipment upstream (the rivers being a lot more efficient when it comes to bringing raw materials to be processed by the Artisan Caste in the named towns).

I've also added a legend explaining the various map symbols.

This time, I opted to leave out the Barony names, since they blotted out many important details. I'm wondering whether I ought to use the lower left corner of the map to put in other, smaller maps. Then I could show the Barony names, as well as which of the nine Baronies form the four Countships of the Duchy, on some sort of political mini-map - and I could have a smaller topographic map as well. If you guys have any other ideas or suggestions, let me know! I've got a lot of empty map real estate to fill.

Since I have to compress all the goodness out of the map to host it here on the Guild, I've put it on Dropbox instead. It's a 76 MB download this time, giving a clear view of every single-pixel stream.

The Duchy of Vistenia - WIP #4 (76 MB)

-Niels