[Award Winner] Some pointers for using GIS

[Hai-Etlik]

A proper tutorial on the theory behind geography, cartography, and geodesy and how to use GIS for fictional maps has been on my todo list for a while, but I thought I'd make a bit of a stop gap with this thread. It moves kind of fast and leaves out a lot, and simplifies things in places so it's not necessary true, but close enough to get by with.


Geographic Information Systems (GIS) are software for storing, managing, processing, analyzing, and presenting data which incorporates a geographic component. Or put another way, it deals with things and where they are. It contrasts with CAD (Computer Assisted Design) which deals more with the design of individual things or sets of things, but doesn't pay much attention to where they are. In very rough terms, you would use a GIS to deal with a country or a city, but a CAD to deal with a house or a refrigerator. Both share properties with graphics software, particularly vector graphics software, but differ in a number of key ways. GIS is also quite diverse in the forms it takes: there are desktop GIS applications, GIS servers of various forms, and various specialized tools.

The big name in desktop GIS is ESRI ArcGIS (http://www.esri.com/software/arcgis/index.html). This is to GIS what Adobe Photoshop is to raster graphics. It comes in a number of license levels of varying cost and there are a wide assortment of supporting extensions available. The most basic license level without any extensions costs 1,500 USD. I doubt there's anyone here willing to pay for ArcGIS just for fantasy mapping, and anyone who already has access to it, doesn't need this thread, so I'll move on from ArcGIS.

Thankfully, there's a healthy Free/Open Source GIS community which has produced a great selection of GIS software. The Open Source Geospatial Foundation (OSGeo, http://www.osgeo.org/) is a major hub and supports many of these projects. They also provide an easy installer for an assortment of FOSS GIS software under Windows called OSGeo4W (http://trac.osgeo.org/osgeo4w/) this can not only install software, but also manages it and can help keep it up to date (It's based on the APT package manager from Debian GNU/Linux).

Probably the most advanced GIS available as open source is GRASS. It's also a bit old and has an interface that most Windows users will probably find incomprehensible. MacOS users will probably die of shock. If you don't like The GIMP, you really won't like GRASS. This really isn't for newbies, and you probably won't need the extra capabilities it offers.

Next up is QuantumGIS (QGIS, http://www.qgis.org/) This looks a lot more like what MacOS/Windows users are used to and resembles ArcGIS in a number of ways. This is my primary Desktop GIS. it has an extensive library of plugins written in C++ and Python, and is quite capable of doing the majority of things a GIS needs to do.

There are a number of other FOSS GISes though I am not that familiar with them: uDig, gvSIG, and OpenJump. Google Earth also has some very rudimentary GIS capabilites, and there are a number of Web based GISes, often fairly specialized.

Now, to store GIS data, there need to be special GIS file formats. These tend to be related to and in some cases are derived from graphics formats. Like graphics formats, there are both raster and vector GIS formats.

Vector data is usually handled using the model of a relational database. Each class of things to be handled is a "feature set" equivalent to a relational database "table". If you aren't familiar with relational databases, a table/feature set is much like a simple spreadsheet with each row being a feature, a 'thing', with a number of fields (the column), one of the fields can be thought of as holding a shape. Almost all GIS featuresets use a a single unique feature id ('fid') attribute as a key. The geometry of the shapes is far more strict than in vector graphics. Most GIS formats only support straight segments, and restrict self intersection, particularly of polygons. Generally a feature set will only allow one kind of shape, points, line-strings (several segments joined together in a sequence), or polygons. There are also 'multipoints', 'multilines', and 'multipolygons' which are sometimes treated as equivalent to or separate from their singleton counterparts.

The dominant vector format is the somewhat confusingly named ESRI Shapefile, which is actually several files. An individual "shapefile" has at least three files, and possibly more. The first is the shape data file (.shp), the second is an index file (.shx), and the third is a dBase IV data file containing the non-spatial attributes, most will also have at least a projection file (.prj) describing the coordinate system of the shapes. All these files need to be in the same directory and have the same name.

The other big vector format is the XML based open standard GML. This was designed to be stupendously flexible and to integrate into existing XML data. As a result it's almost impossible to implement in a truly general way and so it is usually restricted to capabilities and data model similar to that of a collection of shapefiles (one GML file can hold multiple feature sets). In theory, an XML schema file is needed to explain how a particular GML file works, but in practice software often just guesses by looking at the file itself.

Other formats include GeoRSS which is RSS with geographic data added, GeoJSON which is encoded using the JSON interchange format, KML used by Google Maps/Earth, and plain spreadsheet type files containing coordinates.

Raster data is more like familiar raster graphics formats, in fact the most common format is the GeoTIFF, which can sometimes be opened in normal graphics software as an ordinary TIFF. The difference is that GIS raster data isn't restricted to small integers, the data doesn't necessarily represent colour, there is extra metadata indicating the coordinate system of the file, and they are often designed to handle very large files efficiently. In the particular case that the data stored is elevation, the file is called a Digital Elevation Model (DEM), which is similar to greyscale "heightfields" used in graphics. Another common data set is satellite images, which often use a different set of "colour" bands: Landsat 7 for instance records a "panchromatic" (greyscale) image over the visible bands, plus red, green, and blue, as well as near, mid, and thermal infrared; SPOT5 on the other hand has panchromatic, green, red, and near infrared (No blue, though the panchromatic band includes blue.).

Now, I've mentioned Projections and Coordinate Systems, and I need to explain those briefly. The earth is not flat, we've known this since at least 240 BC thanks to Eratosthanes and his well experiment (Christopher Columbus just thought it was a lot smaller than everyone else did, and he was wrong.) Maps on the other hand, are flat. If you've ever tried to flatten out an orange peel or a burst balloon, you know that you need to do some tearing or stretching to make it work. Projections are a way to do that stretching (and sometimes tearing) to make a flat map. If you look at it in terms of numbers, rather than shapes, then it's often called a coordinate system instead (Sometimes a "projected coordinate system" to distinguish from "geographic coordinate systems" which I'll explain next). The stretching means that no matter what projection you use, there is going to be distortion, but you can control what kind of distortion and where it is by choosing the right projection.

Underneath projections you have to worry about the actual shape of the Earth which is formally called the "geoid". This is best described as a fat, slightly lumpy pear shape. it is close to being a sphere, and very close to being a flattened sphere (an oblate spheroid), but it bulges a bit "north heavy" and has lumps. This lumpy shape is not very easy to work with, so the oblate spheroid is usually used, but our measurements of it vary both with our ability to measure it, and as the Earth itself changes. Also, by shifting and distorting that spheroid, we can match it up to the geoid more closely for certain areas. This leads to many, many different spheroids we can use, and these are called datums (Not "data" even though that's normally the correct plural for "datum"), sometimes they are also called "geographic coordinate systems" as opposed to "projected coordinate systems". In the case of another planet (unless it's exactly the same size as Earth), you'll have to define your own datum which is easier than it sounds.

So, now you should either by completely lost, or you should have just enough understanding to be able to actually do something in a GIS, and have some idea of what that something is. In my next post I'll try to explain how to do some simple things using QuantumGIS including loading a shapefile, creating and editing a shapefile, adjusting projections, and defining new projections for a custom world.

[Hai-Etlik]

So, lets start up QuantumGIS. You should get a screen looking like this

1.png

In the upper right you have a list of the layers in your map, much like the layers in a graphics editor. In the centre is the view of the map, and around the edges are toolbars, which can be dragged to new positions as needed.

Now, we'll need some data to start with. A good source of free vector data is Natural Earth at http://www.naturalearthdata.com/

Click Get the Data. This gives you the option to download various kinds of data suitable for several different scales. For now, lets use the low resolution Small Scale data. Under cultural, download the Countries and Populated Places datasets, and under Physical, grab Land, Oceans, River/Lake Centrelines, and Lakes/Reservoirs.

These will download as ZIP files, open them and you should see 4 files all named the same thing except for the extension.

Attachment 42049

Each of these sets of 4 files is a shapefile and contains one particular feature set. Create a directory for your project and extract all of the zip files into it.

Now, back in QuantumGIS you can add these feature sets as layers. Use the Add Vector Layer button (Green square with squiggles and a plus symbol), click Browse, then in the open file dialogue, go to your project directory, and select all the files. Click open in both dialogues and you should see the layers appear in the layers list, and the features appear in the map. They will probably be strange colours and in the wrong order.

3.png

You can drag the layers up and down the list to get things the way you want them. To zoom in and out, use the mouse wheel or the zoom buttons (+ Magnifying glass, - Magnifying glass). Pan by middle dragging, or by selecting the pan tool (Hand) and left dragging. You can also zoom to the entire document (Magnifying glass with four red arrows), or to the currently selected layer (Magnifying glass and blue square)

You can also select features. The select icon is the yellow blob with the mouse cursor over top. Select it, and the layer you want to select things on, and you can click to pick the features you want. Hold control to add to your selection and click the icon with the yellow blob and the red "not" symbol to clear your selection. You can change layers and start a new selection without affecting the selection on the previous layer. You can also click the dropdown menu beside the selection icon to get freehand, polygon, and radius selection tools. Try playing with them. When you have a selection, you can Zoom to Selection (Magnifying glass and yellow blob)

Next to selection, there's the identify tool, which looks like a mouse cursor with an i on a blue dot. Use this and click on a feature (Again you have to have the layer selected). A box will pop up telling you all the information about that feature. Try selecting the countries layer and then clicking on a country.

There is also a measure tool (ruler) Like select, this has a drop down to select distance, area, or angle measurement. Just click several times to create the shape you want to measure. A dialogue box will appear to show you the measurement as you go. Right click to finish the line/polygon. Check the "Ellipsoidal" box to do the measurement on the Earth's surface, otherwise it's done on the flat map.

Next, you can look at a table of all the information about all the features in a layer. You can call up the table for the current layer with Open Attributes Table (Window with horizontal lines in it). Any selected features will be selected in the table, and you can adjust the selection by clicking on the numbers at the left side. You can sort the table by clicking on the column headings. This is useful for figuring out what data there is to work with, though it can still be a bit obscure at times. Good data should come with some documentation to explain what the different fields mean.

4.png

So, now to get rid of those ugly colours.

Double click on the ocean layer to bring up the layer in the list to bring up the Layer Properties, select the Style tab.

5.png

If you click the Change button on the left with the colour swatch, you can quickly change the colour.

If you want to get fancier, click the Change button with the wrench icon underneath the preview. This will bring up the Symbol Properties.

6.png

Right now, the layer has one layer in its symbol which is a "Simple Fill" just a fill with a solid border. You can adjust the size, colour, and style of both the fill and the border. This simple fill is probably just fine for the ocean so lets leave it and go to the Symbol Properties for the countries layer. The countries are blocking out the land underneath, so set the fill to None. Borders are often dotted, so try changing the border style as well. You can adjust the symbols for the other polygon and line layers the same way to get a map that looks more reasonable.

The point symbols for the cities let you pick shapes to use but are otherwise the same. Here's a city symbol made from two simple symbol layers.

7.png

Besides the simple symbols, there are other types you can use as layers in a symbol. You can make lines out of point symbols, points from a font glyph, or patterns from SVG images. If you are going to be editing the map in a graphics editor afterwards, you will probably want to stick to fairly simple symbols at this stage.

That's enough for now. Hopefully up next, symbols that vary with the attributes of the data, labels, projections, exporting.

[Lukc]

Now this thread is what I've wanted for a while! Great! I'll be following this closely

[Hai-Etlik]

Now this thread is what I've wanted for a while! Great! I'll be following this closely

Glad you're finding it helpful. I was starting to wonder if anyone was. It was actually our exchange in my lampshade thread that prompted this one.

I'll probably do the next section tomorrow.

[Jaxilon]

I am planning to give this a good read when I get a chance as well because, no offense, but half the time I barely understand what you are talking about anyway Hai-Etlik Plus, I would like to learn more about GIS stuff.

[Hai-Etlik]

OK, part three. First something I really should have covered sooner. Saving.

You can save the project using File - Save the way you would in most user software. QuantumGIS has its own file format for projects and they use the qgs file extension. It's important to note that when you create a layer, you are not making a copy of the data from the file, you are merely pointing QuantumGIS at it. If you were to edit the shapefile and load the project up, the changes would appear in it. Also, you can't delete or move the shapefiles without breaking the project. The symbols however are all stored in the project file so we have not so far changed the shapefiles in any way even though the way they look is now quite different. If you were to load one of the shapefiles in another project, or even load it again as a new layer in the same project, it would be given a new style.

Now, as of the end of Part 2, you should have had something looking sort of like this (Depending on how you chose to symbolize)

8.png

Now those layer names are a bit unpleasant. You can change them to something a bit nicer by right clicking and choosing Rename. These names will be used in the legend when we create one.

Now, open up the style properties for the cities layer. You should see a combo box in the top left that currently reads "Single Symbol" change that to "Categorized". It should now look like this

9.png

This will let us set up a style that chooses between different symbols based on the values of one of the columns in the attribute table. You can pick that column with the Column drop down. To start, pick ADM0CAP, which is 1 if the city is a national capital, and 0 otherwise. Under that are two controls to pick a symbol and a colour ramp. These will be used to give you a default set of symbols when we break the data into classes. You can change them if you wish.

Click the Classify button, and you will get a list of three values and symbols.

10.png

These were created by looking for all the unique values in the data. In this case, 0 and 1. An extra item was created for everything else in case a new value is added, or you alter the classes so they don't cover everything.

We don't really need all three so delete the class for 0 (not a capital) we'll let that be covered by the "everything else" class. Now, you can edit the remaining classes. Double click on the symbol to edit it the way you did the single symbol styles. Double click a value to change the value that a class covers, and double click the label to change the label for the class which will be displayed in the legend.

You can do the same thing with other other layers where they have an attribute with a restricted number of values. Rivers have ScaleRank for instance.

11.png

Some attributes have too many numeric values for each one to have its own class, the country layer has a GDP attribute for instance (GDP_MD_EST). Open the style properties for countries and change from Single Symbol to Graduated.

This is similar to Classified but you'll note the new Classes and Mode options. Classes controls how many classes you'll create, and Mode controls how it will try to break things into classes. If you want the details you should probably start by learning some Statistics but roughly speaking, Equal Intervals does what it says and breaks it into equal sized chunks of the maximum, Quantile tries to put equal numbers of features in each class, and Natural Breaks and Pretty Breaks look for natural clusters in the data.

Pick GDP_MD_EST, a suitable colour ramp, class count, and mode. Then click Classify and make any adjustments you want.

13.png14.png

One final little thing for this section, you can group layers together. Right click on the layers list and Add Group. You can drag layers into groups, and drag and rename the groups themselves the same way as layers.

[Hai-Etlik]

OK, next, labeling. Labeling seems to be undergoing some changes in QuantumGIS at the moment so there are two different labeling subsystems.

The first is under the layer properties as a tab. Turn on the labelling with the check box and select an attribute to use. You can also set a "buffer" around the text to make it more visible, adjust the positioning, and so on.

15.png16.png

The newer system uses the labeling toolbar. Pick the label tool ("ABC" on top of a blue line)

17.png18.png

This new tool uses a more advanced layout system that tries to position the label intelligently and with consideration to of the shape of the feature and the presence of other labels, both on the same layer and on other layers. You can pick a specific layout strategy and indicate which layers have priority of placement under the advanced tab. For instance, you might want to give country labels priority over city ones.

Now down at the bottom, you should see a status bar with several items on it. First a set of coordinates for the present location of your mouse, second, is the current scale, third are some controls for re-rendering, which can be useful when large data sets cause the program to lag, and fourth is the current coordinate system, which should be greyed out.

The current coordinate reference system (CRS) should be EPSG:4236, or if you hover over it, it should tell you that's WGS84. EPSG is the European Petroleum Survey Group, an organization which has listed a large set of common coordinate systems and given each a specific number, in this case they have assigned the number 4326 to the WGS84 datum, one of the most widely used datums for worldwide use. If you look at the coordinates box, you'll see that they are in the range of -180 to 180, and -90 to 90. That is, they are the latitude an longitude.

Now lets change to a projected coordinate system. Click the little globe icon beside the CRS indicator. This will bring up the CRS properties.

19.png

Now, so far the data in the shapefiles has been in the same coordinate system as the map. They have both been WGS84. Now we are going to change that, but we need to tell QGIS to adjust everything to match automatically. This can be slow, and sometimes a little buggy so it is turned off by default. Click the check box at the top to enable it and then click apply. In the main window, the CRS indicator should turn from grey to black.

Now, open Projected Coordinate Systems, Equidistant Cylindrical, and select WGS84 Plate Carree. Then click apply again.

20.png

The map is now in the Plate Carree projection. This means that distances along the equator, or straight north-south are correct, and in this case measured in metres. Close the properties and go back to the main widow. You may need to zoom in or out to make things update. Now, where you had degrees, you now have metres, so everything is much, much bigger. In fact you probably only see one big field of colour from a single feature. So Zoom Full to see the whole map.

21.png

That really doesn't look all that different, but it is. Move your mouse and watch the coordinates. They are now up in the millions because it's all in metres now. The first is in distance along the equator, and the second is north or south. Both have had large numbers added to them to eliminate negative numbers. Obviously, the scale doesn't work for other distances, the poles have been stretched out by the projection. Select Canada from the countries layer and zoom to it. The arctic islands are a very good example of the distortion. Lets pick another projection to fix that.

22.png

Open the CRS properties again and change to Lambert Conformal Conic. There are a lot of LCC projections so to find the one we want so lets do a search. Down at the bottom of the list, select Search by Name, enter "Statistics Canada" and click Find. This should bring up NAD83 / Statistics Canada Lambert. If not, click Find again until you get it.

23.png

Apply, close the properties, and Zoom to Selection to re-centre on Canada.

24.png

That looks much better, although the ocean has disappeared. Also, if you zoom out, you'll see that other parts of the world have very serious distortion. Antarctica has even been turned inside out. Try turning the countries and land off, and you'll find the same thing has happened to the ocean. Normally we would be working with shapefiles that only contain regional data when making a regional map, which would solve this problem. I'll cover creating a clipped and reprojected shapefile later.

[Robbie]

This is a fantastic tutorial! Have a 5 Compass Rating!

[Hai-Etlik]

This is a fantastic tutorial! Have a 5 Compass Rating!

Thanks. Sorry for the delay on the next section, I probably won't get a chance to work on it until Monday.

[Hai-Etlik]

Sorry for the delay in getting this next section done.

So, you should currently have something that looks like this:

25.png

Select the Cities layer and either right click on it, or go to the Layers menu. Select "Query".

26.png

If you are familiar with SQL, this is a simplified version of the queries you can do in that. You can only manipulate the WHERE clause, and it only looks at one table. If you are not familiar with SQL, you can still probably understand it, and there are some SQL tutorials out there that go into the details.

http://www.sql-tutorial.com/sql-where-sql-tutorial/

In this case, lets query for cities that have been marked as "World Cities". In the Fields list select WORLDCITY and then under the Values box, click All. This will fill the values box with the values in the WORLDCITY field. Now double click on WORLDCITY and it will appear in the SQL Where Clause box at the bottom. The click the = button, and double click on 1 in the Values box. The SQL Where Clause box should now read

"WORLDCITY" = 1

This says to only show cities that have a WORLDCITY field set to 1. So click OK to apply the change.

27.png

Many of the cities have disappeared. They are still in shapefile, we haven't altered it, it is simply being filtered by the query before display. If you look in the attribute table, it will also be filtered.

Now, you can create a new shapefile based on this filter. Again, right click or go to the Layer menu, and this time pick Save As.

28.png

You can browse for a file location, and browse to pick a CRS to save it as. Save it along with your other shapefiles for this project with a name like world_cities_lambert. Choose the Statistics Canada LCC projection, which should now be in the recently used list at the bottom of the CRS selector when you click browse.

When you've set this up, click OK to save your new shapefile. You can add it to your map as a layer if you want.

Now, you may wonder, why would I want the shapefile in the same CRS as the map if the software can reproject it on the fly. The reason is that on the fly transformations need to be fast, which means they cut corners. They are less precise and can cause odd little Glitches. In my case the oceans turned inside out, and a few graticule lines disappeared. I can't be sure you will get exactly the same glytches but assuming you do, you can try saving the file in the CRS you want for the map and then loading that in place of the original.

29.png

Similar to the "save as" option is the "save selection" option. This works the exact same way, except it only saves the currently selected features.

Now, you've probably been thinking that this is all very nice but you want to create something new rather than just presenting some existing information. Well, you can in fact edit a shapefile.

On the toolbar there should be a blue pen. This toggles editing for the selected layer. if it's greyed out, it means you can't edit that layer. One common cause of this is having a query on it. Try opening the query dialogue and clearing it it if you have this problem.

Lets start by editing the countries layer.

30.png

Notice that the buttons beside the editing toggle have turned on. These are the editing controls. The edges of the country polygons have also gotten little red Xs marking the locations of the points that make them up. You may have to drag the toolbars around if you have limited screen space.

To save your changes, you can either toggle editing back off and then select "Save" when asked, or your can click the Save Edits (floppy disk) button beside the toggle button and continue editing.

The next tool is Capture Polygon (Looks like a pentagon with red dots at the corners), which lets you create new features. Left click to add a point, then click again to add the next point and so on as you work along the perimeter of the area. To place the last point, right click instead. You will then be presented with a dialogue that lets you set the attributes for the new feature. The Natural Earth data set has a lot of attributes but feel free to leave them blank while playing with it. Click OK when you have the attributes the way you want them.

31.png32.png

The next tool is Move Features. (Arrows in four directions) This is fairly simple, select it and then drag a feature. If you have multiple features selected, dragging one will drag all of them.

Next is the Node Tool. This lets you drag individual nodes around. You can select a node by clicking on it to highlight it in blue, and then delete it by pressing delete. You can add new nodes by double clicking on an edge.

33.png

Next are Delete, Copy, Cut, and Paste, which operate on the selected features pretty much the same way as in any other application.

Next up are Undo and Redo (curved arrows to the left and right) which likewise work the same way as in any other application.

Next is Simplify geometry which can be used to reduce the amount of detail in a feature. (Strangely this doesn't seem to be working for me)

Then come a set of tools for manipulating "rings" and "parts". A "ring" is a hole in a feature and a "part" is a portion of the feature disconnected from the rest. You create them the same way as with the capture polygon tool. To delete them, pick the appropriate delete tool and click on the edge of the part or ring you want to delete.

The next tool is Reshape, which will take some getting used to but is very powerful. It works in a similar way to the capture tools in that you left click several times and then right click to finish. The line you create must cross the boundary of the feature twice, between the first and last crossing, the boundary will be changed to follow the line you traced. If this applies to more than one feature, then their shared boundary will be updated. That may sound a bit confusing so I suggest you just play with it until it makes sense.

The split tool is similar but you must start and end outside the target feature, and it gets split into two features.

Finally there are two tools to merge features together. They both require you to select two or more features. The first combines the two features into one feature, and then provides a dialogue that finds all the places where their attributes differ, and lets you choose between them. The second tool just merges the attributes but leaves them as separate features.

The rotate point symbols tool only applies when using a special feature of the point symbolizer I haven't gotten to.

34.png

Besides the geometry, edit mode also lets you edit the attributes in the attribute table. Double click on a cell to edit it. Go back to the map view and refresh (Ctrl-R) and your changes should now be visible.

35.png36.png

Finally, you can add entirely new attributes. At the bottom of the Attribute Table look for a button that looks like a window with a little star. This will bring up a dialogue that will let you pick a name and a type. Types can be text, an integer (whole number), or a decimal number. Width is the number of characters/digits, and for decimal types, precision is the number of digits after the decimal place. Width 5, precision 2 allows 123.45 but not 1234.5 or 12.345. All the existing features will get this attribute filled with NULL, which just means "there's nothing here".

37.png

When it's all done, toggle off the edit mode and click Save when prompted. Remember that you have edited the shapefile itself. Any maps using it will reflect these changes.