Recent diary entries
It's been possible to run Linux software on Windows for some time using Microsoft's "Windows Subsystem for Linux". Here's what needs to be done to run a simple tile server there, based on the existing instructions here .
Firstly, install "Bash on Ubuntu on Windows". There's are some intructions here (but don't worry where that says "This won’t work with server software"). Continue down to where it says "You’ll be asked to create a user account and password for use in the Bash environment". It's easiest if that user account is the one that you use in the rest of the switch2osm instructions.
Once installed, at a shell prompt "lsb_release -a" should say:
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
What yo get is surprisingly similar to a standard Ubuntu server machine. Some things don't work (e.g. "screen"), and you'll have to manually start some of the services that will need, but otherwise things are very familiar.
Run "sudo apt update" and "sudo apt upgrade" to upgrade to the latest version of everything.
Next, go over to the switch2osm instructions and follow those as normal, except for the caveats pointed out below:
If you want to ssh in (you probably will) you'll need to "sudo service ssh start".
You'll likely want to allow access to ports 22 (for ssh) and 80 (for http) and possibly others at the Windows Firewall (inbound rules / new rule / port / tcp port 80 / allow / all / "name").
After installing postgresql but before doing e.g. "createuser renderaccount" you'll need to start postgresql manually. "sudo service postgresql status" will say "down", but after "sudo service postgresql start" it will say "Starting PostgreSQL 9.5 database server" and "service postgresql status" will then say "9.5/main (port 5432): online".
Where the switch2osm instructions say "service apache2 reload" you'll need to do two things. One is to work around a "Protocol not available: AH00076: Failed to enable APR_TCP_DEFER_ACCEPT" error that is described here, and the solution is to add "AcceptFilter http none" to the end of the /etc/apache2/apache2.conf file, and the other is to start Apache2 manually via "sudo service apache2 start".
Pointing a web browser at http://yourserveraddress/ should display an Ubuntu Apache "It works" page. If it doesn't, check firewall access and any error messages displayed when trying to start apache.
Before you run "renderd -f -c /usr/local/etc/renderd.conf" you'll need to ensure that postgresql and apache2 are both started (they'd start automatically on Ubuntu but not here). Requests are passed from Apache and mod_tile to renderd, and here's an example of what you might get (this is with Hereford, an English county, loaded):
A previous diary entry explains how I created a basic map legend for a map that I maintain that highlights various things common near me that tend not to get shown very well elsewhere - public footpaths, hiking trails, offices, that sort of thing. It struck me that a lot more things had names than almost any other map (other than OSM Bright) actually shows.
I therefore went through a process of looking to see what tags people used names with (it's almost everything - here's a named cattle grid), and what they used as the "naming tag" for various things (mostly "name", obviously). The result is here:
although to get a better idea of things click here and zoom around.
Why might you want to do this?
You might want to set up a rendering server for a short period of time, or you might want a server that has lots of memory while you're importing data, but much less afterwards. An "always on" Azure server will probably cost more that a comparable VPS somewhere else, but if you only want it for an hour or two at a time, it may cost considerably less.
This page describes using Microsoft Azure to do this, but of course other cloud providers are available, and depending on what you want to do may be better suited to your needs.
I have no commercial relationship with Microsoft (beyond occasionally buying hardware with Windows preinstalled and then later removing it) and this isn't a "recommendation", beyond providing another option to setting up a server yourself.
Sign up to Azure
If you haven't already got one that you want to use, create a Microsoft account. Jump through whatever validation hoops you have to (for me that was an email address or a phone number, but this may vary depending on where you are).
Next, go to portal.azure.com and sign in. There will probably be a few "free trial" buttons around; click one of them to get a 30-day trial of up to £150 / $200. You'll need to provide the Microsoft account details (from above), a phone number (again) and a credit card. If you're in Germany and want to keep your data there see here.
Create a virtual machine
This page is a good starting point. You'll want to install the Azure CLI because it's a lot less fiddly than than the alternatives, though there is a "Cloud Shell" option in the portal (actually that's an account on a remote Ubuntu 16.04.2 LTS VM!) and a "clicky pointy" interface in there too.
First, create a "resource group" to tie everything associated with this machine together, and make it easier to tidy up later. I created this in "uk south", you can pick somewhere closed to you:
az group create -n MyResourceGroup -l uksouth
If you already have a public key associated with this account (typically $HOME/.ssh/id_rsa.pub) then:
az vm create -n MyVM -g MyResourceGroup --image UbuntuLTS --data-disk-sizes-gb 20
az vm create -n MyVM -g MyResourceGroup --image UbuntuLTS --data-disk-sizes-gb 20 --generate-ssh-keys
to create them and create a remote machine. If you've generated ssh keys in Cloud Shell you'll want to store them locally and use them later for access, otherwise you won't be able to log in again.
That'll get a a "default sized machine" - you can easily change it later if you want to. Allow access in on ports 80,443 and 22:
az vm open-port --port 80 --resource-group MyResourceGroup --name MyVM az vm open-port --port 443 --resource-group MyResourceGroup --name MyVM --priority 901 az vm open-port --port 22 --resource-group MyResourceGroup --name MyVM --priority 902
Click "Virtual Machines" in the portal - you can see the IP address to connect to there.
At this point you can head over to the switch2osm instructions and follow those.
Once complete, come back to the portal and click "Virtual Machines" at the left. Click the name of the machine you just created, and then "public IP address" at the right. That'll allow you to type in a personal DNS name (perhaps "mynameswitch2osm"), and the FQDN will consist of that plus ".azureregion.cloudapp.azure.com", for example "mynameswitch2osm.uksouth.cloudapp.azure.com".
If you have control of a DNS you can set up a CNAME record so that the name that you want to refer to the server as (e.g. "myname.mydomain.com" points at "mynameswitch2osm.uksouth.cloudapp.azure.com"), and you can create a certificate for "myname.mydomain.com" using "letsencrypt" if you want to.
When you've finished using the server, stop it from the portal (click the three dots to the right of the server name). In the portal the server will change from "running" to "deallocating" and then "Stopped (deallocated)".
To start it again click the three dots and then "Start" - it'll take a couple of minutes to start up (you may need to wait a minute or so after it has started before yo can ssh in).
When you want to completely delete the VM and everything associated with it, click "Resource Groups" in the portal, click the resource group you created at the top of this process and then "delete" at the top right. You'll need to type in the resource group name as confirmation to delete it.
There's recently been a thread on the talk-gb mailing list where someone has decided that, despite previous custom and practice there, the "name" field in both English- and Welsh-speaking areas of Wales should be a compound of both the English and Welsh names. No-one says "I'm climbing up Snowdon / Yr Wyddfa today", they'll use one name or the other, not both together.
In the Welsh-speaking areas the Welsh names are more likely to be used; in the English-speaking areas the English names. It's not a hard-and-fast rule; this peak in the Black Mountains is referred to about equally by both the Welsh and English names, despite it being in a predominantly English-speaking area.
Wikipedia gives an idea of Welsh-language take-up here. That's a bit broad-brush; for example I don't think there's an isogloss between Carmerthenshire and Swansea where people gain/lose the ability to speak Welsh.
So how is it possible to extract data from OSM with the Welsh name in the Welsh-speaking areas and the English name in English-speaking ones, both when creating e.g. a rendering database for the first time and when updating it as people update OSM? Firstly we'll just consider the "loading the database" part.
Very roughly, the Welsh-speaking area of Wales corresponds to this area. That's not perfect, but it's not a bad approximation for a rectangle. I downloaded the latest Welsh data from Geofabrik and cut that area out of it:
osmosis --read-pbf wales-latest.osm.pbf --bounding-box left=-4.82 bottom=52.02 right=-3.34 top=53.69 --write-pbf wales_cy_before.pbf
Convert the "Welsh-speaking" part to names based on "name:cy":
osmosis --read-pbf wales_cy_before.pbf --tag-transform transform_cy.xml --write-pbf wales_cy_after.pbf
Create a copy of the larger file with names based on "name:en":
osmosis --read-pbf wales-latest.osm.pbf --tag-transform transform_en.xml --write-pbf wales_en_latest.pbf
Merge the two together (do it this way around and the "Welsh" file seems to take precedence):
osmosis --read-pbf wales_cy_after.pbf --read-pbf wales_en_latest.pbf --merge --write-pbf wales_merged.pbf
osm2pgsql --create --slim -d gis -C 2500 --number-processes 2 -S openstreetmap-carto.style --multi-geometry --tag-transform-script ~/src/SomeoneElse-style/style.lua wales_merged.pbf
The "osm2pgsql" command to use obviously varies depending on the data and the style used; I'm using this lua tag transform (that's unrelated to the "osmosis" tag transforms described above) and this map style.
Edit: There's an automatic script to do this (for the style I use) here. I've updated that to use a .poly file (thanks to SK53 for that). Interestingly that includes St Davids in the "Welsh" part - so Glasfryn Road is now "Ffordd Glasfryn"! I've also used Scots Gaelic names for the far northwest of Scotland, and the results can be seen here
This means that it should be possible to follow these instructions to set up a tileserver from scratch that creates tiles in the same style as OpenStreetMap's "standard" style in just a few hours (for small areas the longest period of the setup is waiting for the shapefiles used by the style to download).
What would be really useful would be if people could test that these notes do actually work for them - is there anything that should be made clearer, especially to someone who might not be familiar with Linux at all?
I've created a map legend for the map style that I use. It's a set of map data that can be appended to a rendering database to create a map legend in a blank bit of map (actually in the middle of Australia).
There are currently two parts - linear features and POIs. As far as the map is concerned these are "real POIs" (though they're never added to OSM itself of course - just added to the rendering database) so you need to zoom in to see some of the POIs such as the one for vending machines, which only appears in this map style from zoom level 19. You could use it with other map styles, but some of the features are dependent on this lua script.
Here's a preview of what the "roads" part of the legend looks like:
but to use it properly follow the link and move around.
However, sometimes zoom level 20 isn't enough. Here:
is part of Nottingham at zoom level 20. At least one of the office names doesn't appear (it corresponds to here in OSM). The problem is that the way that the "standard" renderd stores metatiles means that only a certain number of tiles can be stored for each zoom level (see this list post for the details). In order to store more I changed renderd slightly so that more zoom levels can be stored - see here and here for the details.
Rendering works fine at higher zoom levels (up to 28 in my example) so that all of those office names now appear. Here's the same area at zoom level 21:
The principle could be extended to an eventual goal of 1:1 to keep Melchett and Darling happy (roughly zoom level 32 at this latitude) but that seems unnecessary even to me currently.
Because of the way that relations are handled within osm2pgsql, converting route relations into "just another highway type" is pretty straightforward, and once created that new "highway=ldpnwn" can be added to the stylesheet.
In this case the two things to think about are the fill and the text. The "fill" is a short wide purple dot separated by a long gap from the next one. In the code (see the links above) at "line-dasharray" the "1" says "short and the "60" says "separated by a long gap" and further down the "line-width: 4" says "wide".
The text for most names in the stylesheet is handled in one place. Long distance paths have to be different to distinguish them from road names, and so the text fill is purple, the text-dy larger to place it further from the road and text-spacing is further apart at higher zoom levels.
The big advantage of handling relations "just like any other road" is that all the clever Mapnik stuff "just works" - like text going around corners:
On many occasions in the past I've said "it's really easy to create your own Garmin maps" but I don't think I've ever documented the procedure. This is an attempt to do that.
You've got some sort of Garmin device with an SD card in it with some maps that you've downloaded from somewhere on it.
You want to replace those with a map you've created yourself.
You've got a PC with around enough memory in it. The one I used for the examples below had 4Gb; it's possible that less may be needed for a small file such as the one used in the examples below.
All the path examples below for for Windows (actually tested on Windows 7), but it should be relatively straightforward to adapt them to work on other operating systems.
What you need to do
The easiest way to create Garmin maps from OSM data is to use "mkgmap". That's available from:
The download page for the latest mkgmap is http://www.mkgmap.org.uk/download/mkgmap.html and the current version is "3676".
Unzip the zip file and move the folders intact to a known location. In my case I moved it so that "mkgmap.jar" file was in: C:\Utils\mkgmap-r3676
I didn't use a path with a space in it or a path that Windows recognises as "special" to avoid any possible confusion later.
"Splitter" is a separate program used to split a large OSM data file into portions small enough for mkgmap to work with (the resultant Garmin map can still route between portions).
The latest Splitter is here: http://www.mkgmap.org.uk/download/splitter.html and the current version is "437".
Unzip the zip file and move the folders intact to a known location. In my case I moved it so that splitter.jar was in: C:\Utils\splitter-r437
Next, we need to download an OSM extract to use to create maps from. As an example I'll use Romania, and extract for which can be downloaded from here:
That page currently says:
romania-latest.osm.pbf, suitable for Osmium, Osmosis, imposm, osm2pgsql, mkgmap, and others. This file was last modified 7 hours ago and contains all OSM data up to 2016-05-13T19:45:03Z. File size: 146 MB; MD5 sum: b0554ae9632cc6c2be1c1e3525b5dcb5.
After downloading, check that the file is intact:
md5sum romania-latest.osm.pbf b0554ae9632cc6c2be1c1e3525b5dcb5 *romania-latest.osm.pbf
(see links from https://en.wikipedia.org/wiki/Md5sum for where to get an "md5sum" program from if you don't already have one)
That matches so we can continue.
Create a directory to work in - in my case I created "D:\doc\gps\romania" and copied romania-latest.osm.pbf to it.
From a command prompt (on Windows), change the current directory to be where the downloaded .pbf file is.
d: cd \doc\gps\romania
See what version of "java" you have installed. Type "java -version" at the command prompt to find out. In my case that says:
java version "1.8.0_91" Java(TM) SE Runtime Environment (build 1.8.0_91-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
so Java is already installed. If you don't already have Java installed download and install from somewhere like: http://java.com/en/download/windows_xpi.jsp
Next, run splitter. The "-Xmx1200m" tells it how much memory to allocate; the "--max-nodes" how big to make each split piece. The examples for the parameters given here should "just work":
java -Xmx1200m -jar c:\utils\splitter-r437\splitter.jar romania-latest.osm.pbf --max-nodes=800000 --output=xml
That should create lots of ".osm.gz" files. Then run mkgmap. Here "--style" determines the style of map to create - "default" is actually a folder containing several files that make up that style. Unsurprisingly the style called "default" is the default mkgmap style - not particularly optimised for foot, bicycle or car use.
java -Xmx1200M -jar C:\Utils\mkgmap-r3676\mkgmap.jar --style-file=C:\Utils\mkgmap-r3676\examples\styles\default --route --gmapsupp *.osm.gz
When that finishes it should say something like "Total time taken: 621133ms" (i.e. about 10 minutes on a fairly slow PC for a country that is relatively small in OSM terms).
d:\doc\gps\romania: -rw-rw-rw- 1 A.Townsend None 123359232 05-14 13:31 gmapsupp.img -rw-rw-rw- 1 A.Townsend None 4270 05-14 13:31 osmmap.tdb -rw-rw-rw- 1 A.Townsend None 73728 05-14 13:31 osmmap.img
Create a subdirectory and move those files into there for safekeeping:
d:\doc\gps\romania\default: -rw-rw-rw- 1 A.Townsend None 123359232 05-14 13:31 gmapsupp.img -rw-rw-rw- 1 A.Townsend None 73728 05-14 13:31 osmmap.img -rw-rw-rw- 1 A.Townsend None 4270 05-14 13:31 osmmap.tdb
Copy those files to the GPS. The method to do this varies by device. I'd expect that people reading this will already do this with files they've downloaded from elsewhere; if not there are lots of examples on places such as http://help.openstreetmap.org.
When it's copied across, turn on the GPS and check that the new map is visible.
Next, how to change the style slightly?
In C:\Utils\mkgmap-r3676\examples\styles, take a copy of the "default" folder and call it something like "mystyle". We'll leave "default" as it was and make a simple change to "mystyle". As a test we're going to make tracks appear like unclassified roads.
Edit c:\Utils\mkgmap-r3676\examples\styles\mystyle\lines in a text editor. The "lines" file is the one that deals with linear items - things such as roads.
Here's the line in that file that deals with unclassified roads:
highway=unclassified [0x06 road_class=0 road_speed=3 resolution 21]
It's pretty straightforward what is happening - the OSM feature at the left ("highway=unclassified") is to be transformed into the Garmin feature at the right ("0x06").
Here's the line that deals with tracks:
highway=track [0x0a road_class=0 road_speed=1 resolution 22]
change the latter line to:
highway=track [0x06 road_class=0 road_speed=3 resolution 21]
(i.e. so that "highway=track" is treated as if it was "highway=unclassified")
Rerun mkgmap with the new style:
java -Xmx1200M -jar C:\Utils\mkgmap-r3676\mkgmap.jar --style-file=C:\Utils\mkgmap-r3676\examples\styles\mystyle --route --gmapsupp *.osm.gz
Create somewhere safe to store the new files, and put them there:
d:\doc\gps\romania\mystyle: -rw-rw-rw- 1 A.Townsend None 123338752 05-14 14:23 gmapsupp.img -rw-rw-rw- 1 A.Townsend None 73728 05-14 14:23 osmmap.img -rw-rw-rw- 1 A.Townsend None 4270 05-14 14:23 osmmap.tdb
Copy to the GPS again, and check that tracks do indeed appear as roads.
... and that's it.
Obviously there are many more changes that can be made. There's quite a lot of information on the OSM wiki at places such as http://wiki.openstreetmap.org/wiki/Mkgmap/help .
The resulting example "default" files can be found at this link:
The resulting example "mystyle" files can be found at this link:
A while back I described how I was showing tree types in woodland. The "unfinished business" there was "what about forest areas where the trees have been cleared?". Mapping of that is a bit hit and miss. "Forestry" has been suggested, but doesn't have many takers, and "forest" is actually often used for "the entire forestry area" (at least where I'm interested in rendering tiles for - I suspect it varies considerably worldwide). The wiki page and the standard style rendering discussion don't distinguish, but I thought it was worth trying to separate out "natural=wood" and "landuse=forest" where the latter is used for "the entire forestry area, including where there are currently no trees".
Here's the result:
That corresponds to here in OSM's standard style. The dark green bit corresponds to "trees" (natural=wood; if there's a surveyed leaftype then obviously that is shown too). The lighter green bit means "forest, but no trees" (landuse=forest - the lighter green is only visible if there's no natural=wood also there). The forest and wood colours are defined here; here is the leaf_type handling in the stylesheet and here is where the natural and landuse tags are checked to see whether the current object should be treated as "trees with a known leaf type", "trees without a known leaf type" or "forest, but not necessarily trees".
(or if you're English, "Pavements!")
I finally got this working for tertiary and secondary roads:
That location corresponds to here in OSM.
There were three bits to it:
1) Determining which roads should have a sidewalk rendered. This is done in lua, and creates e.g. a road type "tertiary_sidewalk" to go with "tertiary" and "tertiary_link":
2) Handle "tertiary_link" the same as "tertiary" in the style's MML, e.g.:
3) Handle "tertiary_link" the same as "tertiary" in the style's roads.mss, e.g.:
... except where we want to do something different:
"sidewalk_width-z13" etc. are declared at the top:
There's scope for further tinkering, and I've yet to see how usable it is on a small mobile phone screen in poor light, but it looks OK so far...
As you may be aware, mapping of areas of trees in OSM is complicated. It's not possible to tell just by looking at the data which of the four(!) approaches described on that page someone is using "natural=wood" and/or "landuse=forest" to mean. It therefore didn't make a lot of sense to me to display them differently on a map created from OSM data.
Last year there was a proposal to record "leaf_type" and "leaf_cycle" separately, which makes sense (though the wider range of non-European tree types doesn't seem to be catered for as well as previously. Unfortunately a previous version of that page suggested that "wood=deciduous" should be replaced by "leaf_type=broadleaved", and a no doubt well-meaning non-local mapper decided to change some areas of mainly deciduous woodland to "leaf_type". Whilst some of these were correct, clearly there are some issues with this, but as I was changing some that I did have local knowledge of, the thing that mainly struck me was that the situation on the ground was far more complicated than previously mapped, or rendered, on OpenStreetMap. I therefore decided to start trying to record "leaf_type" and (when there was enough data, render it. Initial results can be seen here: That location corresponds to here in OSM. There's a lot more to do there, but at least there's a bit more detail than "a large area of trees".
That rendering is created by a combination of this lua script at osm2pgsql data import time and this stylesheet. It's primarily designed for showing England-and-Wales-specific rights of way, but trees seemed like a natural extension.
A while back I wrote https://github.com/SomeoneElseOSM/Notes01 to enable OSM notes to be converted to a useful Garmin format so that you can actually check them when out and about.
A week ago I added support for extracting "fixme" tags from Overpass. There are about 20 notes within 9km of my house, so I was interested to see how many fixmes there were. It turned out that there were 430! As someone said on IRC, "that'll keep you busy over Christmas".