I’ve added a new page to the switch2osm guide: “Manually building a tile server (Ubuntu 21.04)”. The work that the Debian developers have done to include the required software inside Debian 11 is now in Ubuntu 21.04, so there’s actually less setup work needed than in previous Ubuntu versions.
Ubuntu 21.04 isn’t an LTS release, so Debian 11 or Ubuntu 20.04 are still the recommended options for servers that are expected to stay around for a while. One of the regular faux pas with the Natural Earth download site has just been and gone, but the guide includes a workaround in case problems reoccur there.
Unrelated to Ubuntu 21.04 directly, there’s currently a bit of an issue with the lack of feedback that part of the OpenStreetMap Carto setup gives to the user, but work is under way to improve that (I hope things will be able to return to where they were last year with this). I’ve tried to explain here and in the other guides that it’s to be expected that “things just go quiet for up to an hour”, and that eventually it will complete.
Previously I wrote about checking National Parks in Britain and Ireland for breakage, by counting the number of valid polygons in a rendering database.
A similar approach can also be used to check line features as well as polygons. It’s possible to count the number of contiguous sections of a given polygon. For example, here’s the Lyke Wake Walk, which is in one piece:
psql -d gis -c "SELECT count(*) FROM planet_osm_line WHERE (osm_id = '-1996318');" count ------- 2 (1 row)
and here is the Sabrina Way, which isn’t:
psql -d gis -c "SELECT count(*) FROM planet_osm_line WHERE (osm_id = '-537430');" count ------- 12 (1 row)
If something is changed in OSM that introduces an extra piece, or fixes a gap, that number will change. This script can be used to count the change in the number, like this:
Change in relation components for Sabrina_Way 537430 3c3 < 12 --- > 13
If run from cron it can check every day and will only send an email alert when the number changes.
If there is a change, and can be investigated with a site such as http://ra.osmsurround.org/analyzeMap?relationId=6361532.
Edit 2: Also useful is the answer to this help question that looks for modified ways that make up a relation.
Edit 1: Where this process falls down is where a very long-distance path is added a superrelation and there’s a gap between the relations, which is what this changeset was fixing. Here two long several relations make up two super-relations that share some of their route. For completeness, the edit that I was fixing that created a gap in the Cleveland Way was actually fixing a previous edit of mine that introduced an unwanted spur to E2 at this location.
Taginfo is a way of finding out what tags and values are used within OpenStreetMap. There’s a worldwide instance here, and other instances exist, including various Geofabrik ones such as Antarctica here.
As well as describing keys and values used within OpenStreetMap, the worldwide instance of Taginfo also holds details of what other projects use particular OSM tags. The Taginfo/Projects page explains the update process.
Each project submits a .json file containing details of key and value usage, such as this one for an OSM Mug.
The one for the map style that I look after is here. Currently I’ve just submitted a list of keys, not values, but I have included a description for each one, as can be seen here for “designation”.
The first challenge was to create the initial list. There are references to tags in three main places in my map style - the lua style file, the carto .mml file and the individual carto .mss files such as this one. A bit of investigation (creating a unique sorted list of strings within style.lua on one hand and .mml and .mss files on the other found that although some keys do pass through style.lua unchanged, almost no keys that are later used in carto files aren’t mentioned in style.lua at all. That means that I only needed to worry about keys in style.lua. I created an initial .json in the correct format and then wrote a shell script to generate the “tags:” section of the .json file. I then filled in the descriptions where the shell script had just created placeholders, and checked it in.
Next, I wanted to be able to easily update the taginfo.json file after changes to the map style. Because I’ve also checked the previous generated file into github I can easily create a new one and compare the two:
sh -x ./taginfo_lua_genkeylist.sh > q.txt diff taginfo_lua_nodesc.txt q.txt
for the most recent change the output of that is:
44a45 > { "key": "colour", "description": "qqq" } 118a120 > { "key": "lifeguard", "description": "qqq" } 119a122 > { "key": "locked", "description": "qqq" } 163a167 > { "key": "place_of_worship", "description": "qqq" }
So next I need to:
In some cases the description can be obtained directly from the commit text, such as for “colour” here:
Show black, white, blue, green, grey and gold phone boxes as the correct colour rather than red.
Once that’s done, taginfo should update within a day or so, and updates will appear on pages such as this one. At the time of writing this diary entry, there are no projects listed for that key.
Then in a month or so’s time, after the next map style update, repeat the process.
The OpenStreetMap Community Index is designed to show people what community groups there are within OpenStreetMap, in any part of the world. If I look near me I see a local pub meeting (sadly on hold during the pandemic), some country-level resources, and then international resources.
On a couple of occasions I’ve seen things that are missing or need updating, and this diary entry is designed to complement the existing documentation to describe what needs to be done to fix that.
In my case I already have a fork of https://github.com/osmlab/osm-community-index at https://github.com/SomeoneElseOSM/osm-community-index. There’s a “fork” button in the top right of Github’s UI that allows you to do this.
Next, I’ll create a local branch to work on. The default branch at https://github.com/SomeoneElseOSM/osm-community-index is “main” and the UI will say “This branch is even with osmlab:main.” because (in your case) you’ve only just created it. In github’s branch selector just type the name of the new branch to create. In this case I created “add_forum_at_world_level”, because I noticed that there wasn’t a link to the OSM Forum at the international level, although there are many links to country-level forums from the community index.
Next, “git clone” your repository wherever it is you want to work. I’m working on a completely new virtual machine to do this, so I have to take a slight detour first - see “setting up ssh access to github” below. Once that’s done:
mkdir ~/src cd ~/src git clone git@github.com:SomeoneElseOSM/osm-community-index.git git checkout add_forum_at_world_level
Note that I’ve used a version of “git clone” that is designed to work with ssh. Github is (rightly) moving away from password-based verification and will send you a warning email if you try and use a password during the transfer period.
To make sure that everything works before making any changes, we’ll build everything as it currently stands. The machine I’m running on is a virtual server running Ubuntu 20.04 LTS. It’s actually set up as per here so it already has Apache2 and some node.js software on it. The versions of node.js and npm are the defaults for this version of Ubuntu, and are:
node -v v10.19.0 npm -v 6.14.4
An older OS with and older version of node.js or npm may struggle, and I’m not going into detail about node.js versions or package managers here, because it’s way above my pay grade.
cd ~/src/osm-community-index npm install npm run test
That checks that everything is OK. To build as is:
npm run build npm run dist
That creates various files in e.g. ~/src/osm-community-index/dist/ including “completeFeatureCollection.min.json” (in previous versions called “combined.min.geojson”), which is all the data needed by the website as it runs. On my server “/var/www/html” is the root of the website. To copy the files into there:
sudo mkdir /var/www/html/community sudo cp ~/src/osm-community-index/docs/index.html /var/www/html/community/ sudo cp ~/src/osm-community-index/dist/completeFeatureCollection.min.json /var/www/html/community/
Next, we’ll need to edit the “index” file on the web site so that it refers to the local “completeFeatureCollection.min.json”, not one from an external webserver.
sudo nano /var/www/html/community/index.html
In my case I’ll edit it so that:
var dataURL = 'http://192.168.1.66/community/completeFeatureCollection.min.json';
and restart apache:
sudo /etc/init.d/apache2 restart
If I browse to http://192.168.1.66/community/ the site looks exactly like the real one.
Next, we’ll make a change. This document describes the format in detail. In my case I want to create a “resource” at world level (“OpenStreetMap Forum”) but don’t need to create a “feature” since my new resource is worldwide.
If you think you might have to add a feature please check https://ideditor.codes first (iD and the community index share this). It is quite possible that your region may be already listed, since quite a few unusual values are already there. There are a couple of other caveats to be aware of as well.
I created my new .json file based on one of the other international examples. For “languageCodes” I chose “en” because most (though not all) of the posts at supranational level are in English. “contacts” is a bit complicated, since there are many top-level OSM forums and they don’t all have the same admins. I chose an address that I knew would at least result in messages being passed on to the right admin(s). For “order” I chose “something in the middle of the list”. The current top-level order settings are:
Facebook 3 Reddit 2 Discord unset Telegram unset Twitter unset OSMF unset Help -2 IRC -4
Given OSMF’s FOSS policy, I’m not convinced that Facebook should be at the top of that list, but that’s another issue for another time. Once all changes have been made, build again:
npm run build
Any errors with field contents should be displayed here. Fix this and run “npm run build” again before continuing. When complete:
npm run dist sudo cp ~/src/osm-community-index/dist/completeFeatureCollection.min.json /var/www/html/community/ sudo /etc/init.d/apache2 restart
Now, when I browse to http://192.168.1.66/community/ the new entry appears. We’re ready to check in the changes. The current status of things can be seen via:
git status
To check in:
git add resources/world/OSM-Forum.json git add . git commit
“git commit” does a local update from “files you have edited locally” to “your local copy of the git repository on this server”. Next:
git push
That updates your github copy of the community index with the copy from your local server.
At this point, the branch in my github account says “This branch is 1 commit ahead of osmlab:main.” when I browse to it.
There’s an option from there to create a pull request, which is saying to the maintainers of https://github.com/osmlab/osm-community-index “hello, I think it’d be a great idea if you made this change”. There’s no guarantee that it’ll be accepted (there may be a good reason that I’m unaware of why that resource isn’t listed at https://openstreetmap.community/). If/when it does get accepted it’ll look something like this.
As an aside, because this is the first time that I’ve needed to use this machine for anything ssh-releated, I’ll create ssh keys and upload to github.
cd ~/ ls .ssh ls: cannot access '.ssh': No such file or directory
To create a key pair:
ssh-keygen -t ed25519 -C "anemailaddress@example.com"
You’ll be asked for a passphrase - enter one and verify it. A file, in my case called “~/.ssh/id_ed25519.pub” will be created that begins “ssh-ed25519”.
In Github:
Settings ssh and gpg keys New SSH Key Add ssh key (confirm password) (receive email about new ssh key)
When you get to the “git commit” part, if this is the first time that you’ve done that on this machine, you may be asked to “Please tell me who you are” and suggested to run:
git config --global user.email "you@example.com" git config --global user.name "Your Name"
As you will no doubt have heard, browser support for flash went away at the end of last year. As you can read here, Potlatch 3 has been released and downloads for Windows and Mac are available. Linux users are asked to “Use Wine to run the Windows installer”. You may be wondering “why?” - there’s a section at the end of this diary entry that covers that. First, the “how”:
There are lots of different Linux distributions out there. Most have some sort of package manager and in most of those “wine” will be available as a package. Here is what I needed to do on a couple of desktop Ubuntu systems (both using KDE) - one running 18.04 LTS, the other 20.04 LTS.
Firstly, install Wine itself:
sudo apt install wine64
Next, you’ll probably need to install some fonts that software that normally runs on Windows needs. You can do this via a helpful script Winetricks. First, install cabextract, which is needed by the script:
sudo apt install cabextract
Next, decide where to install the “winetricks” script. I’ll put it in an existing user “bin” directory, so:
cd ~/bin wget https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks chmod +x winetricks
Next, install the fonts:
winetricks corefonts
Finally download Potlatch 3 (choose the “Windows application” option, which is a zip file) from the download page, unzip it, and run it:
wine64 Potlatch.exe
Follow the instructions that are presented and accept the defaults to “just install” it.
When Potlatch runs, you’ll initially see the whole world zoomed out. Rather than zooming in manually you can then “edit with remote control” from the openstreetmap.org site to go to wherever you are currently looking at there, and you can also search for any sort of OpenStreetMap URL to jump straight to it. “GPS / My tracks / load” loads GPS traces.
A resulting changeset can be seen here. Somewhat confusingly, it says the OS was “Windows 7” because that’s what Potlatch thought that it was running on!
As well as clicking “save” at the top left (or pressing “s”), also don’t forget to close changesets by pressing “c” when needed.
If you try and do too much too quickly it can fall over. To reduce the chance of this happening:
This is available from the download page, but there are some problems. These include:
None of these issues occur with Wine:
I’ve found one machine, an old Chromebook running Ubuntu 18.04 LTS via Crouton that “just sat there” when trying to run the P3 self-extracting .exe for the first time. It may just be that it needs more memory to run the installer (it has 2Gb; the 20.04 machine mentioned above has 4Gb). Using the .zip download fixed that issue (hence that recommendation above). Other relatively low-spec machines are also fine; the Ubuntu 20.04 machine was a fairly average spec when I bought it over 10 years ago.
Occasionally, I (like other people I’m sure) have noticed that things “disappear” from maps they should appear on. In the case of “national parks” and “areas of outstanding natural beauty” in the UK, they’re mapped as multipolygons. This might be simple, like Nidderdale or more complicated, like Ynys Môn. What usually happens is that someone accidentally “breaks” the multipolygon, for example by creating a gap in an outer edge, so that rendering software sees it as invalid and ignores it.
What I’ve typically done in the past is, whenever I’ve noticed a problem tried to find the problem using something like the OSM Relation Analyzer or JOSM’s validator. The “area” view at Geofabrik’s OSM Inspector is also useful.
That works, but “happening to notice a problem” as a first step is not ideal. This map does show national parks / AONBs and not a lot else at zoom 7, but those tiles only typically get updated every 4 days, so it can lag behind problems.
I have a rendering database that covers GB and Ireland, and in there “SELECT distinct name FROM planet_osm_polygon WHERE (boundary = ‘national_park’);” will list all those items that are currently being rendered. As an aside, that query is on “boundary=national_park” because the data is processed by this lua style file as it is loaded, and that consolidates the tagging into something that makes the rendering code less complicated.
Next, the challenge is to get the same list from OSM. Thanks to this answer on the help site, I was able to do that. There’s a bit of manual work comparing the two, because:
Once I’ve done that, I have two lists that I can compare.
Chiltern Hills AONB Dorset AONB High Weald AONB Kent Downs AONB Llŷn AHNE Shropshire Hills AONB South Devon AONB Sperrins AONB Suffolk Coast & Heaths AONB Ynys Mon AHNE
The process is the same manual process as before - download the relation as an object in JOSM, validate, and look for errors in the multipolygon. In some cases (for example “unconnected coastline”) you might need to load a bit more data to avoid seeing unrelated errors. Generally speaking you can ignore the warnings, as most of these are unrelated to multipolygon validity and some are completely invalid.
Usually what’s happened is that someone’s redrawn some feature that formed part of an AONB boundary and neglected to add the new part back into the AONB relation. To take Dorset AONB as an example, it looks like previously one of the railway lines formed the boundary, but that had been edited to leave a gap, so I redrew that as a new way slightly to the south in these changesets.
Shropshire Hills AONB was the most difficult to diagnose and fix. JOSM didn’t actually find an error there so I had to go through the warnings. It may have been this change that finally fixed it (zoom in on the green line east of Craven Arms to see the red line and deleted nodes).
Next - what’s the easiest way to detect breakages as they occur? I don’t want to have to manually compare with Overpass every time, but the good news is that I don’t have to - I can just compare with the list that was in the database yesterday, and look for things disappearing from the list.
# ------------------------------------------------------------------------------ # Count national parks, and see if there are more or fewer that when we last # looked. # ------------------------------------------------------------------------------ mv /home/renderaccount/data/nationalparks.justnow /home/renderaccount/data/nationalparks.previously psql -d gis -c "SELECT distinct name FROM planet_osm_polygon WHERE (boundary = 'national_park');" > /home/renderaccount/data/nationalparks.justnow diff /home/renderaccount/data/nationalparks.justnow /home/renderaccount/data/nationalparks.previously #
That now runs as a cron job on map.atownsend.org.uk every morning, and if there are any changes, I’ll get emailed the results.
I’ve added a new page to the switch2osm guide: “Updating your database as people edit OpenStreetMap”.
It’s a bit overdue (it’s based on something I wrote about 4 years ago, and the software it uses predates that by a long way), but I think it is worth making available on the main site.
I’ve tested it both on Debian 11 and on Ubuntu 20.04 LTS, and have tried to make clear what you need to do differently in each case.
For completeness see also Ircama’s excellent guide which is well worth reading as it actually provides a lot more detail about the process, and also suggests how “osmium” may be used instead of “osmosis” to process OSM files.
I’ve added a new page to the switch2osm guide: “Manually building a tile server (Debian 11)”. The Debian developers have worked to include the required software inside Debian 11, so there’s actually less setup work needed than in previous versions.
Debian 11 has not yet been released; it’s still “testing”, so there may be minor changes along the path to release. Also, there’s currently a temporary problem with the Natural Earth download site that requires a workaround, but that’s documented in the guide. When that is resolved I’ll update the notes.
I’ve not yet linked the new page into the menus because I want to give people a chance to test it and identify any problems first. Once that’s happened, I’ll add it to the “Serving Tiles” menu.
Unfortunately, in England at the moment a number of pubs aren’t able to reopen because (especially locally to me) they’re too small and have lots of small rooms or they’re music venues for which “social distancing” somewhat defeats the object of the event in the first place.
I’ve chosen to show these on this map like this:
This is deliberately different from the rendering of “normal” closed pubs:
the idea being that hopefully the “closed due to covid” ones will be able to find a way of working sooner rather than later.
I’ve done this by looking at the “opening_hours:covid19” tag - that seems to be the most used for this both globally and locally. To be able to handle these as a new object, they’re split out in the lua code used by the style and then added to the mss file that actually does the rendering.
I’ve recently added a new feature to https://map.atownsend.org.uk , prompted by the dreaded lurgi that is affecting everyone at the moment - the ability to show pubs with either outdoor seating or a beer garden.
Outside seating is shown like this:
with a black line at the bottom right of the icon.
Here’s a beer garden:
with a green underline instead of black
For the rest of the map legend see here.
I’ve added a couple of short but interesting scripts to github. They fetch and sort posts from an OSM mailing list from some time in the past. For example, here are the topics on the talk list at the height of the licence change furore:
listtopics.sh talk 2012 May 35 OSM : It's a shame !!! 24 Edit review: ele=0 22 Worst of OSM 21 TomTom is thumping us 17 Import of buildings in Chicago 17 Edit review: intermittent waters 16 OSM cycle map - ?excessive focus on long-distance routes 15 Edit review: "building"="levels=N" 14 Cycleways and Access tags: Left, Right, Forward, Backward? 10 OpenSeaMap 10 Cycle lanes & cycle tracks - my findings and a proposal 7 Changing capitalization (Lima) 6 OWL down 6 OSM data density - top regions 6 Old versions of OSM? 6 Copy-and-paste remapping 5 TTTracklog and TomTom core 9.4 4 OSMCoastline / OpenStreetMapData.com 4 Mapnik rendering issue with *_link roads 4 (dis)Honesty and Copyright
and here’s the tagging list today:
listtopics.sh tagging 2020 May 106 Reviving the path discussion - the increasing importance of trails in OSM 75 Tag:amenity=motorcycle_taxi not approved 58 Remove non-prefixed versions of 'contact:' scheme 49 RFC ele:regional 42 relations & paths 35 Doorzone bicycle lanes 34 Permanent ID/URI --- off topic email 30 Feature Proposal - RFC - Recreational route relation roles 26 Change of wiki page Key:access 25 Adding values healthcare=dispensary and healthcare=community_care? 24 track vs footway, cycleway, bridleway or path 23 Is there any tagging scheme for carillons already? 15 Meaning of "administrative" in boundary=administrative, in your country? 14 With leisure=common deprecated, Senegal & Mali need a replacement 14 [OSM-talk] Should we map things that do not exist? 14 oneway=yes on motorways 11 Section numbers in hiking routes 11 Running but no hiking/walking 9 Quality and the Openstreetmap value chain 8 Fwd: Section numbers in hiking routes
looking at the names, some of these from 2007 are still familiar:
listpeople.sh talk 2007 May 79 Frederik Ramm 59 David Earl 46 Nick Whitelegg 33 Richard Fairhurst 31 Christoph Eckert 27 Jon Burgess 26 Andy Robinson 25 Martijn van Oosterhout 21 Lester Caine 18 SteveC 18 Robert (Jamie) Munro 17 Nick Black 16 Joerg Ostertag (OSM Munich/Germany) 15 Steve Chilton 14 Francisco R. Santos 14 David Groom 14 Barnett, Phillip 13 Hakan Tandogan 12 Robert T Wyatt 12 Matthias Julius
I’ve added a new page to the switch2osm guide: “Manually building a tile server (20.04 LTS)”. At the same time, the old “with packages” instructions has been retired from the list as that depended on Ubuntu 14.04, which has now dropped into Extended Maintenance.
Ubuntu 20.04 (“Focal Fossa”) has now been released. There are actually very few changes from the 18.04 version - mostly just updated versions of software (including postgres 12). Following these instructions shouldn’t take more than a couple of hours for small areas - the longest period of the setup is waiting for the shapefiles used by the style to download.
As before, the page is designed to be “the least you need to do” to get a rendering server working to create tiles like the ones that you see at openstreetmap.org. If it still sounds a bit complicated, don’t forget that you can try things out with Docker very quickly - and that should work anywhere that Docker is supported.
For some time I’ve been experimenting with how other information can be added to established icons. For example, here’s how I’ve been representing various tourist information items:
In those examples there are only two pieces of information - that it’s something to do with tourist information, and whether it’s a board, guidepost, plaque, etc.
With pubs, I already show several sorts of information together. For example this one:
is a pub that’s open (it’d have a cross in the middle of the beer glass if it wasn’t), serves real ale (you can tell from the maroon liquid in the glass), is a micropub (the μ at the left shows that), serves food (the F at the right) and has a noncarpeted floor that means if you’re wearing muddy boots you don’t need to take them off before going to the bar (the brown underline).
A couple of other modifiers are also available - here’s a map showing a pub with a blue “roof” that offers accommodation as well as food:
I thought that it was possible to also show the values from a “wheelchair” tag, so I added that at the left-hand side of the “floor” bar:
Here yellow is used for “wheelchair=limited”, green means “wheelchair=yes” and red for “wheelchair=no”. If there is no wheelchair tag the original icon is used.
Obviously, specialist maps such as wheelmap already exist for this sort of thing, and they can provide much more information (e.g. nearest wheelchair accessible toilets), but there’s no reason why a more general map style can’t show this sort of thing too.
Here’s an example of what the resulting map looks like. The extra colour is deliberately not intrusive but is designed to be visible if you’re looking for it.
As you may be aware, the “switch2osm” site recently moved from Wordpress to Github. This means that it’s easy to request changes to the site over at the github repository. Any changes made there will automatically go live on the main site after a minute or so.
“Requesting changes” might mean commenting on existing issues or creating new ones (which will need someone else to make the code or text changes you’ve requested), or it might mean suggesting the changes yourself via a “pull request” - a list of specific changes you’d like to see made. Here is an example of a pull request - you can see what it’s a change from and to, and on this page you can see any comments and when it was merged in and by whom.
Obviously, if people can contribute via pull requests rather than just creating issues it’s better for everyone - it means that you’re not relying on someone else reinterpreting the changes that you’d like making, and it’s quicker for the maintainers too.
Normally if you wanted to make a major change to something on this site you’d “fork” a copy of the repository into your own github area, create a “branch” from that that relates to the change that you’d like to make, “clone” that locally into an environment on a local machine that allows you to test the site locally, make your changes, test your changes in a local web browser, submit your changes back to your copy of the site, and then create a pull request explaining the change you’d like to make.
What you may not know is that for simple textual changes you can actually do the whole process online. That’s not appropriate for anything that might break links, but for simple text changes (like the example above) it’ll work just fine, and this is what the rest of the diary entry describes.
If you don’t have a github account already, go to https://github.com/switch2osm/switch2osm.github.io. If you don’t have a github account click “sign up” at the top; if you do have a github account already, continue from “Next, at the top right” below.
Select a plan - for the purposes of this exercise “individuals” and “choose free” will work just fine. Then “complete this step”.
That takes you back to https://github.com/switch2osm/switch2osm.github.io.
At the top of the screen there’s a big green button labelled “read the guide”. That’s a useful introduction - you can either read it now or click on the link later.
While that was happening, you’ll have been sent a “Please verify your email address” email from Github. Click through the link from there to prove that you are who you say you are.
Next, at the top right of the https://github.com/switch2osm/switch2osm.github.io page (just below “read the guide”) there’s a button labelled “Fork”. The tooltip says “Fork your own copy of switch2osm/switch2osm .github.io to your own account”. Click that.
Wait a few seconds. When the repository appears you’ll get a prompt to “set up github actions” - you can dismiss that for now.
You’ve now got your own copy of the “switch2osm” github repository. Next, you’ll want to create a branch to hold your changes. In this example I want to make a small textual change to how “renderd” is described on one of the “manually building a tile server” pages, so I’ll call it “renderd_text_change”. From your copy of the repository (something like “https://github.com/yourgithubusername/switch2osm.github.io”), To create a branch, click on where it says “Branch master”, type in the new name and click “create”.
Now browse to the page that you’d like to change. In my example it’s “https://github.com/mygithubusername/switch2osm.github.io/blob/renderd_text_change/serving-tiles/manually-building-a-tile-server-18-04-lts.md”. It’s just a text file in Markdown format. At the top right-hand corner there’s an edit button. Click that, and make the necessary changes.
When you’re happy that your change is complete, at the bottom of the screen type in a short one-line summary of your change followed by a longer description. Ensure that “Commit directly to the branch" is displayed (not "master"). Click "commit changes".
Go back to “https://github.com/yourgithubusername/switch2osm.github.io”. You’ll now see that the screen now shows “Your recently pushed branches:” and gives you the opportunity to “Compare & Pull Request”. Click that. Review your change (old version on the left, new version on the right), and change your summary and description if you want. When you are happy, click “Create pull request”.
Github will then take you out of your copy of the repository into the new pull request in the repository that you forked your copy from. In this example, that’s here. If you scroll down the page a bit you’ll see that initially some background checks are taking place, and then eventually “All checks have passed” and “This branch has no conflicts with the base branch”. You, and other people, can comment on your proposed change. Someone will review it and (probably) merge it without comment. It’ll then be live for all to see at https://switch2osm.org/.
Imagine I’ve just installed OsmAnd on an Android phone and what to change the rendering slightly.
For completeness, the OsmAnd version is 3.5.5, I downloaded it from Google Play, where it’s called “OsmAnd - Offline Travel Maps & Navigation”. The Android phone is a Blackberry DTEK50 (actually a badged Alcatel design) running Android 6.
I ran OsmAnd and skipped the initial download (because I’ve already got some .obf files to install), but you’ll probably want to download some local data.
I want to copy files to and from the device, so I’ll plug it into a PC. How you copy files depends on the OS on your PC. Next, plug the device in via USB and select “copy files via MTP” from the prompt that appears on the device (this actually varies by Android version, and what options are available to copy files to and from a device depend on that and the phone itself). A window showing an MTP connection to the device contents will probably appear - check you see something in there. If you don’t, unplug it and plug it in again.
I’m running Ubuntu Linux on the PC and want to copy files to and fro from the command line, so I’ll install something to allow me to do that, check that I can read something from the device, and unmount it again:
cd mkdir mtp sudo apt install jmtpfs jmtpfs ./mtp ls ./mtp fusermount -u ./mtp
For me the “ls” above shows ‘Internal storage’. Before copying anything used by OsmAnd in the steps below I’ll make sure that it isn’t running on the phone. How to do this varies by phone - on mine it’s “menu” and then “close”.
Next, we’ll look at what maps are currently installed. How to do this varies by Android version, but this is how things appear on this Android 6 device:
ls -alt ./mtp/Internal\ storage/Android/data/net.osmand/files/ total 0 -rw-r--r-- 1 ajtown ajtown 1585986668 Dec 6 19:47 Gb_england_europe.obf drwxr-xr-x 11 ajtown ajtown 0 Dec 3 00:10 . drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 backup -rw-r--r-- 1 ajtown ajtown 359 Dec 3 00:10 favourites.gpx -rw-r--r-- 1 ajtown ajtown 269 Dec 3 00:10 ind.cache -rw-r--r-- 1 ajtown ajtown 4413090 Dec 3 00:10 regions.ocbf drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 roads -rw-r--r-- 1 ajtown ajtown 39985159 Dec 3 00:10 World_basemap_mini.obf drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 fonts drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 sounds drwxr-xr-x 3 ajtown ajtown 0 Dec 3 00:10 tiles drwxr-xr-x 13 ajtown ajtown 0 Dec 3 00:10 voice drwxr-xr-x 4 ajtown ajtown 0 Dec 3 00:10 .. -rw-r--r-- 1 ajtown ajtown 0 Dec 3 00:10 .nomedia drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 rendering drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 routing drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 tracks
Here’s the directory with the .xml file used for deciding how things in a “.obf” file appear on screen:
ls -alt ./mtp/Internal\ storage/Android/data/net.osmand/files/rendering total 0 drwxr-xr-x 11 ajtown ajtown 0 Dec 3 00:10 .. -rw-r--r-- 1 ajtown ajtown 691532 Dec 3 00:10 default.render.xml drwxr-xr-x 2 ajtown ajtown 0 Dec 3 00:10 .
We’ll take a copy of that locally to edit it, and then unmount it:
cp ./mtp/Internal\ storage/Android/data/net.osmand/files/rendering/default.render.xml . fusermount -u ./mtp
At this point, it’s probably worth a look at the initial version of the default.render.xml file. It actually contains information about how something might appear depending on how a number of different flags are set (in that file you can see things like “nightMode”, “moreDetailed” and “baseAppMode” flags). To see an example of that look at this code - you can see how a bridleway might be rendered from zoom levels depending on various flags, all inside a top-level
<case minzoom="12" tag="highway" value="bridleway">
test. The last line of that section
<apply color_5="$bridlewayColor" pathEffect_5="4_2"/>
sets the colour to one defined above.
One of the simplest changes that it’s possible to make is simply changing the rendering colours.
Let’s make a simple series of changes from e.g.
<renderingAttribute name="heathColor"> <case attrColorValue="#d7ebba"/> </renderingAttribute>
to
<renderingAttribute name="heathColor"> <case attrColorValue="#E6E8C5"/> </renderingAttribute>
(and a number of similar ones to match the less “in your face” colours here.
The resulting XML file looks like this.
Stop OsmAnd on the device, copy the new file to the phone, and unmount it:
jmtpfs ./mtp cp ~/src/osmand_files/default.render.xml ./mtp/Internal\ storage/Android/data/net.osmand/files/rendering/default.render.xml fusermount -u ./mtp
Here’s what it looked like before, and after:
Following recent discussions on OSM mailing lists about tag homogenisation it struck me that there probably wasn’t a good summary anywhere of the tools that people use to change the tags in OSM data into something that they can use. You might wonder why on earth we need to do this, given that OSM has natural language names for everything, but unfortunately many words used in OSM might have different meanings around the world, such as “city” and “highway”. However, this isn’t a diary entry about that - it’s a brief summary of what I’m aware of and have used for converting the “super detailed” tagging in OSM into something more appropriate for e.g. something rendering OSM data.
The documentation for this is here. I wrote a diary entry about how I use this method to change the “name” language of Welsh-speaking places in Wales to Welsh, and Scots Gaelic-speaking places in Scotland to Scots Gaelic.
It’s pretty simple to use but it’s a once-off conversion designed for when you might use osmosis at data load - it’s not especially useful within some other process.
The main alternative for “things that you used to do with osmosis” these days is osmium, which is available both as a library and as a standalone tool. I’m not aware that it supports tag transformation directly, but it wouldn’t surprise me if it did (it can do almost anything else).
This is documented here. Out of the box, osm2pgsql contains a “filter_tags_generic” function that does a bit of filtering but doesn’t actually change any tag synonyms.
For a simple “change tag X to Y” example, have a look here in the lua tag transform definition that I use for the map at the top of this page. There you can see various “not very much used” tags (and in some cases “really shouldn’t be used”, such as “highway=unsurfaced”) changed to a “lowest common denominator guess”.
Elsewhere in that file you’ll see some tag values that you won’t see in OSM at all. These get handled in the rendering like this and the benefit is that you get rendering code that’s easier to understand - each “logical thing” has its own rendering code and it isn’t cluttered by lots of further tests either in the .mss or the .mml that has to say all the way through “is it X or Y”.
One of the things that the maps here display is England and Wales’ “public rights of way” (tagged in OSM with a “designation” tag). I use series of tag transformations to display different legal statuses in different colours, and paths without that legal status in grey.
osm2pgsql is typically used when creating a database for rendering or doing geocoding from. Alternatives exist, such as imposm. I’m not aware of alternatives that support tag transformation, but that doesn’t mean that there aren’t any.
mkgmap is typically used by people converting OSM data into something you load onto Garmin devices (eTrex, Nuvi, etc.). If you download an OSM-based map for Garmin from somewhere, the chances are that it was created with mkgmap.
Garmin’s internal format has a series of hex codes for everything, so essentially everything in a Garmin style definition is a series of tag transformations (like this one for “points”).
In that example you’ll sometimes see two different things at the left-hand-side converted to the same Garmin feature (e.g. “amenity=conference_centre” and “amenity=convention_center”) and tests combined with an OSM tag (e.g. “amenity=fuel & shop=convenience”).
I wrote a diary entry a while back explaining how to do very simple changes to a Garmin map style.
OsmAnd is probably the most powerful, most customisable OSM renderer that almost no-one thinks of.
Out of the box you get the ability to download .obf files from OsmAnd’s servers (up to a limit if you go for the free version) or you can create your own maps using OsmAndMapCreator.
What’s less well known is that it’s fairly simply to change the rendering style to something that you have created (see here) and that that can be done independently of the .obf file loaded. The tricky bit is that OsmAnd’s default style definition is a bit of a monster, as it seems to contain “all the style variants that OsmAnd can display” with lots of if/then/else logic for e.g. “is it currently night” and “are we in a car” etc. You can however add your “extra style rules” to a separate file to keep it simpler.
In addition, you can also perform tag transformations directly in OsmAndMapCreator - see here for the sort of thing that already exists, for example.
I’m currently working on a version of the two OsmAnd rendering files that perform the same “designation” processing that the online maps that I use do, though that’s some way off. It uses the same method as the osm2pgsql style.lua - change the things that I am interested in to a new tag name based on other OSM tags, ensure that gets added to the .obf file, and then render it as required.
There are lots of others that I haven’t used - this list post mentions the routing engine Valhalla, and I’m sure there are others.
There are a lot of peaks mapped in OpenStreetMap, ranging from proper mountains to mere pimples. They’re usually rendered the same regardless of how high they are:
However, with a bit of preprocessing and a bit of help it’s possible to render higher ones at a higher zoom level:
and when all are displayed, only display the names of the higher ones:
It’s also possible to make use of the topographic prominence if we have that data too (as there is for lots of peaks in Scotland):
In that example we’re deliberately not emphasising Sgùrr Eilde Mòr, not named in that image, because its prominence is only 271, despite it being over 1000m high, whereas Stob Coire Easain is shown sooner because it has much greater prominence.
I was prompted to write this because of a few discussions over the last couple of months - people talking about the default tags on various OSM elements, reactions to Amazon mappers’ additions of “last mile” service roads and discussions about how to tag England and Wales’ public rights of way network.
This is very much a personal view and one that’s very “England and Wales” centric - there are lots of different approaches to rural tagging in general and footpath tagging in particular, and many places around the world (enlightened places like Scotland and Scandinavia) have different default access rights to the curious set we have ended up with in England and Wales.
Let’s start with an example - what’s the reasoning for the tags on this way in OSM?
Firstly - “highway=service”. That’s simply the physical type of highway that it is most like. Functionally it offers access to the farm and other businesses. Maybe people might argue it should have a driveway tag; maybe someone might think it was a paved track, but service is what it’s been in OSM for 9 years and it’s near enough for me not to change it.
Next, “designation=public_bridleway”. Where does that come from? The answer, I think is roughly here back in 2009. That thread (and parallel ones on talk-gb at the time) discussed how to tag a thing that’s legally one thing, but physically a bit like something else as well, very similar to “my” service road. The source for me to add the tag here was “Public Bridleway” signs along the route.
Next, the basic access tags - “foot=yes”, “horse=yes” and “bicycle=yes”. These say that you’ve got a legal right to walk, cycle or ride a horse up here - they’re also implied by the “Public Bridleway” signs. To an extent it makes sense in OSM not to have duplicate tags for things (it’d be a pretty unusual “highway=motorway” that needed “motor_vehicle=yes”), but it’s not really fair on routers that have to work internationally to expect them to know every nuance of “designation” tagging (which in this case doesn’t even apply to a whole country).
So far, that’s pretty close to the National Trust’s proposed public right of way (PRoW) tagging here. There’s one difference - they’d suggest “=designated” instead of “=yes” on the access tagging, so it’s looking at where that came from. There’s a discussion about the use of those keys in the context of England and Wales PRoWs here.
The reason I’ve not gone with “=designated” on this service road is that it has it’s own meaning documented here - think of a sign people of transport mode X to use a particular route. An example of this is here near Sheffield - there’s a poorly-signed public foootpath across the road there, but signage diverting pedestrians along the road via the roundabout (playing human frogger on a dual carriageway is not a good idea). The usage of “=yes” vs “=designated” on “designation” ways varies around England and Wales. Compare here (roughly the same number of each) with here (hugely in favour of “=designated”).
Other tagging would also make sense, such as a surface tag on the service road (I added one here but forgot further on), but there’s always the option to add more detail later.
Returning to one question from the start of this diary entry - do Amazon mappers’ additions additions of “last mile” service roads cause a problem? Imagine if someone added a track or driveway between two public roads that wasn’t public - wouldn’t it confuse people into thinking that there was a public route between the two when there really wasn’t? I’d argue not really, since without access tags you simply can’t assume (in England and Wales) that you have access along a track or driveway, even on foot. Where people have mapped access properly it’s easy to see where you know you definitely can go and where you’re not sure, and someone can always add access tags later - OSM has always progressed incrementally like that. This is if you’re using a map that shows it, of course but since 2009 I’ve always used e.g. Garmin maps that do.
Best Regards,
Andy
(following on from this diary entry)
Supposing you would like to overlay a set of boundaries over the top of any existing map style - what would you need to do?
Using this as a reference, create a database to hold the new boundary data.
If a historical data file is what you want, download the data for the desired date and rename it to a file name that the “update_boundaries” script will recognise:
-rw-rw-r-- 1 renderaccount renderaccount 459723464 Nov 2 01:46 ukraine_2018-11-01T21:14:02Z.osm.pbf
You can also make other changes to this data if you wish using tools like osmium, osmfilter, osmosis, etc.
Modify this script to process your data.
Run the script to load the data that you have already downloaded:
sudo update_boundaries.sh current
Create a version of “mapnik.xml” (or whatever style file you are using) that looks in the database that you’ve loaded your boundary data into. In this case I just changed “dbname” references from “gis” to “gis3”.
Ensure that “/usr/local/etc/renderd.conf” knows about the new tile layer. The default one I use just has “ajt” in it; I created a copy of the last section, called it “ajt3” and changed URI and XML values.
[ajt3] URI=/hot3/ TILEDIR=/var/lib/mod_tile XML=/home/renderaccount/src/openstreetmap-carto/mapnik3.xml HOST=localhost TILESIZE=256 MAXZOOM=20
Restart renderd and apache2, and your new tiles will appear. Then in a simple Leaflet map add an overlay layer for your new tiles. This is an example that shows Leaflet overlays but there are many, many others, including on the main Leaflet site itself.
Here’s what it looks like:
It’s not perfect - there may be name label collisions between the two layers of data, especially as the labels from OSM Carto haven’t been removed (unlike with this style). It is however a very easy way to see boundaries corresponding to a particular viewpoint over the top of existing map tiles.
If you find downloading a processing OSM data too complicated, then you can just use the tiles I created for this example. An example tile is:
http://map.atownsend.org.uk/hot3/6/38/22.png
and the generic definition for a Leaflet overlay layer is:
var somenameUrl='//map.atownsend.org.uk/hot3/{z}/{x}/{y}.png';
Note - something like this approach to boundaries is something that’s been written about before. See for example this diary entry by PlaneMad from 2 years ago that replaced one set of boundaries in the data with another. This approach is I hope easier, although it doesn’t change the actual data used to create the main map tiles.
When I developed the map style that you can see here from OSM-carto one thing that I didn’t carry forward was the representation of borders. Partly this was because it’s not easy to separate maritime ones from non-maritime ones, the the purple maritime country borders frankly don’t look very nice:
Partly it was also due to “clutter” - I didn’t want an arbitary boundary to get in the way of a river or stream or other feature.
Sometimes however, it would be useful to see what borders are where, so why not do it as an overlay? This turned out to be rather easier than anticipated.
Firstly, a separate database was created (called “gis2”; by default an OSM Carto-based style will use “gis” as a database). That was set up as per the createdb section of the switch2osm guide. The reason for using a separate database was mainly to allow boundaries to be loaded independently of the rest of the data (and for a larger area if desired), but it would also allow different sets of boundaries to be loaded, if that was needed (some countries have legal requirements about how they are displayed on maps, and these may not match on-the-ground reality or the requirements of neighbouring countries).
The next job was to load some data into it. Normally I’d reload data using this script. For boundaries I created a version of that that used osmium, osmconvert and osmfilter to select only boundaries from the data and remove non-boundary tags that might render too.
After processing, the file with just boundaries in it was around 1/30 the size of the original file, so loading the second database with all of Europe’s boundaries wasn’t a problem.
Next the style needed to be changed so that boundaries were displayed over transparent tiles. This was done by changing water-color and land-color in style.mss to “rgba(0,0,0,0)” in each case.
Finally and most importantly, the style needs to read data from “gis2” rather than “gis”, and that’s defined as the “dbname” in project.mml.
The results can be seen here. Select “Controls” and then “Boundaries” from the layer switcher to turn them on and off.
An example of a map without boundaries is:
and with is
Surprisingly, even though walking routes and boundaries are both labelled in purple, because boundaries are the only purple linear feature, there isn’t any real confusion between the two.
