This isn’t directly related to OSM, but many times in “software” or “development” OSM channels someone has asked “how do I do X … on Windows 10?”.

The reason that’s often a question is that, for historical reasons, much of the OSM software used to do things has been developed on Linux. That’s great for people working on Linux themselves, and the Apple Macintosh users can normally jury-rig something to run that software too.

Windows users can do the same too, by installing “Windows Subsystem for Linux”. Within Windows, search for “Windows Features”:

In Windows Features, tick the box for WSL and click OK:

Windows will want to restart

After the reboot, search for “Ubuntu”:

As noted in the comments, you may need to add “Ubuntu 22.04.1 LTS” from the Microsoft App Store here. When I deleted and reinstalled WSL and Ubuntu to create these notes, I didn’t need to - but it may be that a Microsoft uninstall didn’t do a clean uninstall. Plus ça Change, eh? :)

Ubuntu will install:

and will then prompt for a userid and password:

After that type “lsb_release -a” at the prompt to see the version installed:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Apply any pending updates:

sudo apt update
sudo apt upgrade

(this will take a while the first time)

and you can then head over to e.g. https://switch2osm.org/serving-tiles/ to do whatever it was you wanted to do.

Just after the New Year, there were a few reports of people getting stuck and having to be rescued while hiking in the English Lake District.

There are writeups of this in “The Great Outdoors” magazine here and on Alex Roddie’s blog here. They really are worth reading - the consensus on IRC, Mastodon etc. was that the quality of the article was streets ahead of what you might expect to see in the generalist national press.

Lots has been written about how map and app developers can try and convey information to a user beyond “this is a way of getting from A to B” (including an OSM diary entry that I wrote in December 2022), so I won’t go over that again.

However, one particular quote from the Great Outdoors article did stand out for me, and it’s this:

A spokesperson for Keswick Mountain Rescue Team, which carried out the rescues, said: “There is no path via this route – only a scramble of loose scree which also requires the walker to negotiate the rocky outcrop of Slape Crag. It’s the scene of previous callouts.”

The reason why that stood out is that “path” can mean different things to different people:

• an actual signed path suggesting that people on foot are encouraged to go via a particular route. In England and Wales there’s the concept of “public footpaths” (and public bridleways), which is a legal right of foot (and horse) access across what might be otherwise private land.

• some indication on the ground that people often go via a particular route

Often the two coincide, and “where you’re supposed to go” matches “where people do actually go” correspond. Sometimes, however they don’t.

As an example, let’s imagine you want to go from the road north of Carlton near Helmsley to the old ruined buildings at Stork House, across the moor. That’s on Ordnance Survey map OL26, and you can also see that mapping at Bing maps here. You’re expecting the public bridleway to follow a track until a small bridge and then head left across moorland to Stork House.

At the western end there’s a public bridleway sign:

and the track is pretty clear:

So off we go, accompanied by the calls of local grouse letting everyone know there is someone unexpected around. Soon, however, there is a bit of a problem - there’s no path visible northeast from the bridge

and there’s no “public bridleway” signage. Let’s carry on along the track for now. As the OS 1:25k maps attest, the moorland area is all access land, so public access on foot is allowed. Eventually we get to the guidepost here:

Although the paint has mostly worn off, it’s just about discernable as a “public bridleway” sign (well, it was last year when I first mapped it - a year later when I took these pictures it’s a bit less obvious). From there, the path that we need to take to get to Stork House is clear:

Soon, we can see the ruin of Stork House:

Not visible in that picture are the rabbits that scattered as soon as I walked down the hill. Eventually, we reach the point where the alleged bridleway that didn’t exist on the ground “joins” the path that we’re on:

That Garmin picture shows an OSM-derived map that I created superimposed over OS Explorer Mapping. The green dashes are the alleged bridleway, the brown dashes are OSM paths, and the blue line is the route I’ve walked. So is there any sign of the bridleway marked on OS’ mapping from this side?

Not really:

In that picture, the path we’ve walked in on from the southeast is on the left, and the bridleway route would go across the moorland to the right. There’s nothing obvious there. If we go through the gate into Stork House itself we can see that there are two plastic “public bridleway” signs:

The blue colour has almost entirely worn off so the direction shown isn’t clear, but one presumably points back the way we came and the other to the northwest, where we’re expecting another bridleway to go.

In conclusion

One thing that I think we can say is that the non-existent-on-the-ground public bridleway doesn’t belong in OSM as any sort of path, as there simply isn’t a path on the ground there. Had there been fingerposts across open moorland then that probably would have been worth adding (perhap with a “trail_visibility=no” tag), but as there’s not even that, it’s really not worth adding as a path, despite appearing on OS maps and (in this area) being available from the local authority under an OSM-compatible licence.

As an aside on “trail_visibility”, it has been used around here. For example, the path across the road from where we started from is here:

and isn’t very clear on the ground either, hence the “trail_visibility” tag on it.

The second conclusion is that “on the ground legal signage” of routes can diff from what the local authority thinks the route ought to be, and where it does, and seems on survey to be “more correct” then it’s best that OSM reflects that, rather than the local authority’s view. That’s why, in OSM, I’ve marked the signed route a the bridleway (the blue dashes and dots here. The red line on that map is the overlay of local authority rights of way. Turn that off and you’re looking at data from OSM only.

Copyrights etc. in maps and screenshots:

• OSM map data wherever it appears is © OpenStreetMap contributors
• PRoW overlay data is © local authorities under OGL & rowmaps.com
• The Bing map is labelled by Bing as © 2023 Microsoft
• The OS data on the Garmin picture is Ordnance Survey © Copyright 2023

(or, what’s wrong with the general purpose map on my phone that just seems designed just to show me adverts?)

I decided to write this after reading some of the comments on the wiki here and in the community forum here.

People tend to view “things that appear on a map” as “somewhere that they can go”. For example here:

someone who didn’t do their research properly might think that they can just wander along that path to the top of Everest from the south, despite the underlying data suggesting “difficult alpine hiking”. This diary entry is an attempt to try and avoid such misunderstandings.

What options does a map have to different things?

The map at map.atownsend.org.uk shows linear paths and tracks as a single line, It shows legal rights of way (some special categories in England and Wales) and also tries to give some indication of what a walker might expect on that path.

First, before we talk about “showing what a walker might expect”, a note on what options maps have to show linear things differently:

• colour
• width of line
• pattern along the line
• other things drawn alongside the line

How are those shown?

The colour of the line is used for the legal designation - brown for “motor-vehicley” ones, blue for “horsey or cycley” and red for “foot only” and grey for no legal designation, like this:

In addition, an overlay colour is used for access rights:

The width of the line is only used to control the “visual weight” of features on the final map - it’s tuned by colour at each zoom level, so that a particular sort of feature does not dominate the map at that zoom level.

The spacing of the pattern along the line shows the width of the path (widely spaced for wider than 2m, narrow spaced for less than):

Also, slightly different patterns are used for some slightly different legal designations (for example, bridleways and restricted byways).

“Other things drawn alongside the line” are used for tunnels, long fords, bridges and embankments, for example:

What other OSM tags do we want to show?

“informal” is sometimes used to show that a path has been created by users rather than by sanctioned by land managers. This only makes sense in places where there are formal paths (which isn’t everywhere), and an informal path isn’t necessarily in any way illegal to use.

“trail_visibility” is used to indicate how easy it is to see a path on the ground.

“sac_scale” is used to show how difficult a path is to use.

The first two are essentially about showing that a particular path might be difficult to see, or that a walker might not want to follow it because it’s not an “official path”; the third is about how to show (and in some cases not to show at all) paths that might actually be dangerous to an unsuspecting walker.

A final tag that isn’t shown is smoothness. Unlike other modes of transport, a low sac_scale path won’t become impassable to most pedestrians because it’s not very smooth.

What is actually the goal of this map style?

It’s designed to show all legal rights of way there is an actual path (theoretical routes exist where there isn’t really a path - the walk across Morecambe Bay is one example).

Paths that are difficult (high SAC scale), informal or not very visible are de-emphasised in the rendering.

Very difficult paths and very poor visibiliry paths that are not public rights of way are suppressed from the default rendering; an overlay is available for these.

What does this mean in practice?

Here’s a section of bridleway in the North Yorkshire Moors with a “trail_visibility=bad” section in the middle:

An informal human-created shortcut off the Foss Islands Track in York

West of Sca Fell, with “difficult” paths excluded by default:

The same view, but with the overlay of “difficult” paths turned on:

The net result is that

• walkers who see a path on the map will see a path on the ground
• paths that are less clear on the ground will be de-emphasised on the map

The problem

Historically, the map at map.atownsend.org.uk didn’t change the URL as you moved around, and you had to press a “permalink” button to save the current location. This wasn’t ideal, so I had a look around for alternatives. Helpfully, the Leaflet site has a list of suitable plugins. Of those, leaflet-hash was suggested elsewhere. That seems to be still in use, although it hasn’t been updated for a while. It is spectaculary easy to use, and “does what it does” with no problems.

The challenge is that it stores the current location (for example “https://map.atownsend.org.uk/maps/map/map.html#11/54.4900/-3.1098”) but doesn’t store the current layer. That map currently has a choice of 5 base maps (test instances have 6) and 5 overlays:

How to solve this

I had a look for an alternative, and the nearest that I could find was leaflet-hash-plus. The hosting of that seems not to be there at the moment, so the demo does not currently work, but the code does (if you host it yourself). It’s a bit “some assembly required” though - the way it works is that it allows you to encode layer information in a “meta” array and store and retrieve that when needed, which is a bit of a challenge for someone who (like me) isn’t a Javascript developer.

To do that, I need to add handlers for events so that the code knows when the base layer changes or when overlays are added or removed.

The “meta” that I use is just a list of layers, so “HPN” would mean that the default base layer is used, together with the “LA_PRoW” (“P”) and “No vis paths” (“N”) overlays. When the URL changes, the code must change the displayed layers to match. When layers are changed manually using the layer selector, the URL must change. The code makes sure that only one base layer is selected, and overlay layers are removed as required. The mapping between tile layer URLs and the representation in the current site URL is handled here.

What we can now do

The URL https://map.atownsend.org.uk/maps/map/map.html#16/54.4540/-3.2155/H/N turns on the “No vis paths” layer so that this path appears. It’s suppressed by default on the default layer because “sac_scale” is set to “demanding_mountain_hiking” and “trail_visibility” is set to “bad”, and it doesn’t have a designation tag on it.

Boundaries are turned off by default, but a URL of https://map.atownsend.org.uk/maps/map/map.html#8/54.143/-4.878/H/B turns them on.

This (https://map.atownsend.org.uk/maps/map/map.html#15/54.3771/-0.5528/H/P/N) shows an overlay of the legal rights of way over surveyed paths of varying visibility, and how they significantly differ from what is actually there.

The OSM wiki contains links to various downloads for Garmin maps, as well as some information about creating your own from OSM data.

For many years now I’ve used maps on Garmin devices that I’ve created myself. I’ve done this so that I show things like England and Wales public footpaths and bridleways that I’m not aware that other maps can show. Until now, I’ve never quite got around to making the mechanism publicly available, or indeed the maps themselves. I’ve finally got around to fixing that.

The “readme” here explains how to run the script yourself, if you want to do that. If you don’t, you can download the latest files for Great Britain. New maps should appear there about weekly.

Differences between these maps and other maps for Garmin devices include:

• They’re routable. Many maps for Garmin devices are, but not all.
• Public Footpaths have “(FF)” added to the end of the name, or used instead of the name if there was none.
• Public Bridleways, Restricted Byways and Byways Open to All Taffic similarly add “(FB)”, “(RB)” or “(BY)”.
• Members of foot and hiking routes have the name of the routes in brackets after the road name, if any.
• Ways inaccessible on foot (private tracks etc.) are removed from the map.
• Pubs and bars selling real ale have “PQR” appending to the name

Some of these mirror how things are represented on maps at map.atownsend.org.uk. Those maps use osm2pgsql, osmium, osm-tags-transform and lua scripts to determine what is shown on the map.

These Garmin maps currently only use mkgmap style files such as this one, which isn’t quite as flexible. Eventually I may get a change to change that and do more of the transformations in lua which should and also allow more precise changes to be made, and also reduce the size of the style files,

Since the release of Ubuntu 22.04 LTS, it’s now much easier to keep a tile-serving database updated. This version of Ubuntu ships with much more recent versions of osm2pgsql and osmium, so it’s easier to set up updates using those.

First things first, the new “setting up a rendering server” instructions for Ubuntu 22.04 are here. These are much shorter than they used to be because most of the software setup is done automatically as it is installed.

osm2pgsql-replication

The first option is to use osm2pgsql-replication. The manual page for this is here. The switch2osm page walks you through initialising replication, and creating scripts to apply updates and mark tiles as “dirty”, so that they will get rerendered as required when the underlying data has changed. Download sites such as Geofabrik and download.openstreetmap.fr provide extracts and replication feeds for those extracts - for example, you can load data for Europe, and then apply updates only for Europe - your database will not grow with other updates from all around the world.

PyOsmium

Another option, slightly more flexible, is to use Pyosmium. The documentation for this is here. Again, the switch2osm page walks you through initialising replication, and creating scripts to apply updates and mark tiles as “dirty”, so that they will get rerendered as required when the underlying data has changed. Example scripts are provided that use Zverik’s trim_osm.py script to cut-down replication update files to the area that you’re interested in.

A script to measure that measures the current replication lag and a script to call that from munin are also included.

Alternatives

Osmosis

Using osmosis should still work with Ubuntu 22.04, but comes with rather more dependencies.

Using osm2pgsql-replication or Pyosmium in earlier releases

This should also be posible if you build from source.

It’s been a while since I wrote about updates here.

Map.atownsend.org.uk is a map style that started out as a fork of OSM Carto in 2014, when it was obvious that the direction of travel for that style was towards “a map that was nice to look at” rather than “a map that was useful”.

For example, here’s the area that I linked to then in OSM Carto now:

and here it is at map.atownsend.org.uk:

A key difference is the ability to see a walking route from one side of that map to the other.

The map style is in github - see the links at the top of the changelog. It’s designed to be a pedestrian-focused style, in particular showing England and Wales’ “public rights of way”.

From a design point of view, one of the key choices was to try and overload basic features with more detail. An example of that was described previously here. With the pubs that that diary entry describes, 8 different attributes are considered (“does it serve food”, “can I get a wheelchair in there”, etc.). There are now about 300 different combinations, but they’re all recognisably variations of the same basic icon. It’s a similar story with footpaths and paths - there are about 50 variations of those.

Technically, the approach taken was to lean heavily on lua tag transforms - invoked when the database is reloaded and also when data is loaded into it on the fly (the site applies updates every minute). Changes to the style that involve lua (and most do) do need a database reload. This is normally done once a month or so, and takes a few hours overnight - if 24*7 uptime was needed it’d be easy to switch between a pair of databases or servers (people with very long memories might remember the A/B approach of system like IBM’s System/38), but that hasn’t been a priority so far.

Making extensive use of lua has huge advantages - it makes the styling code much easier to write, since there need to be fewer complicated and brittle selection statements in the styling code. As an example, look at the code added to render parcel lockers in 2017. At the time, before “amenity=parcel_locker” was widely used, variations on vending machine tended to be used. The corresponding rendering code change is then really simple - just the reference to an icon and a name. It would also be possible to refactor the main two parts of the style to make far more use of this and vastly simplify much of the selection logic from the original style, but that hasn’t been a priority.

Another thing that lua processing makes easy is the ability to show walking, cycling and horse riding routes - the lua code adds tags so that the rendering code can render them a a dotted line. For my money (used as I am to other familiar UK map styles) these are absolutely a key part of a general purpose map.

With that in mind, it’s worth looking at some of the changes since the changes made to show zero-waste shops using a small “E” as a modifier to existing shop icons, described in a previous diary entry.

Many former phoneboxes now contain defibrillators, mini-libraries. tourist information points, etc. These are shown using a unique, but phonebox-derived, icon. Phone boxes that aren’t red are shown with an appropriate colour:

Stations on preserved and miniature railways are shown with the name in “tourist” colour. Grosmont has two stations, one on the national rail network, one on the preserved North Yorkshire Moors Railway:

If a walking route has no name but does have a colour, the colour is used as the name:

The style already used the trail_visibility tag to render some paths and tracks less overtly; this has been extended to also include various informal and overgrown paths. In this example an informal north-south link path is shown; as is a section of razed railway, with slightly different dots:

Some mappers have been mapping closed shops etc. with “opening_hours=closed”

Locked gates are shown with a “locked” variation on the gate icon:

Footpaths and cycleways alongside a road are often tagged with the name of the road. This map style supresses this names, and “is_sidepath” has been added to the list of tags that forces this removal. For example, the tags here:

show like this:

The context of this diary entry is of course lots of discussions about OSM’s “standard” style, “OSM Carto”, particularly this bin fire of a pull request. One of the maintainers has written in detail about the process of changing the style here and especially here. As an aside, I disagree with a couple of the points there, in particular “we outgrew the possibilities of the technical framework we were using” - two of the three pinned issues here are essentially “as yet unexploited technical possibilities”. I’d also suggest that more widespread lua use, which I’ve tried to suggest the benefits of above, would be hugely beneficial if it was to be made use of.

That said, the triangle in imagico’s blogpost explains the conflicting pulls on the project, and I actually believe that the maintainers of OSM Carto have an impossible job, because the cartography guide contains these as the first two goals:

• It’s an important feedback mechanism for mappers to validate their edits and helps to prevent unfavorable fragmentation of tag use.
• It’s a major part of the public face of OpenStreetMap, for many people the map on osm.org rendered with this style is OpenStreetMap.

I believe that these have been in conflict for a long time (it’s essentially the problem I commented on in 2014). Both requirements are important but (as I’ve said elsewhere) I don’t believe that one style can meet both. We used to have a “show everything” style (Osmarender) that prioritised detail over cartography. Right now we have the data layer:

but that relies on individual objects rendered by the browser client and simply isn’t practical at anything other than the highest zoom levels. At osm.org we currently have 6 map layers - “OSM Carto”, two cycle maps, two public transport layers, and the “Humanitarian” layer. Arguably none of them satisfy OSM Carto’s “important feedback mechanism” goal.

For completeness, it’s also worth mentioning the OWG’s “New Tile Layer” policy here - interestingly, complementing the existing osm.org map styles isn’t one of the hard criteria set out there, although soft criteria are that it should be “unique” and “interesting”.

Over at map.atownsend.org.uk I use a wider variety of icons for different sorts of objects within OSM. I previously explained here how simple changes to a basic icon make it still recognisable as the basic object (“this is a pub”) but also convey more information (“this particular pub is wheelchair accessible, and has a beer garden”).

The same principle can be used with “shop” icons - a selection of different types of shops can be shown differently if they have one of the tags commonly used in OSM to indicate “zero waste” or “eco” shops. This is done here in the lua file that processes all OSM data before loading into the rendering database. In the case of “zero waste supermarkets”, the code looks like this:

if ((  keyvalues["shop"]               == "supermarket"         ) and
    (( keyvalues["zero_waste"]         == "yes"                )  or
     ( keyvalues["zero_waste"]         == "only"               )  or
     ( keyvalues["bulk_purchase"]      == "yes"                )  or
     ( keyvalues["reusable_packaging"] == "yes"                ))) then
   keyvalues["shop"] = "ecosupermarket"
end

Then all that we need to do is to create a new icon and make sure that that icon is used:

[shop = 'ecosupermarket'][zoom >= 16]::shop {
  point-file: url('symbols/shop_ecosupermarket.p.16.png');
  point-placement: interior;
}

and that the name for “ecosupermarkets” is displayed just like it is for “supermarkets”:

[shop = 'supermarket']::shop,
[shop = 'ecosupermarket']::shop,
[shop = 'department_store']::shop {
  [zoom >= 16] {
    text-name: "[name]";
    text-size: 9;
    [zoom >= 20] { text-size: 18; }
    text-dy: 11;
    text-fill: #939;
    text-face-name: @book-fonts;
    text-halo-radius: 1;
    text-wrap-width: 20;
    text-placement: interior;
  }
}

The “lua” script is called for every object whenever the database is reloaded, and also called every time it is updated with whatever people have edited in OSM, which typically happens 5 minutes or so after the edit takes place.

In England, the Environment Agency has an extensive network of river level monitors that are used to provide flood warnings such as (at the time of writing) this one.

However, these alerts tend to be only issued when land or property is likely to flood, and there are many paths that flood much earlier. An example is here, where if the downstream station reads just over 1m, you’ll get your feet wet. The Environment Agency doesn’t issue a warning at that level because it’s well within the normal range, and the river’s still well within the levees that have been provided for it for the last 50 years.

It’d therefore be useful to know which paths are likely to be flooded, and this can be done by noting what upstream and downstream measured levels were when a path was or was not flooded. In this case the downstream level is the more reliable indicator since lots of land drains empty into the river below the upstream station and there’s nowhere for the water to go before the downstream one. This has to be done on a case-by-case basis; EA’s flood maps are great for predicting low-level land that’s at risk but don’t have enough granularity to help here.

I’ve done this here. The map is a simple overlay available at https://map.atownsend.org.uk/, and it’s currently using the same database as the normal map there. The “is it currently flooded” information is obtained from EA’s station information every few hours (and station data is cached locally to avoid multiple unnecessary reads of that site). When a path is flooded, it shows in blue, as can be seen above. The overlay tiles are removed before each resample to avoid any tile caching issues. This map isn’t designed to replace EA’s own maps - when there are warnings of property flooding many areas where I’ve never measured “path X floods when station Y reads Z” will be affected.

Currently the comparison data is in the script itself not in OSM - I did ask about how it might be possible to do that earlier, but the replies seem to mostly miss the point that I was actually already recording “is it flooded or not here” rather than just guessing current flooding issues based on EA station values. It it proves useful to add more data it might be worth moving the comparison values into OSM itself (in the spirit of “any tags you like”), but that’s not a priority.

This approach won’t work everywhere. Nearby, Clifton Ings is one of many areas regularly deliberately flooded as part of flood prevention measures, but someone literally has to open a barrier to let water in and open another one to let water out - and presumably that is based on a mix of current and predicted future rivel levels, in turn based on previous and predicted rainfall elsewhere.

I’ve added a new page to the switch2osm guide: “Manually building a tile server (Ubuntu 21.04)”. The work that the Debian developers have done to include the required software inside Debian 11 is now in Ubuntu 21.04, so there’s actually less setup work needed than in previous Ubuntu versions.

Ubuntu 21.04 isn’t an LTS release, so Debian 11 or Ubuntu 20.04 are still the recommended options for servers that are expected to stay around for a while. One of the regular faux pas with the Natural Earth download site has just been and gone, but the guide includes a workaround in case problems reoccur there.

Unrelated to Ubuntu 21.04 directly, there’s currently a bit of an issue with the lack of feedback that part of the OpenStreetMap Carto setup gives to the user, but work is under way to improve that (I hope things will be able to return to where they were last year with this). I’ve tried to explain here and in the other guides that it’s to be expected that “things just go quiet for up to an hour”, and that eventually it will complete.

Previously I wrote about checking National Parks in Britain and Ireland for breakage, by counting the number of valid polygons in a rendering database.

A similar approach can also be used to check line features as well as polygons. It’s possible to count the number of contiguous sections of a given polygon. For example, here’s the Lyke Wake Walk, which is in one piece:

psql -d gis -c "SELECT count(*) FROM planet_osm_line WHERE (osm_id = '-1996318');"
 count 
-------
     2
(1 row)

and here is the Sabrina Way, which isn’t:

psql -d gis -c "SELECT count(*) FROM planet_osm_line WHERE (osm_id = '-537430');"
 count 
-------
    12
(1 row)

If something is changed in OSM that introduces an extra piece, or fixes a gap, that number will change. This script can be used to count the change in the number, like this:

Change in relation components for Sabrina_Way 537430
3c3
<     12
---
>     13

If run from cron it can check every day and will only send an email alert when the number changes.

If there is a change, and can be investigated with a site such as http://ra.osmsurround.org/analyzeMap?relationId=6361532.

Edit 2: Also useful is the answer to this help question that looks for modified ways that make up a relation.

Edit 1: Where this process falls down is where a very long-distance path is added a superrelation and there’s a gap between the relations, which is what this changeset was fixing. Here two long several relations make up two super-relations that share some of their route. For completeness, the edit that I was fixing that created a gap in the Cleveland Way was actually fixing a previous edit of mine that introduced an unwanted spur to E2 at this location.

Taginfo is a way of finding out what tags and values are used within OpenStreetMap. There’s a worldwide instance here, and other instances exist, including various Geofabrik ones such as Antarctica here.

As well as describing keys and values used within OpenStreetMap, the worldwide instance of Taginfo also holds details of what other projects use particular OSM tags. The Taginfo/Projects page explains the update process.

Each project submits a .json file containing details of key and value usage, such as this one for an OSM Mug.

The one for the map style that I look after is here. Currently I’ve just submitted a list of keys, not values, but I have included a description for each one, as can be seen here for “designation”.

The first challenge was to create the initial list. There are references to tags in three main places in my map style - the lua style file, the carto .mml file and the individual carto .mss files such as this one. A bit of investigation (creating a unique sorted list of strings within style.lua on one hand and .mml and .mss files on the other found that although some keys do pass through style.lua unchanged, almost no keys that are later used in carto files aren’t mentioned in style.lua at all. That means that I only needed to worry about keys in style.lua. I created an initial .json in the correct format and then wrote a shell script to generate the “tags:” section of the .json file. I then filled in the descriptions where the shell script had just created placeholders, and checked it in.

Next, I wanted to be able to easily update the taginfo.json file after changes to the map style. Because I’ve also checked the previous generated file into github I can easily create a new one and compare the two:

sh -x ./taginfo_lua_genkeylist.sh > q.txt
diff taginfo_lua_nodesc.txt q.txt

for the most recent change the output of that is:

44a45
>     { "key": "colour", "description": "qqq" }
118a120
>     { "key": "lifeguard", "description": "qqq" }
119a122
>     { "key": "locked", "description": "qqq" }
163a167
>     { "key": "place_of_worship", "description": "qqq" }

So next I need to:

• Update the file with descriptions with those keys (taking care to make sure that the JSON is still valid, and that lines which need a comma on the end have one).
• Move the new output file (q.txt) in this example into place as the new “taginfo_lua_nodesc.txt”.
• Update taginfo.json with a new “data_updated” date at the top.
• Check everything in
• Ensure that within a day or so it’s updated in the project list.

In some cases the description can be obtained directly from the commit text, such as for “colour” here:

Show black, white, blue, green, grey and gold phone boxes as the correct colour rather than red.

Once that’s done, taginfo should update within a day or so, and updates will appear on pages such as this one. At the time of writing this diary entry, there are no projects listed for that key.

Then in a month or so’s time, after the next map style update, repeat the process.

The OpenStreetMap Community Index is designed to show people what community groups there are within OpenStreetMap, in any part of the world. If I look near me I see a local pub meeting (sadly on hold during the pandemic), some country-level resources, and then international resources.

On a couple of occasions I’ve seen things that are missing or need updating, and this diary entry is designed to complement the existing documentation to describe what needs to be done to fix that.

Creating a local copy

In my case I already have a fork of https://github.com/osmlab/osm-community-index at https://github.com/SomeoneElseOSM/osm-community-index. There’s a “fork” button in the top right of Github’s UI that allows you to do this.

Next, I’ll create a local branch to work on. The default branch at https://github.com/SomeoneElseOSM/osm-community-index is “main” and the UI will say “This branch is even with osmlab:main.” because (in your case) you’ve only just created it. In github’s branch selector just type the name of the new branch to create. In this case I created “add_forum_at_world_level”, because I noticed that there wasn’t a link to the OSM Forum at the international level, although there are many links to country-level forums from the community index.

Next, “git clone” your repository wherever it is you want to work. I’m working on a completely new virtual machine to do this, so I have to take a slight detour first - see “setting up ssh access to github” below. Once that’s done:

mkdir ~/src
cd ~/src
git clone git@github.com:SomeoneElseOSM/osm-community-index.git
git checkout add_forum_at_world_level

Note that I’ve used a version of “git clone” that is designed to work with ssh. Github is (rightly) moving away from password-based verification and will send you a warning email if you try and use a password during the transfer period.

To make sure that everything works before making any changes, we’ll build everything as it currently stands. The machine I’m running on is a virtual server running Ubuntu 20.04 LTS. It’s actually set up as per here so it already has Apache2 and some node.js software on it. The versions of node.js and npm are the defaults for this version of Ubuntu, and are:

node -v
v10.19.0
npm -v
6.14.4

An older OS with and older version of node.js or npm may struggle, and I’m not going into detail about node.js versions or package managers here, because it’s way above my pay grade.

cd ~/src/osm-community-index
npm install
npm run test

That checks that everything is OK. To build as is:

npm run build
npm run dist

That creates various files in e.g. ~/src/osm-community-index/dist/ including “completeFeatureCollection.min.json” (in previous versions called “combined.min.geojson”), which is all the data needed by the website as it runs. On my server “/var/www/html” is the root of the website. To copy the files into there:

sudo mkdir /var/www/html/community
sudo cp ~/src/osm-community-index/docs/index.html /var/www/html/community/
sudo cp ~/src/osm-community-index/dist/completeFeatureCollection.min.json /var/www/html/community/

Next, we’ll need to edit the “index” file on the web site so that it refers to the local “completeFeatureCollection.min.json”, not one from an external webserver.

sudo nano /var/www/html/community/index.html

In my case I’ll edit it so that:

var dataURL = 'http://192.168.1.66/community/completeFeatureCollection.min.json';

and restart apache:

sudo /etc/init.d/apache2 restart

If I browse to http://192.168.1.66/community/ the site looks exactly like the real one.

Adding a new feature

Next, we’ll make a change. This document describes the format in detail. In my case I want to create a “resource” at world level (“OpenStreetMap Forum”) but don’t need to create a “feature” since my new resource is worldwide.

If you think you might have to add a feature please check https://ideditor.codes first (iD and the community index share this). It is quite possible that your region may be already listed, since quite a few unusual values are already there. There are a couple of other caveats to be aware of as well.

I created my new .json file based on one of the other international examples. For “languageCodes” I chose “en” because most (though not all) of the posts at supranational level are in English. “contacts” is a bit complicated, since there are many top-level OSM forums and they don’t all have the same admins. I chose an address that I knew would at least result in messages being passed on to the right admin(s). For “order” I chose “something in the middle of the list”. The current top-level order settings are:

Facebook  3
Reddit    2
Discord   unset
Telegram  unset
Twitter   unset
OSMF      unset
Help      -2
IRC       -4

Given OSMF’s FOSS policy, I’m not convinced that Facebook should be at the top of that list, but that’s another issue for another time. Once all changes have been made, build again:

npm run build

Any errors with field contents should be displayed here. Fix this and run “npm run build” again before continuing. When complete:

npm run dist
sudo cp ~/src/osm-community-index/dist/completeFeatureCollection.min.json /var/www/html/community/
sudo /etc/init.d/apache2 restart

Now, when I browse to http://192.168.1.66/community/ the new entry appears. We’re ready to check in the changes. The current status of things can be seen via:

git status

To check in:

git add resources/world/OSM-Forum.json
git add .
git commit

“git commit” does a local update from “files you have edited locally” to “your local copy of the git repository on this server”. Next:

git push

That updates your github copy of the community index with the copy from your local server.

At this point, the branch in my github account says “This branch is 1 commit ahead of osmlab:main.” when I browse to it.

Creating a pull request

There’s an option from there to create a pull request, which is saying to the maintainers of https://github.com/osmlab/osm-community-index “hello, I think it’d be a great idea if you made this change”. There’s no guarantee that it’ll be accepted (there may be a good reason that I’m unaware of why that resource isn’t listed at https://openstreetmap.community/). If/when it does get accepted it’ll look something like this.

Setting up ssh access to github

As an aside, because this is the first time that I’ve needed to use this machine for anything ssh-related, I’ll create ssh keys and upload to github.

cd ~/
ls .ssh
ls: cannot access '.ssh': No such file or directory

To create a key pair:

ssh-keygen -t ed25519 -C "anemailaddress@example.com"

You’ll be asked for a passphrase - enter one and verify it. A file, in my case called “~/.ssh/id_ed25519.pub” will be created that begins “ssh-ed25519”.

In Github:

Settings
ssh and gpg keys
New SSH Key

Add ssh key
(confirm password)
(receive email about new ssh key)

When you get to the “git commit” part, if this is the first time that you’ve done that on this machine, you may be asked to “Please tell me who you are” and suggested to run:

git config --global user.email "you@example.com"
git config --global user.name "Your Name"

As you will no doubt have heard, browser support for flash went away at the end of last year. As you can read here, Potlatch 3 has been released and downloads for Windows and Mac are available. Linux users are asked to “Use Wine to run the Windows installer”. You may be wondering “why?” - there’s a section at the end of this diary entry that covers that. First, the “how”:

What do I need to do?

There are lots of different Linux distributions out there. Most have some sort of package manager and in most of those “wine” will be available as a package. Here is what I needed to do on a couple of desktop Ubuntu systems (both using KDE) - one running 18.04 LTS, the other 20.04 LTS.

Firstly, install Wine itself:

sudo apt install wine64

Next, you’ll probably need to install some fonts that software that normally runs on Windows needs. You can do this via a helpful script Winetricks. First, install cabextract, which is needed by the script:

sudo apt install cabextract

Next, decide where to install the “winetricks” script. I’ll put it in an existing user “bin” directory, so:

cd ~/bin
wget https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks
chmod +x winetricks

Next, install the fonts:

winetricks corefonts

Finally download Potlatch 3 (choose the “Windows application” option, which is a zip file) from the download page, unzip it, and run it:

wine64 Potlatch.exe

Follow the instructions that are presented and accept the defaults to “just install” it.

Running Potlatch 3

When Potlatch runs, you’ll initially see the whole world zoomed out. Rather than zooming in manually you can then “edit with remote control” from the openstreetmap.org site to go to wherever you are currently looking at there, and you can also search for any sort of OpenStreetMap URL to jump straight to it. “GPS / My tracks / load” loads GPS traces.

A resulting changeset can be seen here. Somewhat confusingly, it says the OS was “Windows 7” because that’s what Potlatch thought that it was running on!

What might I still need to be careful about?

As well as clicking “save” at the top left (or pressing “s”), also don’t forget to close changesets by pressing “c” when needed.

If you try and do too much too quickly it can fall over. To reduce the chance of this happening:

• Wait for imagery to load before moving the map again
• Press “save” before moving the map. If something does fail, at least you won’t have lost any edits.
• If you know you want to move the map to somewhere else manually (scrolling, rather than using search), set the background to “none” first.

Why not just run a Potlatch 3 AIR file on Linux?

This is available from the download page, but there are some problems. These include:

• The AIR downloads have disappeared from Adobe’s site. Harman is now looking after it, but it’s not immediately clear where you’d start from there.
• The Linux runtime is very, very old and doesn’t support the latest transport layer security versions, resulting in SSL issues.
• The old Linux versions are 32-bit, meaning that there are hoops to jump through to get the necessary libraries in place on a modern 64-bit Linux system.

None of these issues occur with Wine:

• A recent version of AIR can be distributed within the Windows executable
• It supports TLS 1.2
• 64 vs 32 bit issues are handled automatically by “Windows Side by Side” (WinSxS). Neither you nor Potlatch need to worry about it.

What else might go wrong?

I’ve found one machine, an old Chromebook running Ubuntu 18.04 LTS via Crouton that “just sat there” when trying to run the P3 self-extracting .exe for the first time. It may just be that it needs more memory to run the installer (it has 2Gb; the 20.04 machine mentioned above has 4Gb). Using the .zip download fixed that issue (hence that recommendation above). Other relatively low-spec machines are also fine; the Ubuntu 20.04 machine was a fairly average spec when I bought it over 10 years ago.

Occasionally, I (like other people I’m sure) have noticed that things “disappear” from maps they should appear on. In the case of “national parks” and “areas of outstanding natural beauty” in the UK, they’re mapped as multipolygons. This might be simple, like Nidderdale or more complicated, like Ynys Môn. What usually happens is that someone accidentally “breaks” the multipolygon, for example by creating a gap in an outer edge, so that rendering software sees it as invalid and ignores it.

What I’ve typically done in the past is, whenever I’ve noticed a problem tried to find the problem using something like the OSM Relation Analyzer or JOSM’s validator. The “area” view at Geofabrik’s OSM Inspector is also useful.

That works, but “happening to notice a problem” as a first step is not ideal. This map does show national parks / AONBs and not a lot else at zoom 7, but those tiles only typically get updated every 4 days, so it can lag behind problems.

How to find out which ones are missing?

I have a rendering database that covers GB and Ireland, and in there “SELECT distinct name FROM planet_osm_polygon WHERE (boundary = ‘national_park’);” will list all those items that are currently being rendered. As an aside, that query is on “boundary=national_park” because the data is processed by this lua style file as it is loaded, and that consolidates the tagging into something that makes the rendering code less complicated.

Next, the challenge is to get the same list from OSM. Thanks to this answer on the help site, I was able to do that. There’s a bit of manual work comparing the two, because:

• My database has some “overseas” national parks in it in error, where the “only update the rendering database with local data” test hasn’t worked.
• My database has some different names in some cases - I set the name field for the Pembrokeshire Coast National Park to name:cy (“Parc Cenedlaethol Arfordir Penfro”) rather than name (“Parc Cenedlaethol Arfordir Penfro / Pembrokeshire Coast National Park”) based on its location in Wales.
• There’s at least one “extra” AONB in OSM (Dunstable Downs, which is actually part of Chiltern Hills AONB).

Once I’ve done that, I have two lists that I can compare.

Which ones were missing

Chiltern Hills AONB
Dorset AONB
High Weald AONB
Kent Downs AONB
Llŷn AHNE
Shropshire Hills AONB
South Devon AONB
Sperrins AONB
Suffolk Coast & Heaths AONB
Ynys Mon AHNE

Fixing the missing ones

The process is the same manual process as before - download the relation as an object in JOSM, validate, and look for errors in the multipolygon. In some cases (for example “unconnected coastline”) you might need to load a bit more data to avoid seeing unrelated errors. Generally speaking you can ignore the warnings, as most of these are unrelated to multipolygon validity and some are completely invalid.

Usually what’s happened is that someone’s redrawn some feature that formed part of an AONB boundary and neglected to add the new part back into the AONB relation. To take Dorset AONB as an example, it looks like previously one of the railway lines formed the boundary, but that had been edited to leave a gap, so I redrew that as a new way slightly to the south in these changesets.

Shropshire Hills AONB was the most difficult to diagnose and fix. JOSM didn’t actually find an error there so I had to go through the warnings. It may have been this change that finally fixed it (zoom in on the green line east of Craven Arms to see the red line and deleted nodes).

Detecting future problems

Next - what’s the easiest way to detect breakages as they occur? I don’t want to have to manually compare with Overpass every time, but the good news is that I don’t have to - I can just compare with the list that was in the database yesterday, and look for things disappearing from the list.

# ------------------------------------------------------------------------------
# Count national parks, and see if there are more or fewer that when we last
# looked.
# ------------------------------------------------------------------------------
mv /home/renderaccount/data/nationalparks.justnow /home/renderaccount/data/nationalparks.previously
psql -d gis -c "SELECT distinct name FROM planet_osm_polygon WHERE (boundary = 'national_park');" > /home/renderaccount/data/nationalparks.justnow
diff /home/renderaccount/data/nationalparks.justnow /home/renderaccount/data/nationalparks.previously
#

That now runs as a cron job on map.atownsend.org.uk every morning, and if there are any changes, I’ll get emailed the results.

I’ve added a new page to the switch2osm guide: “Updating your database as people edit OpenStreetMap”.

It’s a bit overdue (it’s based on something I wrote about 4 years ago, and the software it uses predates that by a long way), but I think it is worth making available on the main site.

I’ve tested it both on Debian 11 and on Ubuntu 20.04 LTS, and have tried to make clear what you need to do differently in each case.

For completeness see also Ircama’s excellent guide which is well worth reading as it actually provides a lot more detail about the process, and also suggests how “osmium” may be used instead of “osmosis” to process OSM files.

I’ve added a new page to the switch2osm guide: “Manually building a tile server (Debian 11)”. The Debian developers have worked to include the required software inside Debian 11, so there’s actually less setup work needed than in previous versions.

Debian 11 has not yet been released; it’s still “testing”, so there may be minor changes along the path to release. Also, there’s currently a temporary problem with the Natural Earth download site that requires a workaround, but that’s documented in the guide. When that is resolved I’ll update the notes.

I’ve not yet linked the new page into the menus because I want to give people a chance to test it and identify any problems first. Once that’s happened, I’ll add it to the “Serving Tiles” menu.

Unfortunately, in England at the moment a number of pubs aren’t able to reopen because (especially locally to me) they’re too small and have lots of small rooms or they’re music venues for which “social distancing” somewhat defeats the object of the event in the first place.

I’ve chosen to show these on this map like this:

This is deliberately different from the rendering of “normal” closed pubs:

the idea being that hopefully the “closed due to covid” ones will be able to find a way of working sooner rather than later.

I’ve done this by looking at the “opening_hours:covid19” tag - that seems to be the most used for this both globally and locally. To be able to handle these as a new object, they’re split out in the lua code used by the style and then added to the mss file that actually does the rendering.

I’ve recently added a new feature to https://map.atownsend.org.uk , prompted by the dreaded lurgi that is affecting everyone at the moment - the ability to show pubs with either outdoor seating or a beer garden.

Outside seating is shown like this:

with a black line at the bottom right of the icon.

Here’s a beer garden:

with a green underline instead of black

For the rest of the map legend see here.

I’ve added a couple of short but interesting scripts to github. They fetch and sort posts from an OSM mailing list from some time in the past. For example, here are the topics on the talk list at the height of the licence change furore:

listtopics.sh talk 2012 May

 35  OSM : It's a shame !!!
 24  Edit review: ele=0
 22  Worst of OSM
 21  TomTom is thumping us
 17  Import of buildings in Chicago
 17  Edit review: intermittent waters
 16  OSM cycle map - ?excessive focus on long-distance routes
 15  Edit review: "building"="levels=N"
 14  Cycleways and Access tags: Left, Right, Forward, Backward?
 10  OpenSeaMap
 10  Cycle lanes & cycle tracks - my findings and a proposal
  7  Changing capitalization (Lima)
  6  OWL down
  6  OSM data density - top regions
  6  Old versions of OSM?
  6  Copy-and-paste remapping
  5  TTTracklog and TomTom core 9.4
  4  OSMCoastline / OpenStreetMapData.com
  4  Mapnik rendering issue with *_link roads
  4  (dis)Honesty and Copyright

and here’s the tagging list today:

listtopics.sh tagging 2020 May

106  Reviving the path discussion - the increasing importance of trails in OSM
 75  Tag:amenity=motorcycle_taxi not approved
 58  Remove non-prefixed versions of 'contact:' scheme
 49  RFC ele:regional
 42  relations & paths
 35  Doorzone bicycle lanes
 34  Permanent ID/URI --- off topic email
 30  Feature Proposal - RFC - Recreational route relation roles
 26  Change of wiki page Key:access
 25  Adding values healthcare=dispensary and healthcare=community_care?
 24  track vs footway, cycleway, bridleway or path
 23  Is there any tagging scheme for carillons already?
 15  Meaning of "administrative" in boundary=administrative, in your country?
 14  With leisure=common deprecated, Senegal & Mali need a replacement
 14  [OSM-talk] Should we map things that do not exist?
 14  oneway=yes on motorways
 11  Section numbers in hiking routes
 11  Running but no hiking/walking
  9  Quality and the Openstreetmap value chain
  8  Fwd: Section numbers in hiking routes

looking at the names, some of these from 2007 are still familiar:

listpeople.sh talk 2007 May

 79 Frederik Ramm
 59 David Earl
 46 Nick Whitelegg
 33 Richard Fairhurst
 31 Christoph Eckert
 27 Jon Burgess
 26 Andy Robinson
 25 Martijn van Oosterhout
 21 Lester Caine
 18 SteveC
 18 Robert (Jamie) Munro
 17 Nick Black
 16 Joerg Ostertag (OSM Munich/Germany)
 15 Steve Chilton
 14 Francisco R. Santos
 14 David Groom
 14 Barnett, Phillip
 13 Hakan Tandogan
 12 Robert T Wyatt
 12 Matthias Julius