OpenStreetMap
I’ve released a new version of tilemaker, the command-line utility that takes OpenStreetMap data in .osm.pbf format and makes vector tiles out of it.
It’s now between 45% and 85% faster - you’ll notice the difference particularly in places with complex multipolygon geometries. Memory usage is reduced, particularly in the polar regions. Plus it’s compatible with Geofabrik’s new CC0 “Shortbread” schema for vector tiles.
A handful of minor changes, mostly around changeset behaviour:
Plus a couple of other little fixes. You can download P3 at https://www.systemed.net/potlatch/download/ .
P3 continues to be offered with Mac and Windows builds plus an AIR file. I’m hopeful that there’ll be a Linux build in due course but that’s waiting on licensing decisions by Harman, who maintain AIR. The Mac build is fully notarized; the Windows build is self-signed so you need to click through a warning dialogue (I’ve experimented with using signtool to fix this but the whole process is so deeply painful that I gave up after many hours of trying).
As many of you will be aware, Flash Player is about to be disabled in web browsers.
I’m pleased to announce that Potlatch continues to be available, now as a desktop application. You can download it for Mac and Windows at
https://www.systemed.net/potlatch/
Potlatch 3 includes OAuth login, support for Remote Control, and other features to ease desktop editing. Other new features include support for Overpass API queries in the Tasks palette, and there’s some exciting stuff to come soon… but more of that in due course. As yet we haven’t found a way of reliably running it on Linux but investigations continue.
Potlatch is of course open source. You can get the source and report issues at https://github.com/systemed/potlatch3.
Thank you to the OpenStreetMap Foundation for supporting this work and to Potlatch’s many users for their edits.
You can now download a development version of Potlatch 3.0 to run on your desktop.
At https://www.systemed.net/potlatch/download/ , you can download a Mac native app, or an AIR file for use on Windows or (possibly) Linux.
Potlatch 3.0 includes desktop-editing features such as Remote Control support, overview zoom levels, and an OAuth login flow for saving. Rendering is faster and there’s now Overpass API support in the tasks palette.
Installation on Linux is likely to be complex and I’d welcome reports from anyone who has successfully achieved it.
There’s much more to come in future weeks so watch this space. You can report issues at http://github.com/systemed/potlatch2/issues pending the creation of a dedicated 3.0 repository.
So I figured I’d add it to cycle.travel’s elevation graphs. It was already in the turn-by-turn instructions, but this is still more fun.
mountain_pass=yes or natural=saddle as a node on the highway, with a name= tag too (obviously).
Potlatch gets fewer new features in a year than iD does in a day, but occasionally something creeps through.
This week/month/year, that’s the ability to add a function key shortcut (F1-F4) to a relation. Open the relation dropdown menu in the advanced relation panel, choose “Assign to F1” (or whatever), and from then on, you can add the current selection to that relation by pressing the key. It’s particularly useful when building route relations.
Function keys can, of course, also be used for memorising tag combinations and for switching between background layers.
Bringing third-party data into OpenStreetMap has always been a pain point. Simple bulk imports can bulldoze existing community work and cohesion. Editors such as JOSM or even Potlatch 2 have extensive merging functionality but a steeper learning curve and little ability to remap/rewrite data internally. Yet third-party data can often be valuable - especially if it was originally hand-collected to the same standards that OSM volunteers work to.
OSM Live Conflation is a new tool and workflow that builds on the approach taken by OSM Conflator, Cygnus, the JOSM Conflation plugin and the Potlatch 2 merging tool. It remaps data from vector tiles into OSM data, and lets you bring items across one-by-one quickly and efficiently.
A picture is worth a thousand words:
Or if you want to play with a live instance, here’s a volunteer-collected dataset of cycling infrastructure in West Oxfordshire. (Don’t upload unless you actually want to make an edit to OSM, of course!)
Last week there was a small change to the API which means some GPS traces are now returned unordered whereas previously they may have been ordered. This obviously has implications for editors’ GPS display. Potlatch 2 now copes with this by doing some basic proximity testing in an attempt to ‘reorder’ the unordered points, which restores the GPS layer to some degree of sanity.
At the same time, I took the opportunity to fix something that’s been bugging me for months. Historically, Potlatch 2 draws all the features crossing the current viewport, and then removes the sprites when they’re no longer visible. This is fine except when you pan across a 2000km power line, which takes ages to draw as a dotted line at z19, and bogs down Flash’s display performance for evermore. For such large objects, P2 now only draws the on-screen part, which makes it a whole bunch snappier.
Well, you know, it’s been a while since I wrote an OSM editor and I found myself missing the experience.[1]
We’ve pretty much reached the apotheosis of the traditional OSM editor workflow. There isn’t any point building another iD, because iD is terrific and does everything that needs to be done. There might be some merit in building a new native power editor for macOS, say, but even that would be a marginal gain and the sort of project best attempted by a company which employs dozens of mappers.
But we certainly haven’t reached the limit of OSM editing. The traditional workflow was never designed for integrating third-party datasets, for working with street-level imagery, for surveying on a smartphone, for QA. These activities were unheard of in 2006 when JOSM and Potlatch 1 were first conceived, but are big parts of today’s OSM data landscape.
So we should be experimenting with different editor experiences, to see if we can attract wider contributions through presenting an alternative workflow/UI.
osm2pgsql and OSRM can both make use of Lua scripting for tag processing, which in many cases is the best way to make sense of the often conflicting and confusing tag soup in OSM.
Firing up an osm2pgsql/OSRM run each time is, however, not the fastest way of debugging your Lua. So here’s a little script that reads your Lua osm2pgsql code, passes it the tags from a way, and returns the result:
# Test osm2pgsql Lua scripting with Ruby
require 'rufus-lua'
require 'overpass_api_ruby'
require 'pp'
# Initialise
overpass = OverpassAPI::QL.new
lua = Rufus::Lua::State.new
lua.eval('require "name_of_my_lua_script"')
# Get tags for the way
way_id = ARGV[0]
response = overpass.query("way(#{way_id}); (._; > ;);out;")
tags = response[:elements].find { |h| h[:type]=='way' }[:tags]
pp tags
# Convert keys from symbols to strings
lua['way'] = Hash[tags.map{|(k,v)| [k.to_s,v]}]
# Run Lua code
cmd = "res = {}; filter,tags,poly,roads=filter_tags_way(way,nil); return tags "
pp lua.eval(cmd).to_h
Call it with a way ID like this:
ruby test_lua.rb 35222450
When I do that on my osm2pgsql script, I get:
{:access=>"private", :highway=>"service", :source=>"OS_OpenData_StreetView"}
{"highway"=>"private_road",
"access"=>"no",
"z_order"=>0.0}
In other words, the input tags followed by the output tags.
It’s fairly easily adaptable for OSRM profiles, or for other functions in your osm2pgsql Lua script, or whatever.
I chose to write it in Ruby because there’s a ready-made Overpass gem and I’m generally a bit more comfortable in Ruby than Lua, but you could of course do the whole thing in Lua if you were so inclined.
I’ve just added support for a couple more tags to cycle.travel’s directions and thought it worth mentioning - everyone likes seeing their mapping being used.
First up, cycle.travel now includes ‘knooppunten’ (cycle node networks) in turn-by-turn directions. These are found in the Netherlands, Belgium and parts of Germany, and help you navigate dense cycle route networks. Here’s an example: https://cycle.travel/map/journey/51597
This picks up rcn_ref= or lcn_ref= tags on nodes.
cycle.travel also includes mountain passes in the turn-by-turn directions, for people who like riding somewhere hillier: https://cycle.travel/map/journey/51600
OpenStreetMap is navigable for bikes, on foot, and increasingly so for cars. But one thing we’re not yet great at is truck routing.
HGVs, lorries, trucks, whatever you call them, need to get from A to B without breaking either the road or themselves. Which means the map needs to know about height and width restrictions. 11foot8.com is a good example of what happens when truck drivers don’t have this information (and also can’t read):
OSM coverage is good in parts but patchy. Fortunately, the existence of open street-level imagery means it’s really easy to map this sort of thing from the comfort of your own armchair. Here’s a brief how-to.
The majority of important restrictions are height restrictions, and the great majority of height restrictions are railway bridges. (There are a few canal aqueducts too, though canal-related restrictions are generally weight restrictions on overbridges.)
A new version of Potlatch 2 with several improvements and bugfixes:
Potlatch 2 is somewhere between ‘active development’ and ‘maintenance mode’: there’s no massive new features that I’m planning, but I intend to keep making small improvements to it along these lines, plus extra features as and when I’m doing some mapping and figure out a way to make it easier or quicker to use. OSM is lucky to have such an excellent default editor in iD, which gives P2 the freedom to develop as an efficient and comfortable editor for those who like its way of doing things.
A new little feature for Potlatch 2: you can add a bookmark for your current location, accessible via a new ‘Bookmarks’ dropdown menu. Use the ‘Add’ button at the bottom of the menu to add a new one. It works pretty much like every bookmarks menu you’ve ever used, to be honest.
There’s also a new trademark obscure keypress, ‘N’, which moves to the other end of the currently selected way.
Everyone has their own reason for contributing to OpenStreetMap. Maybe you wanted a better map for your favourite activity - hiking, cycling, skiing. Maybe you were using a site or device with OSM data, and found it lacking. Maybe a friend got you involved. Maybe you believe in our aims as an open project. Or maybe you just thought it was cool.
What was your reason?
I’m giving a talk at State of the Map US in just over a week, in which I’d like to share people’s stories as to why they contribute. I’d love to hear yours.
You can post in the comments here, drop me a line at richard@systemeD.net, send me an OSM message, or reply on Twitter. (Anonymity offered if you want!)
Folks, I’ve made an exciting discovery. I’ve stumbled across a country on the map which has 46 million inhabitants but is barely mapped. There appears to have been an import several years ago of a poor-quality dataset, and since then it’s languished untouched. There’s no indigenous mapping community. Can we help this poor beleaguered country to get a decent map?
Ok, you may have figured where I’m talking about. It’s the rural US: 72% of the landmass, 15% of the population. And it needs your help.
For the past couple of months I’ve been using idle moments to address a particular, and very widespread issue in OSM’s coverage of the rural US - the highway=residential problem.
TIGER data, which forms the bedrock of OSM data in the US, classes roads by CFCC - ‘Census Feature Class Code’. By far the most prevalent is A41 - “Local, neighborhood, and rural road, city street, unseparated” - and the TIGER import translated this to highway=residential. This was a good fit in urban areas but covers a multitude of sins in rural areas - everything from good, fast state highways to rutted forest tracks or worse.
The effect is that our map of the rural US shows pretty much everything, save the biggest roads, as a residential road. Tarmac road with sweeping curves and a painted centreline? highway=residential. Gravel road? highway=residential. Forest track? highway=residential. Vague two-foot clearing through the woods where someone perhaps rode 50 years ago? highway=residential. Ploughed field? Etc, etc.
Task-based fixup workflows, such as MapRoulette, To Fix, and the HOT Tasking Manager, have become popular in OSM in recent years, presenting a series of map-improvement tasks for a user to complete.
MapRoulette is wondrous in many ways but has three issues. First, if you’re using an online editor, the time taken to open each task is significant - around 3-5s to open a new instance of iD for each task. Second, that online editor has to be iD, so it’s not suited for those of us who use P2. Third, the process for creating your own tasks is fairly cumbersome, especially if you’re working on a small series of tasks to be completed by one or a few people.
Potlatch 2 now implements a fast, light way of moving through a series of tasks. It’s intended for small-scale personal work rather than planet-sized challenges. There’s no ‘resolve’ button - it’s just a way of moving between locations efficiently.
It works like this:
If it’s a GPX file, it should contain waypoints. Each one can have a description in a <desc>, <cmt>, or <sym> element.
If it’s a GeoJSON file, it should contain Point features. Each one can have a description in the ‘name’ property (or, failing that, the first property).
I’ve created a set of tiles from US Forest Service road data for the 155 US National Forests.
This is to help with TIGER fixup in these rural areas, where tracks, trails and entirely non-existent paths are often tagged with a bare “highway=residential”. The US Forest Service data is greatly superior to the original TIGER data and has metadata on surface type/quality, but is unsuitable for automatic import into OSM because it would overwrite mappers’ existing work in these areas.
You can access the tiles at:
and they’re included in the editor-imagery-index list used by P2, iD and Vespucci. The tiles are available up to z19. Use of Potlatch 2’s new floating imagery window mode is recommended, so that you can work from both Bing imagery and these tiles at the same time.
I know. Two new features in the space of a week. Don’t get too used to it.
P2 now has a ‘Show floating window’ checkbox in the Background menu. Select this, and you’ll get a second set of imagery in a floating window:
The window follows your main cursor location. So you can have Bing in the main window and Ordnance Survey StreetView in the floating window, or a blank background in the main window and Bing in the floating window, or whatever you like.
There’s a “lock zoom” checkbox to stop the floating window zooming in or out - for example, edit at z18 with Bing but see OSSV at z16; or edit at z14 with Bing but have the floating window at z19 for a close-up view. It also respects the “max_zoom” parameter of the editor imagery index.
I coded most of this at a hack weekend a year ago but had never got round to finishing it off. Hope it’s useful.
And so is Potlatch… just about. (Insert obvious Flash reference here.)
Potlatch 2’s core purposes are (a) being an intermediate-level editor and (b) annoying Germans. It’s never going to get major new functionality or important rewrites, but for those of us happily using it, I’ve always intended to add and fix little things over time.
In the summer it gained support for the editor-imagery-index project, which provides a central list of background imagery suitable for tracing. Today it has a couple of improvements to the long-standing feature where you shift-click away from a way to add a new point (roughly comparable to JOSM’s “Improve Way Accuracy” mode, but we don’t do modes round here) - which is a really useful technique for TIGER fixup.
