OpenStreetMap

Extracting Data from OpenStreetMap History Files Using the *New* Osmium

Posted by dalek2point3 on 24 May 2014 in English (English)

TLDR; Version

I wrote a small script to get some data out of OpenStreetMap history files using Osmium. You can find it here: https://github.com/dalek2point3/osmium-tools/

The Story So Far ...

I'm working on a project to analyze OSM contribution history in the United States. One way to do this is to use the changeset dumps -- the changeset dumps contain fields for the username of the contributor, the time of the contribution and the bounding-box of all the edits made in a particular changeset. Using this data, and approximating the location of the edit as the center of the bounding box it is possible to do a lot of analysis about how contribution activity has changed and evolved in different regions around the country. My previous diary entry is one example of such an analysis.

History Files Here We Come!

However the approach of analyzing changesets comes quickly to a dead end if you want to understand the type of contributions. What are people actually doing when they edit in a certain area? Are they adding new subdivisions (likely to be lots of ways and "highway" tags here), are they adding useful metadata to existing streets (lots of maxspeed and oneway tags here) or are they adding POIs (amenity tags) and natural features like water bodies, hiking trails etc?

The changeset files are absolutely useless to understand this kind of activity. If you want to do such an analysis you have to look to the history files. History files are exactly like planet XML files, but with every past version of a particular node, way or relation also recorded. Like the planet XML, each feature comes with a changeset id, so you can reconstruct changesets from this file, and then look into what is actuall going on in the data.

Great! So how do we do this exactly?

The one problem with using history files is that a number of traditional tools do not work with this file format. Osmfilter which is great for filtering certain feature types does not work with .osh files and neither does Osmosis -- both tools I was using extensively in my previous work.

One way to go is to use Osmconvert to convert the entire .osh file to csv, and then manipulate this file using standard command line tools like sed. Unfortunately I this approach scales poorly -- I was using the history file for all of the US, and the csv version of this file can get pretty large, pretty quickly.

So, how do we get there? Enter Osmium! Osmium is a wonderful tool written by Jochen Topf that provides C++ libraries and header files to work with OSM data. Yes, your read that right C++, finally it was time for me to leave my comfortable world of python and dynamic typing and try to remember the C++ back in engineering school a few years back!

Fortunatately, osmium ships with some nice examples and very helpful community members that makes it not terribly hard to pick up! What's more -- Osmium has recently be redesigned and now comes with a nice shiny new website and some nice documentation.

Exact Steps to Osmium Glory!

So using all those as helpers, this is how I proceeded:

  1. First, thanks to wonderful work by MaZderMind there are a number of "history extracts" available so that you dont have to begin with the planet history file. I downloaded the north america extract to start with

  2. Then using MaZderMind's wonderful OSM history splitter (which is based on Osmium) and a bounding box for the continental USA ((-124.848974, 24.396308) - (-66.885444, 49.384358)) I created an extract for my analysis. Installing OSM history splitter is fairly straightforward if you follow the instructions on the github page.

  3. Then, in order to extract relevant elements from the history file I turned to Osmium. The original plan was to use OSM History importer which is another tool based on Osmium, to import all of the data into a PostGIS database and then run queries on this data -- but given the nature of my requirement, I thought this was overkill. Installing Osmium was fairly straightforward for me (although I've read that others have had trouble) -- I installed the debian packaged versions of all requirements listed here and then git clones the repo and I was set!

  4. And voila! Here is the script that extracts amenities from the history file and extracts highways and records the lat / lon for each (for the highways it records the lat /lon of the first node).

  5. I took me a while to understand how Osmium works, so I thought I would make a few notes to help others out.

  • First, you must remember that Osmium is a header-only library project. There are no executables that come with libosmium, but you should definitely play around with osmium-contrib -- for my particular requirement I found this program to be very helpful.

  • Second, osmium comes with a program called osmium-tool that can do a limited number of things. Your requirement might actually be satisfied by one of these pre-coded tools, so you should be all set! Look at the usage here

  • I order to understand my script, you need to understand a few things. Osmium can read a large number of osm-related file formats. So I create a reader object that reads the file I'm interested in. Then osm:apply interates through all the objects in the file (i.e. nodes, ways and relations) and for each object calls the "handlers". In my case, I have a "location handler" that reads in the locations of all the nodes and associates them with the way (In OSM, the nodes have coordinates, while ways only have node references) -- and once the location handler is called, I call the "names handler". The names handler calls the "node" function for all nodes and "way" function for all ways. In these functions I include logic for what I want to do with the data, in this case extract features that have the relevant tags and write them to stdout.

  • Here are links to some more documentation to help with your Osmium Project.

Osmium is an extremely powerful (and fast!) to do lots of amazing things with OSM data. In fact my favorite Taginfo used Osmium on the backend. I'd highly recommend it as a tool for any heavy duty history file processing!

Location: East Cambridge, Cambridge, Middlesex County, Massachusetts, 02142, United States of America

Comment from mvexel on 25 May 2014 at 22:06

Thanks for this write-up - Osmium and the tools that use it can be a little tough to get into, and articles like these help.

I have used osmjs, the Javascript v8 interface to Osmium, for a few projects that may be of interest to you - see this diary entry for a discussion.

Hide this comment

Comment from dalek2point3 on 26 May 2014 at 17:04

cool! great to see your article too. I'm going to keep plugging away at this and report progress here. I think we need a project that does "instant area history videos" in an interface like walking papers. Learning osmium seems to be the first step to creating such a tool.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment