A new way of fast browsing of latest changes

Posted by lxbarth on 2 May 2013 in English (English)

Finding out fast who's modified the map is hugely valuable to review changes in areas you care about, to connect to new mappers or to just show how fresh the map is.

Unfortunately, the part of that's supposed to provide this functionality - the history tab - is functionally broken. I'd like to suggest here a straightforward way to fix it, punting on some of the hard engineering problems that fast browsing of historic changes bring with them.


To recap if you're new to the issue, here's why the history tab doesn't work today: virtually anywhere in the world you'd like to see the latest changes of a particular area on OpenStreetMap, what you'll actually get is large-scale changes whose bounding box happens to intersect with with the area on the map you're viewing while not actually impacting any data in the area you're viewing.

This is a well known problem in OpenStreetMap and people like Matt Amos, Ian Dees and more recently PaweĊ‚ Paprota have worked on coming up with a solution for this issue.

The underlying engineering problem is Hard: changes to OpenStreetMap are organized in changesets, each one of which can contain up to 50,000 edits and whose modifications can be geographically huge. Querying all changesets that actually modified data in an arbitrary bounding box of the world and displaying them in reverse chronological order is computationally expensive while at the same time it should happen in milliseconds to satisfy a web request and allow for fast browsing.

Fixing the history tab

At the Chicago hack weekend Tom and I created a prototype that completely punts on the expensive problem of fast browsing for the entire changeset history. The approach we've taken is essentially to present you with a map and a list of the latest changes on visible elements first, then only reveal the history of an element when you click on it.

Conceptually, this is very straightforward and supported by existing APIs, computationally this is dirt cheap. This is not actually a novel approach in OpenStreetMap (most editors do something like that) but it is viable as an alternative to today's history tab.

The prototype doesn't do any data processing itself and is actually just a simple HTTP and JS application hosted on Github pages. It uses the OpenStreetMap API's map call. The latter means it is querying OpenStreetMap in an unefficient way, but this is a comparatively simple problem to fix. It could just as well query a very simple tiled data source.

Check out the result for yourself. I think it is already a very useful browser for exploring changes on OpenStreetMap. With few iterations this could be much faster and a viable fix to OpenStreetMap's history tab.

Related conversation on [dev] list can be found here:

Comment from mikelmaron on 2 May 2013 at 17:41

Nice! Seems obvious in retrospect.

Issue is that it only works efficiently at high zoom level, and that also varies depending on data density ... DC is having trouble loading at same zoom levels at Witchita. How do you design reasonable fall throughs to traditional history?

Comment from seav on 2 May 2013 at 17:45

I think another vital tool is a way to easily analyze a single changeset. You should be able to see what objects were modified, created, and deleted. What the tags are on those objects and what modifications were done on those tags. How the shapes of objects changed. If ways were reversed, split, or joined.

The OSM History Viewer goes halfway there but I think it could be much much more.

Comment from lxbarth on 2 May 2013 at 18:12

Mikel -

Issue is that it only works efficiently at high zoom level

we could get to lower zoom levels by speeding up the API endpoints - i. e. getting tiled data. Not sure what'd be involved there in terms of setting up infrastructure or what existing APIs (overpass?) we could use here but it's a solved problem from an architecture perspective.

Comment from lxbarth on 2 May 2013 at 18:16

seav -

I think another vital tool is a way to easily analyze a single changeset.

Interesting... didn't know of OSM History Viewer. I see it's slow computing some changesets, how could it be fast? Just preprocess them all?

Comment from tyr_asd on 2 May 2013 at 22:28

Awesome, I really like it!

I hacked it to only shows changes of last week. Not showing older items one may not be interested in helps saving bandwith and rendering time and/or allows lower zoom levels. Wouldn't it be a good idea to add a dropdown/slider where one could select the time-frame (24h, 1 week, 1 month, etc.) one is interested in?

Comment from Tom Chance on 3 May 2013 at 10:28

One UI suggestion based on something I really like in the OWL tool... highlight the objects on the map when you hover over a changeset in the list.

It's nice to see different attempts to improve the history tab, this one shows promise but the OWL tool is far more developed with nice features like a built-in summary of what has been changed.

Login to leave a comment