OpenStreetMap logo OpenStreetMap

tyr_asd's Diary

Recent diary entries

iD development update n2

Posted by tyr_asd on 27 November 2024 in English.

My Talk at State of the Map 2024

(How the iD Editor still can to do better in the long term)

This September, I presented some ideas for the future of OSM’s iD editor at the State of the Map conference in Nairobi. Now, the video recording of the talk is online. To summarize the talk’s content: I identified two core challenges faced by the iD editor which need to be addressed in the upcoming years:

1. Data Complexity

Naturally, OSM’s data got more complex over time – as an example, a simple school might have typically been mapped as a single POI node in OSM a couple of years ago, while nowadays the state of the art is to model it using multiple map features that are nested within each other. While this allows to represent the reality in a high level of detail, the added complexity needs to be accommodated for by the tooling, especially in an editor tailored for beginner mappers like iD. For a deep dive into this topic, see also Sarah Hoffmann’s talk at the same conference.

schematic mapping of a "school" consisting of different features representing objects like the school ground, school building, playgrounds, access paths, entrances, etc.

2. Map Maintenance

Another important point for the development of iD is that at least in some regions of the world, the mapping is moving away from mapping objects for the very first time on a blank map canvas. Instead, it becomes more any more important to efficiently keep the map up to date and perform map maintenance tasks. This insight is by the way definitely not new, as it was already discussed at State of the Map US 2013 by Alan McConchie. One of the proposed ways to address this topic could be to more prominently support the last check date tag in the editor.

mockup of a user interface element in the iD editor, which prominently displays the last check date of a map feature

This is all still at an early stage, and a lot is still to be figured out. So, any form of feedback is very welcome. Please get involved.

Object lifecycle mapping

One of the above mentioned ideas to make iD fit for the mapping tasks of the future is to make it more intuitive to change the lifecycle status of an object: For example when marking a feature that was under construction as completed, or when tagging a POI as temporarily or permanently closed. Mattia, who has already completed a Google Summer of Code project around the iD editor, has recently started to work on this as part of a university training course for his master studies in computer science.

In a preliminary analysis, we have already identified that the current state of which tagging schemas are used to map the lifecycle statuses of different kinds of map features is not very consistent in all cases. This means that we will definitely spend some time to figure out a good way to make the implementation as compliant as possible with how these statuses are currently mapped: As a first step, there will be a discussion thread in the community forum to discuss the involved details. Stay tuned to hear more about this from Mattia and me in the near future!

PS: If you are also interested to tackle a particular feature of the iD editor, for example in form of a university thesis, don’t hesitate to contact us!

Upcoming

  • An update of iD in form of version 2.31 is coming soon. The most notable improvements of this release include a bugfix for the flickering while switching background imagery layers, as well a reworked unit test setup which hugely improves the developer experience: Instead of waiting around for minutes to see if your changes comply with iD’s test suite, you now typically get the results within seconds. Thanks Kyle for this very valuable contribution!
  • The next online community chats are happening at the following dates, feel free to join if you like!

overpass turbo just got a new Auto Styler

Posted by tyr_asd on 3 November 2024 in English.

Attending the recent hack weekend in Berlin, I got inspired to work on a neat feature in overpass turbo. Honestly speaking, in the past few years, the tool did not the amount of attention from the developer’s (my) side that it deserves. What came clear to me at the hacking event is that a lot of people still use it on a very regular basis and that it is an invaluable part of the OSM bag of tools, which was really nice to hear.

Auto Styler (beta)

So, what someone proposed was to extend the current query wizard feature to also support the styling of the results using mapcss. What I took from that suggestion was that it should not be hard to implement a dedicated helper dialog that auto-generates MapCSS to style the data according to the values of a selectable tag and color palette. And I’m glad to say that after a quite productive two days, there’s now already a beta version already available to test out on overpass-turbo.eu.

Screenshot of the dialog of the new auto styler dialog in overpass turbo.

How it works is that after you’ve written an overpass query (or used the query wizard to write one for you), and have run the query, a “Style” button appears on the menu-bar at the top next to the wizard button. In the dialog that opens, you simply choose a tag and a color palette. The result is a set of auto-generated MapCSS and a potentially pretty colorful map:

Screenshot of the result of the new auto styler dialog in overpass turbo.

By the way, this new feature made it possible to close one of the oldest open issues on overpass turbo’s issue tracker (#30), which has been lurking around since 2013! Feels good to finally be able to close that one.

Finally, please let me know if you find this new auto styler useful and about any potential further improvement ideas. :)

More MapCSS features

In the meantime, overpass turbo also got a few more useful features around MapCSS in general:

  • You can now use the * selector to apply a style to all OSM features that match a given tag (this is a shorthand for repeating the selector for node, way and relation respectively)
  • The displaying of “small” line or area features as collapsed points (when they would otherwise have very tiny dimensions at the current zoom level) is now customizable via a custom new mapcss statement render which accepts one of three values: auto is the previous behavior that automatically collapses features to point markers (if not disabled by the user in the settings) should they otherwise have tiny dimensions on the current zoom level, native renders map features always in their native geometry type (i.e. they are never collapsed to points), and point renders all features as points at their centroid position (very similar to using out center; in the overpass QL query directly).
  • overpass turbo now understand a whole bunch more MapCSS label properties: head over to the changelog to find more about this. PS: Thanks Simon for working on this and all other great improvements in and around overpass turbo.

Screenshot of overpass turbo specific MapCSS statement to change the rendering of small features

  • Left: (default) auto mode collapses small sections to point markers
  • Right: native mode which renders all map features in their original geometry type. {{style: line { color:red; } }}
  • Try it yourself!

iD development update n1

Posted by tyr_asd on 21 August 2024 in English.

It’s been a while since I last wrote about iD’s development and thought it would be a nice idea to make this a more regular thing in addition to the regular online community chats.

Recently, there have been some new features implemented that are worth showcasing, including several great ways that community members are stepping up to contribute to iD.

iD v2.30 release

screenshot of new Panoramax layer in iD

In his Google Summer of Code project, Mattia Pezzotti has implemented support for street level imagery from Panoramax. Panoramax is a relatively new completely free and open-source alternative to other street level imagery providers such as Mapillary, KartaView or Bing Streetside. It even includes features like a federated system architecture and the possibility to self-host the server software if you need to. Check the project out on panoramax.fr to read more and how to contribute your own pictures.

You can read more about the Google Summer of Code project’s progress in Mattia’s personal blog. It’s been excellent to mentor Mattia through his project, and I think he is well set up to make more contributions in the future.

Other than that, the latest release also includes the ability to directly start iD with a note selected (e.g. #id=note/1234), and a bunch of bug fixes. Head over to the full changelog to read more about the details.

id-tagging-schema: new contributing guidelines

The tagging schema has also seen a lot of updates (see latest changelog if you’re interested), but the biggest change there is that we now have new documentation for how the tagging schema is maintained. The guidelines explain how the repository is managed from a content point of view; for example about what requirements a tag must fulfil to be included as a preset. The second new document is about how to contribute and support the project. We already have two volunteers who help in the co-maintainer role: Tobias and Kyle. There is still space for a few more people helping out. If you’re interested to helping moderating issues or being a co-maintainer of the repository, please contact me.

Coming Up

The next times to meet are the community chat on September 3, 15:00 UTC online, as well as at the State of the Map where I’m going to hold a talk about the bigger picture of the future of iD. There will also be the possibility to chat about these ideas in a bit more detail in a BoF session at the conference, as well as in one of the upcoming online community chats.

Another thing I’m planning of starting to work on in the coming weeks is the long anticipated building mapping mode in iD, with the goal to make this common mapping task more efficient and enjoyable.

My First Year with iD

Posted by tyr_asd on 14 February 2023 in English.

About a year ago, I started working as the maintainer of the iD editor for the OSMF. Here’s a short recap of the last year or so and a bit of outlook of what’s lying ahead according to me.

A look back

The year was characterized by the reactivation of processes which had been dormant for a little while before I started:

The tagging schema repository has received regular releases (about one major update every second month on average) after been mostly dormant in the year before. Many of the improvements to the tagging schema came in by various contributors from the community: Thanks to everyone who took part in this, including everyone actively translating on Transifex!

For iD itself, I soon learned that it is better to perform small iterative improvements, rather than tackling big reworks all at once. Therefore, my focus in the last year was primarily on stability, bug fixes and incremental improvements. There were still a few improvements to point out from last year:

  • version 2.21 switches the development build system to a more modern system based on esbuild and droped support for Internet Explorer 11
  • version 2.22 contains a first step towards making iD understand lifecycle prefixes of tags
  • version 2.23 significantly improves startup times, improved input fields in various ways and introduced a few improvements under the hood to manage presets
  • version 2.24 adds a remaining input length indicator for fields which are constrained to OSM tags’ maximum length of 255 characters and introduces a new field type for directional tags

A glimpse ahead

Another highlights of the year 2022 was the State of the Map conference in Firenze, where I spoke about the history of the iD editor (in summer 2023, it will celebrate 10 years of being the default editor on osm.org!), and tried to explain what the mid to long term high level goals are which I would like to tackle for iD. I identified five big topics to work on:

  • iD is the first mapping experience for many contributors: it should remain friendly and easy to use a nd accessibility is an important topic where iD can still be improved
  • as OSM matures, it will become more important to streamline mappers’ editing experience to their individual needs: away from a single general approach which has to work for everyone, towards a system which allows for more customization
  • as the amount of data in OSM grows continuously, performance bottlenecks become apparent: iD should be as responsive as technically feasible
  • mapping should be a delightful experience: recurring mapping tasks should not feel cumbersome to perform in iD
  • when a region gets more and more “completely” mapped in OSM, the topic of map gardening becomes important: iD should make it easy to keep the map up to date

A next step: community gatherings

Starting this year, I would like to pick up the “tradition” of hosting regular online community sessions to chat about the iD editor. I’d like to do these on a regular interval, starting about every month and at varying times of the day.

For each of these meetings, I will pick a topic which I’d like to highlight in form of a short tutorial, hands-on session and/or presentation. My goal for this section is to share tips and tricks with you as well as hopefully trigger interesting discussions. Of course these meetups will also be a platform to ask general questions, and to give updates about current works in progress. Any other feedback is of course also very welcome!

Here are some examples of topics which could be the focus of these sessions: * reorganizing the background imagery layer widget * adding a “improve geometry” mode to iD * how to make ui fields better: iconography, grouping of … * iD’s map rendering technology stack * how localization is done in iD * lifecycle prefixes in OSM tags

Are there any further topics you are eager to discuss?

I’ve scheduled the first of these chats on February 20 at 4pm UTC. It will take place on the OSMF’s BigBlueButton server in the room https://osmvideo.cloud68.co/user/mar-ljx-3qx-58s and the first topic will be: the state of the tagging presets: recent changes & development tips.

See you soon!
Martin

PS: To everyone who has recently opened a PR or issue on the iD repository and is still waiting for my feedback: I have not forgotten about you and will do my best to get back to you as soon as possible. In any case, thank you very much for your efforts and patience!

OSM community – you're great!

Posted by tyr_asd on 11 November 2018 in English.

Normally, I blog here more about technical topics, but today I would like to take the opportunity to honor three small parts of the OSM community, who I happened to have the chance to get to know over the years that I’ve been active in mapping. All of these groups deserve a shout-out for their great work they do on OpenStreetMap month after month.

  • The OSM community in Graz (Austria) managed to map their city in great detail and keeps it complete and up to date. It was a pleasure to be a part of the monthly meetups back when I studied there. I remember many fruitful discussions over good food an a nice beer or two. Keep up the good work, guys!
  • Geofabrik is a company from Karlsruhe (Germany) which provides OSM related services since many years, some of which are available for free to the OSM community. Thank you for maintaining your OSM data extract download service, and for hosting your iconic hack weekends. You’re awesome!
  • The disastermappers heidelberg are a group of students of Heidelberg University who regularly organize mapathons and workshops, and try to raise awareness of the benefits OpenStreetMap data towards a wide audience. Don’t stop educating the mappers of tomorrow!

Do you also know groups or individuals that do great work in OSM? Let them know! :)

Latest Changes

Posted by tyr_asd on 18 February 2018 in English.

Want to get an idea about what contributions are happening in OSM in your region? Maybe you’re even looking for a way to better review map changes (hint, hint)? A good starting point is probably my latest-edits page tool. It displays all objects that have been modified during the last week alongside with their respective changesets.

One major drawbacks of the tool was that until recently, deleted map objects (and changesets consisting solely of deletions) were not displayed on the page. Now, deleted objects are displayed as faint “ghostly” outlines on the map. In the same way, you can now also see how modified objects looked like before the respective contributions happened. This can result in a nice looking “shadow” effect when for example buildings have been realigned to better aerial imagery.

Also, you can now select between the latest changes from the last day, week, or month, and directly zoom to the location of a particular changeset:

screenshot

The tool still doesn’t show intermediate states of objects that have been modified more than once in the selected time period, as well as modifications to relation objects.

Programmatically, this new version is implemented by using an Overpass augmented diff query (switching over from a “regular” OverpassQL query using the newer statement) and some massaging of the returned data to get them properly in shape (it sorts the data into two buckets – one for the state of the data before the respective edits and one for the current state – then uses osmtogeojson to generate GeoJSON which can be put on the map).

10 years of OSM data history

Posted by tyr_asd on 6 October 2017 in English. Last updated on 9 October 2017.

Tomorrow is the 10 year anniversary of OSM’s API version 0.5. This is the version of the OSM-API that first exposed (among other things) the version number on all OSM objects, making it possible to access the full history of every object modification from this point onward.

This means that very soon, the full history planet file published on planet.osm.org will contain more than 10 years of editing history which can be investigated, evaluated and analyzed (using tools like the OSM history database oshdb that’s currently under development at HeiGIT on the University of Heidelberg, which I presented earlier this year at the State of the Map).

Of course, OpenStreetMap as a project exists for a bit longer than that (about 13 years now) and there was already quite some data mapped before the OSM API 0.5 was introduced 10 years ago.

Here’s an interesting side note: There already existed a history call in OSM’s API 0.4 (and apparently even in 0.3, see the comments below), but unfortunately this historic data apparently hasn’t been preserved in the newer versions of the OSM API, meaning that it is also not available in the (relatively) easy to use full-history planet dumps which are available nowadays. As far as I can tell, this “prehistoric” OSM data has basically been lost (though some of it might be reconstructible by analyzing the list of very old planet dumps on planet.osm.org). Does someone of you perhaps know what exactly happened to that data back then?

(Cake made by Liz LeBreuilly, picture uploaded by Blackadder, CC-BY-SA-2.0.)

Location: Neuenheimer Feld, Neuenheim, Heidelberg, Baden-Württemberg, 69120, Germany

OSM Node Density – 2017

Posted by tyr_asd on 2 July 2017 in English.

The latest installation of my yearly osm node density visualization is now online: https://tyrasd.github.io/osm-node-density shows the freshest data from mid 2017 (while the results from previous years starting with 2014 are also available on the site’s layer selection menu).

Can you guess where the following example screenshot was taken?

Click here to find out.

Visualizing OSM.org's Map Views

Posted by tyr_asd on 4 September 2016 in English. Last updated on 19 September 2016.

OpenStreetMap’s standard map layer is used by many people each day. OSM even provides a dump to the access logs from which one can see which parts of the world is viewed how many times for each day. Inspired by Lukas Martinelli’s work on Parsing and Visualizing OSM Access Logs, I’ve worked on a web app to visualize OSM’s tile access logs. Contrary on Lukas’ approach, I wanted to focus on an accurate representation of the data and wanted to make something works for the whole (Mercator-projected) world.

I’ve ended up with a viewer that takes an (uncompressed) log file from planet.osm.org and produces a two-dimensional histogram for each zoom level: For example, at zoom level 6 in the viewer each pixel on the viewer represents the number of accesses of the corresponding osm.org-tile at zoom level 14. That’s 8 zoom levels further in – or, put another way, each 256x256px² osm.org-tile is represented by a single pixel in the visualization.

The number of accesses of each tile is represented by a logarithmic color palette:

screenshot osm-tile-access-log-viewer

You can play with the tool at http://osm-tile-access-log-viewer.raifer.tech, the source code is found on github and openly licenced (MIT), of course.

With this one can for example compare map views before and after a specific event, for example the recent earthquake in central Italy:

map of map views before and after the recent earthquake in central Italy

But one can also see some interesting artifacts in the data, for example the large amount of tile views around null island or those (to me inexplicable) “comet tail” shaped patterns at some Russian cities. Do you have an idea where these artifacts stem from?

Making-of

warning: the rest of this post will be a bit more technical

What bugged me a bit at first was that my initial implementation was quite slow and made the webapp unresponsive. On my machine the first version took about 40 seconds for the initial processing step (between dropping the log-file onto the page and the first displayed results), which is quite a lot! Meanwhile those calculations were blocking the main UI thread and even causing this nasty browser-popup to appear:

page unresponsive warning

So, what can we do about that? As always, optimizing this kind of stuff starts with some profiling and goes through multiple iterations of optimizing and refactoring with more profiling in between. In the end, I managed to cut the time down from 40 seconds to a mere 9 seconds in the current version:

  • 40 s – initial version
  • 24 s – low hanging fruit
  • 15 s – optimized parser (ArrayBuffer)
  • 13 s – default rbush accessors
  • 14 s – web worker to render tiles (1 worker)
  • 14 s – web worker to render tiles (4 workers)
  • 9 s – parsing in own (single) thread (4+1 workers)

Let’s go though each of these steps, but let’s start with a short overview of the code structure:

Code overview

The code of the visualization isn’t very intricate, it basically just parses the tile log files (which are txt files containing pairs of tile-coordinates z/x/y and their respective view counts, see below), put’s them into a spatial index (I’m using rbush) and finally grabs the data from the index whenever a tile is requested to be rendered. (Then, the rendering of the tiles is just some pixel-pushing onto a canvas which is quick and wasn’t an issue that I had to look much into here.)

Here’s what the access logs look like: (this goes on for ~6,000,000 lines, or about 100MB of data)

13/4316/2511 20
13/4316/2512 18
13/4316/2513 16
13/4316/2514 14

low hanging fruit

flame graph of first profiling session

That’s the flame graph of the first profiling session. There are clearly two distinct processing steps, one relatively flat calculation, talking about 25 seconds and another more recursive which took about 10 seconds. The second portion is quickly identified as rbush building up it’s indexes (which is already pretty much fully optimized, I’d say). But what really stoked me was that the other part corresponding to the following few lines of code took up much more CPU:

    data = data.split("\n")
    data = data.map(function(line) {
        return line.split(/[ \/]/).map(Number)
    })

Pretty basic string operations, as it looks at first. Looking at the profiler again reveals that, of course, there’s a regular expression (/[ \/]/) in a hot loop and converting strings to integers using the Number constructor isn’t the fastest one can do either.

Getting rid of those two results in the first performance win:

line = line.split(' ')
var coords = line[0].split('/')
return { x: +coords[1], y: +coords[2], zoom: +coords[0], count: +line[1] }

optimized parser (ArrayBuffer)

Now, we’re still opening a function scope for each line in the input data and are working with relatively costly string-operations such as split and the + operator (to convert strings to numbers). Getting rid of that was quite fun and resulted in the biggest performance gain after which parsing was 90% faster than at the beginning!

What I ended up doing was to implement a custom parser that works on the raw byte-data (using ArrayBuffers), presuming that the log files are well structured. In its heart is a for loop that walks over all bytes of the data and manually constructs the data:

for (var i = 0; i<view.length; i++) {
    switch (view[i]) {
    default:
        currentInt = currentInt*10 + (view[i] - 48 /*'0'*/)
    break;
    case 10: // '\n'
        data.push({ x: currentCoords[1], y: currentCoords[2], zoom: currentCoords[0], count: currentInt })
        currentCoords = []
        currentInt = 0
    break;
    case 32: // ' '
    case 47: // '/'
        currentCoords.push(currentInt)
        currentInt = 0
    break;
    }
}

One interesting line to note is currentInt = currentInt*10 + (view[i] - 48 /*'0'*/): whenever we don’t see a separating character (newline, space or /), we assume that it must be a numeral, whose value we can get by subtracting the ascii code of 0 from.

default rbush accessors

The next optimization is a rather small one, but one I came across after the recent 2.0.0 release of rbush: apparently, it’s faster to access named attributes of a javascript object rather than elements of a javascript array. Changing the parsed data output to something that can be digested easily by rbush shaved off a few more seconds of the preprocessing.

web workers!

Even after all those optimizations, the calculations (even though they are relatively quick by now), are still blocking the UI. That isn’t a big deal during the initial processing, but the main-thread implementation means that rendering of the histogram tiles also blocks the browser. And even though each rendering is quick (typically only a couple up to a few tens of milliseconds), these small interruptions can add up significantly especially in situations when one pans around quickly or zooms in or out. It’s a bit hard to see in the gif below, but trust me: it feels quite laggy!

laggy zooming/panning

The only solution to this issue is to do rendering in a separate web worker thread. The implementation is a matter of refactoring the data parsing plus rendering code into a web worker and making sure that the returned data is a transferable buffer object. Using a single web worker, this is a bit slower than the non-threaded version, but not too much.

multi threading, first try

When we run a web worker anyway, why not multiple in parallel? That should make rendering of the tiles even faster, right? Well, not really in a naive approach: As every worker needs to have its own spatial index and there’s no way to effectively split the input data into distinct chunks that can be rendered independently later on, the total time with 4 workers is basically the same as with a single one (the overhead of having to duplicate the input data eats up any later gains in faster building up of the indices).

multi threading, second try

Doing multi threading properly in this case is a bit of a larger refactoring, but the effort is worth it in the end with additional ~30% faster processing and even smoother map panning and zooming.

Here, I’ve split the data parsing into a separate web worker which runs single-threadedly (this could in principle also be parallelized, but it’s not worth the effort in my opinion, as this step is quite quick with 2 seconds already – but, potentially one could shave of another second or so). The results of this parsing are then divided up into buckets of transferable ArrayBuffers (which are always important when working with web workers) and distributed among the rendering workers.

That’s how deep I dared to explore this rabbit hole of code optimization this time. I hope you liked my adventure. ;)

History of all Tags

Posted by tyr_asd on 31 August 2016 in English. Last updated on 1 September 2016.

TL;DR: head over to http://taghistory.raifer.tech/ for usage graphs of arbitrary OSM tags over time (by number of OSM objects).

In OpenStreetMap, tags define what an object is. Whether it is a mountain, a river, a house, or a postbox: Every map feature has it’s own tag (or set of tags).

OSM doesn’t have a fixed set of object categories. Over time, a more and more faceted and diverse set of features got mapped in OSM, thus the amount of different tags grew. At the same time, sometimes, tagging of a specific thing changes: Features that used to be mapped with one tag, get newer, better and more refined tags. That’s OpenStreetMap evolving.

Of course, OpenStreetMap is also still growing, but not all the tags are getting more widely used at the same pace: For example, while it’s quite possible that most of the world’s railway stations are already mapped in OSM, there are still many juicy pastures left to be mapped out there.

a friendly goat

While there exist superb tools to get to know about the current state of all tags used in OSM (Taginfo most notably, but also the Overpass API to some extend), until now it was quite difficult to get oneself a good picture of the data evolution process. For example, questions like: from when on a specific tag was getting used, when an obsoleted tag got taken over by a different one or which tags got more traction lately are difficult questions to answer with OSM’s current tool set.

For some of these questions, people programmed their own solutions, each answering their own question, like how many km’s of Italy’s roads were there in OSM over time (link), or how many buildings have been mapped in Austria (link). Similarly, the OSM-Analytics platform has recently started to provide such statistics for arbitrary regions for a limited set of map features (currently one can choose between buildings and roads, but there are plans to add more in the near future). What all of those tools have in common is that they can’t handle the full variety of tags that’s so essential in OSM.

To step into the gap between tools like taginfo (where the full variety of OSM’s tags is so beautifully visible – stay tuned for Jochen’s talk on SOTM in a couple of weeks!) and the more specialized tools like osm-analytics, I’ve created taghistory which allows one to get a historical usage graph for each of OSM’s tags (with daily granularity) and to compare different tags against each other:

highway=ford vs. ford=yes

The tool is currently in it’s very early stage, the’re many things to do and improvements to be done. It’s also important to note that the historical usage of a tag is currently only defined as a the respective number (count) of OSM objects! That’s similarly to the statistics produced by taginfo, this metric is subject to the some limitations, most notably the effect that one cannot directly compare the number of tags used for different linear and polygonal features such as roads, land cover, etc. because such features are typically divided up into many OSM objects of different sizes. For example, an existing road may be divided up into two pieces when a new turn restrictions is added, resulting in that the count of each of the tags used on the road (even obsolete ones) is increased by one in the OSM database. That means that one needs to pay close attention when comparing tags that are typically used on such features, even when comparing subtags that are typically used on the same kind of parent object (e.g. different values of the highway tag).

That being said, have lot’s of fun while digging into the depths of OSM tags’ history. Here’s the link of the tool again: http://taghistory.raifer.tech/ (and the link to the project’s source code repository and issue tracker: https://github.com/tyrasd/taghistory). What’s your favourite tag? I find the created_by graph quite interesting:

history of the usage of the created_by tag

OSM Node Density – 2016 Update

Posted by tyr_asd on 7 July 2016 in English.

My OSM node density visualization has just been updated to show the freshest OSM data!

A significant difference from the 2014 edition is that the density is now calculated as OSM-nodes per m² on the ground (as opposed to nodes per projected pixel in the previous version). If you want to learn more about why this change makes a difference, I’d strongly recommend the following article by Christoph Hormann (imagico).

The updated slippy map lets you chose between the layers of different years (always created around end of June of each year) as well as a display of what changed during each 12 month interval. For example you can see that in North America, there were some imports as well as some major import-cleanups going on during the last year:

Last, but not least: you can now find the source code and instructions to generate all of this in the repository on github: https://github.com/tyrasd/osm-node-density/tree/gh-pages/code

2 years of overpass turbo

Posted by tyr_asd on 8 February 2015 in English. Last updated on 28 September 2020.

overpass turbo has been around for a little over two years now. In this time, it arguably changed how developers and mappers interact with OSM data. Let us take this opportunity to look back and take a glimpse at some statistics:

users

The user-base has more than quintupled from the initial group of early adopters as can be seen in the following Piwik graph:

visitor stats

Note that the actual absolute number of visitors is likely significantly higher than what is reported here, because surely many of you have the do-not-track flag activated or are using tracker blocking software in your browsers. Speaking of it – as of today you can opt-out from any tracking on overpass-turbo.eu also by simply switching it off in the settings dialog under the privacy tab.

shared queries

Shortly after its release, overpass turbo got the ability to share queries in the form of short URLs (e.g. http://overpass-turbo.eu/s/4). Here is some insight into what queries people have been sharing since then:

This map shows the locations associated with each shared query: shared query locations

Of course, central Europe is quite the center of activity, but in general the tool seems to be used all over the planet, which is nice.

The next thing we’re looking at are the two query languages. In the beginning overpass turbo preferred the Overpass XML variant (in code examples and queries generated by the wizard). Later, this default was switched over to the QL query language. This can be seen in the following graph: red is XML, blue stands for QL, brighter colours stand for queries that are taken or derived from output produced by the query wizard. Each column represents a set of 512 consecutively shared queries. Note that this means that the x-axis isn’t a linear function of time [timestamps are not stored in the short-url database as they aren’t needed to provide the service].

query language stats

One can immediately identify two main events: First, the introduction of the query wizard in Dezember 2013, and the above mentioned switch from XML to QL as the default query language in October 2014.

Another interesting fact: about 10% of all shared queries use some amount of MapCSS styling.

The question now is what will the next few years bring? Let’s find out! ;)

Full geometry ahead!

Posted by tyr_asd on 24 October 2014 in English.

This summer brought us some nice new features to the Overpass API universe, that are now finally available in overpass-turbo (and osmtogeojson). Here are some of the highlights:

geometry options

The Overpass API now has three new geometry types, which give you even more flexibility in your queries:

  • out center; – this additionally prints the center coordinate for every OSM object
  • out bb; – this additionally prints the bounding box coordinates for every OSM object
  • out geom; – this additionally prints the full coordinates of every OSM object

The first two options are particularly usefull if one is only interested in the approximate location of some features, rather than their exact outline. For example, one finds that POIs are often mapped on building outlines. By requesting only the center coordinates (out center;), one saves transfer bandwidth and gets an overall quicker query.

Here is an example how this looks like (note that out of the 7 displayed POIs, 6 are mapped on ways in OSM): (try it on overpass turbo)

The full geometry (out geom;) option replaces the need to use object recursions to get the geometry for a certain OSM way or relation. This also saves bandwith and generally comes with faster query execution times. (In this example it saves almost 50% of the data.)

With osmtogeojson version 2.2.0, all this is also available on your command line or to be used as a javascript-library in your projects!

regular expressions for keys

I know that lots of people requested this feature, and now you can finally use regular expression on keys, for example to get all nodes with some kind of name-tag:

node[~"(^|_)name($|_|:)"~".*"]({{bbox}}); out;

Here, the additional tilde in front of the key indicates that the regex-search should be extended to the key as well.

This is also accessible via overpass turbo’s query wizard. Try this for example: building=* and ~"addr:.*"~".*"

Overpass QL is now the default

Queries generated by the wizard, a template and the query examples are now printed in Overpass QL. Overpass QL has a more concise syntax, is faster to write and more and more documentation and help is available for it. (Of course anyone can still continue to write, use and execute queries in the older XML syntax.)

OSM Node Density 2014

Posted by tyr_asd on 27 June 2014 in English.

Exactly one year ago I published my first visualization of the global OpenStreetMap data density. This is the updated 2014 edition.

(click image for slippy map or here for high-res images)

Each pixel shows the number of nodes in its corresponding area¹. But this year every point that has data in it is shown (i.e. there is at least one node at that location - last year only locations with more than 1000 nodes were included). Also, the slippy map has two more zoom levels which reveal even more impressive details like on this crop of the central Alps:

Here is a low-zoom image of the whole planet:

¹ Yes, this is Mercator map tile area, not actual on-the-ground area. Keep this in mind when comparing regions at different latitudes!

² Copying: visualizations © Martin Raifer, CC-BY - source data © OpenStreetMap contributors, ODbL

PS: The visualizations are based on a planet file I downloaded one or two weeks ago. It was processed using some custom scripts based on node-osmium, the graphics were made with gnuplot (just like last years’) and finally the map tiles for the slippy map were cut using imagemagick. I could probably explain the individual steps in a separate blog post, if anyone was interested - let me know!

overpass turbo updates

Posted by tyr_asd on 22 June 2014 in English.

In the last weeks a few new features got into overpass turbo. Here is a quick heads-up:

Text Markers in MapCSS

You can now finally use MapCSS to display text markers on the map. This is very useful to show the names of displayed features. The following MapCSS statement adds text markers to each node, way or relation on the map and displays whatever is in the tag name:

{{style:
  node, way relation {
    text: name;
  }
}}

Or take a look at the following example:

Export to Level0

Overpass turbo can now export data into the Level0 editor. Level0 is a very lightweight low-level textual online OSM editor. It can be very handy for example when inspecting tags of a collection of similar objects (such as when checking the names of all restaurants in town).

<todo: gif?>

Additional Stats

When you hover the data-counter box (which shows you how many elements have been loaded and are displayed), you get presented some additional statistics. Currently, only the lag of the Overpass-API server behind the main OSM db is shown, but there is room for more. Is there something you would like to see there? Let me know!

Wizard as URL-parameter

You can now create links to overpass turbo that use the query wizard to create a query (which is loaded on startup and presentet to the user). An example is http://overpass-turbo.eu/?w=pub+in+dublin. Such URLs to overpass turbo are both quite short and at least somewhat human readable.

Über Imports

Posted by tyr_asd on 11 April 2014 in German (Deutsch).

Imports sind für OpenStreetMap ein sehr zweischneidiges Schwert: Natürlich führen sie kurzfristig zu besseren Daten, aber sie führen zu keiner nachhaltigen Stärkung der Community. Vor allem von Außen kommende Imports sind für OSM nicht nachhaltig, weil sie keine neuen Mapper anlocken und nicht einmal bestehende Mapper einbindet. Bei OSM kommt es nicht so sehr auf die Quantität der Daten an. Viel wichtiger sind Qualitätsmerkmale wie Aktualität, Ausgewogenheit, Nützlichkeit, relative Genauigkeit, usw.

Was ich meine ist: Wenn eine Gegend/Dorf/… in OSM noch schlecht erfasst ist, ist auch wenig geholfen, wenn auf einmal alle Gebäude da sind, wenn die eigentlich wichtigeren Dinge wie Straßen, Straßennamen, POIs, usw. fehlen. Vielmehr würde das eher eine falsche Vollständigkeit vortäuschen („Da sind ja sogar schon alle Gebäude gezeichnet, da muss ich nichts mehr mappen“). Wenn aber eine Gegend in OSM schon recht gut erfasst ist, werden die lokalen Mapper von alleine irgendwann entscheiden, dass es an der Zeit ist, Gebäude nach-und-nach einzutragen. Dabei werden sie sich dann automatisch der jeweils besten verfügbaren Datenquellen bedienen: Entweder vom Luftbild abzeichnen (zur Zeit die einzige Option), oder ein Import von etwaigen offenen Datensätzen. Wie man sieht, ein von Außen “aufgezwungener” Import ist so oder so nicht notwendig.

Ich kann mehrere Beispiele nennen, wo verfrüht durchgeführte Imports zu einer Verschlechterung der Gesamtsituation für die OSM-Community geführt haben: Meran war eine der ersten öffentlichen Institutionen, die ihre Daten (speziell für OSM) freigegeben haben. Das kam damals sehr gut an, schaffte es sogar in eine OSM-Hall-of-Fame. Jetzt, fast 7 Jahre später, kann man feststellen, dass Meran in OSM zu den am schlechtesten gemappten Gemeinden Südtirols gehört (im Verhältnis zur Einwohnerzahl wahrscheinlich das Schlusslicht). Ähnliche Stories findet man in OSM zu Hauf: in Frankreich, im Veneto, in der Emiglia Romagna, … Und von dem “wenig erfolgreichen” TIGER-Import in den USA muss ich wahrscheinlich auch nicht viel erzählen.

Für bestimmte Daten kann bzw. sollte man schon Ausnahmen machen. Dazu gehören Sachen wie administrative Grenzen, weil diese alternativ kaum anders zu erfassen sind. Adressen gehören in gewisser Weise wahrscheinlich auch zu dieser Kategorie.

Versteht mich bitte nicht falsch: OSM freut sich natürlich über alle Daten, die es bekommt, aber, wie oben geschrieben, nur wenn diese für OSM jeweils auch sinnvoll einsetzbar sind. Auch wären wir natürlich auch über jede technische Unterstützung dankbar. Nur gäbe es in OSM genug Dinge, die viel dringender Arbeit benötigten als als ein Import fehlender Gebäudeumrisse. Hier in Südtirol denke ich da einmal spontan an die Vervollständigung der Straßennamen, Qualitätskontrolle für POIs, Kontrolle von sicherheitsrelevanten Merkmalen von Wanderwegen, usw.

Location: Historischer Ortskern Meran, Meran, Burggrafenamt, Bozen, Trentino-Südtirol, 39012, Italien

Overpass Query Wizard ++

Posted by tyr_asd on 31 January 2014 in English. Last updated on 7 June 2014.

overpass turbo’s query wizard just got a little bit more useful! It understands not only OSM tags, but even interprets object classes: Just use simple terms such as Restaurant, Museum, Hotel, Hospital or any other thing, the iD editor already has a preset for.

This even works in your own language, as you can see in the following screenshot (Trinkbrunnen is German for “drinking fountain”):

Btw: if you can read German, you can find some more information about this in my recent guest blog article on blog.openstreetmap.de.

The Overpass Query Wizard

Posted by tyr_asd on 12 December 2013 in English.

overpass turbo is a helpful tool for hundreds of mappers every week. But it could be an even better tool for even more people if only there was a way to assist with the creation of actual overpass queries. You know that it’s quite tedious to type all those queries that are mostly the same all the time. Not to mention that one must remember all those overpass statements and their parameters all the time.

But here comes a way that makes overpass turbo both easier to use for beginners as well as quicker to use for experienced data-miners:

The Query Wizard

Designed to transform simple, human-readable search terms into functional Overpass queries.

Here is an example: To get restaurants, now all you have to do is fire up the Wizard from the tool-bar (or using the keyboard-shortcut Ctrl+I) and type in the appropriate tag amenity=restaurant:

Query Wizard in action

The wizard is quite powerfull: It understands different kinds of tag searches as well as some meta data filters which can be joined together with logical operations like and or or.

By default, data is searched in the current map viewport. But the Wizard also recognizes some other location filters. For example, you can simply write tourism=museum in Vienna and the query will work just magically.

Query Wizard in action

Read the Wiki for more details and technical background.

New Overpass-Shortcuts

If you tried the museums in Vienna example (tourism=museum in Vienna) from above, you may have noticed the following line in the produced Overpass query:

<id-query {{nominatimArea:Vienna}} into="area"/>

As the Overpass API doesn’t really know which Vienna it should use to search in, we ask Nominatim instead. The {{nominatimArea:Vienna}} part is then replaced with the details of the Nominatim search results (just like {{bbox}} is replaced with the bounding box coordinates).

Appart from the just mentioned nominatim-shortcut (which actully comes in several flavours depending on your use-case) there is also a new {{date:*}} shortcut, which allows one to specify relative dates (typical use-case: <newer than="{{date:1 week}}"/>). Read more in the docs.

What else is new

You can now inspect tiny features more easily by zooming in one more level (overscaling): Like here for the smallest house of Vienna:

PS: You can now Flattr my project, if you want. Flattr

SASA bus stops import – results

Posted by tyr_asd on 10 October 2013 in English. Last updated on 3 December 2013.

A while ago, the SASA (a local public bus service operator in South Tyrol) gave us the permission to import their bus stops from opensasa.info into OSM.

Here is the result:

Btw: In the meantime SASA is working on getting their real time bus information out to the public. Take a look at their public beta! And of course they are using a beautiful OpenStreetMap background map. :)

Location: Zona Industriale Bolzano Sud - Industriezone Bozen Süd, Oltrisarco-Aslago - Oberau-Haslach, Bolzano - Bozen, South Tyrol, Trentino – Alto Adige/Südtirol, 39100, Italy