OpenStreetMap Notes: some interesting stats

Posted by joost schouppe on 1 January 2018 in English (English)

OpenStreetMap Notes make it easy to share some info about mistakes or missing data in OpenStreetMap. It's a very open system, allowing for posting and commenting from third party apps and even without logging in.

I'm working on an exhaustive analysis of how notes are used. Here are some prelimanry findings.

Notes are often personal

While the wiki doesn't encourage using Notes as a personal to-do list, very many Notes get closed by the original poster. In fact, of all Notes that were posted by a logged in contributor, that are at least 90 days old, and are already closed, 50% were closed by the original mapper. The vast majority of those "self-closed" Notes (96%), never saw interaction. Notable though, is that IF there was some sort of interaction, there was a reaction by the original contributor in almost 90% of the cases.

One of the things that I want to investigate, is if these people who are closing so many of their own Notes, are also up to closing other Notes. What I would like to know is if the surge in Notes created more note-closers, and if so if they were new to notes or not. is huge

When implemented Notes into their app, the impact was huge. Basically, the number of Notes doubled overnight. The graph shows the monthly opened notes. In the first months of Notes, the added notes were about the same as all other notes together! surge

Since its peak, the number of Notes has been steadily going down, while the number of other notes is going up. My method of identifying notes is looking for "#mapsme" or "" in the opening comment, so maybe this string isn't included anymore in some versions?

And we can't keep up, right!?

Of course, this surge had a negative impact on the closing rates of Notes. However, the overall statistics that we are used to seeing, paint a somewhat overly negative image.

A tiny bit of methodology. I'm interested in how we handle incoming Notes. That means you need to look at rates, not raw numbers. One way to do that, is to define a goal, and see if we're attaining it. For example, the graph below takes the goal "Notes should be closed within 90 days". (note: I did not take account of the re-opening of Notes here.) That of course means you have to wait till a Note is at least 90 days old before you make a decision on the stat.

The graph shows that until the surge, we had a slightly downward trend, but usually at least 60% of Notes were closed within 90 days. The "overall" line shows that this rate went steeply down as surged. However, this is mostly caused by the notes having a much worse performance. The closing rates of app-free notes did go down a bit, but in most months stayed close to 60%.

Closing rates

Those annoying users?

The reason for a slower closing rate might be simple: Notes usually aren't made by people who close their own Notes. So a hypothesis I want to check is, if you split Notes between "made by people who tend to just work on their own Notes" and "Notes made by people who just want to share some information", is there still a difference between and other notes?

One of the main complaints about users is that they don't respond to comments. Of course, there can only be something to respond to, if their Note first gets commented. For that to happen, the Note has to be non-anonymous and commented by someone else who was logged in. To my surprise, this is a relatively rare situation. Less than 4% of all existing Notes were made by logged in users and responded to by other logged in users.

In this rare situation, there is a reaction from the original contributor in about 30% of the cases. If the Note was made through or an unknown app, this rises to about 40%. However, if the Note was made with, response rate was only about 10%.

Since Navmii makes only anonymous notes, this rate can't be counted for that app. The recent StreetComplete notes get a 26% response rate in similar circumstances. So there is clearly something going on with Some further measures to increase comparability might be necessary, for example excluding Notes from heavy mappers. But before we saddle the horses, don't forget what the actual number is we're talking about. In 2016, if users were exactly like the general population, there would be just 2500 more reactions from original contributors, on a total of 123.797 notes that year.

Other explanations

So the lack of communication might be relatively minor compared to the "abuse" of Notes by people who don't know what they are meant for. If someone would like to do some text analysis to see if automatic classification of usefulness is possible, I could offer a file that is ready for consumption. (the Notes dump is relatively easy to parse though)

A bit harder to analyse, would be the hypothesis that the difference is because notes tend to contain some info that is hard to ignore (like "this place is closed now"), but hard to verify ("did you really mean Place X"? If you don't trust the contributor, and they don't answer, you have to go check).

In other news:

What on earth happened here? From May to July 2017, anonymous comments went absolutely through the roof, and then just as quickly went down to normal again. Most seemed to have been nonsense comments. Does anyone know what happened here?

Comment spike

Oh, and if you're working with the Notes Dump yourself, be mindful of this minor bug I found.

Work in progress

I've been dreaming of a global database of local basic statistics about OpenStreetMap for years. That turned out to be a little over-ambitious for me. Hence my retreat to local statistics for Belgium only. Analyzing Notes is a kind of proof of concept. This article is a little side to the process of publishing statistics about Notes for the world, but also all continents, countries and "regions" in the world. Since these are the tools I'm fluent in, I wrote the analysis scripts in SPSS. They are available on Github. It should be relatively straightforward to translate to other, more open languages. Publishing will be on the platform. Unfortunately closed source payware, but they donated a version for OSM analysis purposes.

Shape this project

I usually start of projects like this with a clear goal in mind, then lose track of that goal completely because of all the interesting side streets I find. Working towards the platform helps shape that a bit. But what kind of questions would you like to see answered? I intend to make it easy to play around with the numbers yourself, but maybe there's some aspects I'm missing entirely. Let me know! Feel free to post here or as a Github Issue.

Comment from mmd on 1 January 2018 at 22:57 has some addntl details on the large number of anonymous notes last year.

Hide this comment

Comment from Math1985 on 2 January 2018 at 10:39

In fact the wiki does not discourage using Notes as a personal to-do list (it explicitly says 'Although you can use notes as a reminder to yourself, you are also inviting others to look at it.'.)

I can't speak for others, but I solve a lot of the notes that I create myself, but also solve a lot of notes created by other users.

Hide this comment

Comment from philippec on 2 January 2018 at 14:48

As long as notes are not categorized according to cause or task, discussion is difficult e.g. - How do I tag that ? - Can someone take a picture or have a look ? - I'll have a look one day, but someone may do it before me. - Someone should place that on the map. ... - It was dark I could not see a thing.

I often resolve my notes just to clean up. That does not mean that they were on my todo list.

Hide this comment

Comment from joost schouppe on 2 January 2018 at 18:45

@mmd: thanks, I posted there

@math: to me that sounds a strange after first saying "Don't use it to put your personal notes." I'm a bit confused by the wording of that bullet point. Anyway, I don't mind people doing that, in fact I find it mostly interesting because it might point to a need to make it easier to see your own un-closed notes (you have to go to a third party website now)

Hide this comment

Comment from Mateusz Konieczny on 2 January 2018 at 19:03

If you don't trust the contributor, and they don't answer, you have to go check

wait, there are people that delete POIs based solely on notes, without verification?

Hide this comment

Comment from Mateusz Konieczny on 2 January 2018 at 19:07

to me that sounds a strange after first saying "Don't use it to put your personal notes." I'm a bit confused by the wording of that bullet point.

I attempted to improve it in

I think that intended meaning of "personal note" is "note that only original author is able to use" not "note that original author intends to solve".

I frequently add notes using Vespucci/StreetComplete that are much easier to solve using JOSM.

Hide this comment

Comment from Mateusz Konieczny on 2 January 2018 at 19:11

But what kind of questions would you like to see answered?

Recently I was looking for tool that would list unsolved notes made by StreetComplete, with plan of commenting where somebody may be advised how to use SC.

Maybe also spot some usability issues that happen regularly with different people.

Hide this comment

Comment from Geonick on 8 January 2018 at 16:58

Interesting web app I'm collecting such tools for OSM Notes management here: Perhaps I should move this content to the page?

Hide this comment

Comment from westnordost on 8 January 2018 at 18:13

I think that would make sense. ENT8R is a contributor to StreetComplete, I am the author.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

  • Image

    ![Alt text](URL)

Login to leave a comment