OpenStreetMap

mvexel's diary

Recent diary entries

How would you map this?

Posted by mvexel on 11 May 2017 in English (English)

I am trying to figure out mapping complex intersections and I am a little stumped :) To completely represent all possible lanes and turns in an intersection, you would need to define:

  1. The lane layout (turn lanes, through lanes)
  2. The lane connectivity (which lane connects to which at the far end of the intersection)

If I asked you to map this intersection completely, how would you do it? Which (combination of) turn / lane tagging schemes would you use?

inters

(If you're interested, this type of intersection is a CFI or continuous flow intersection.)

Latest MapRoulette newsletter

Posted by mvexel on 28 April 2017 in English (English)

I am going to stop posting the full monthly MapRoulette newsletter here from now on, but the April newsletter has just been sent to the subscribers! You can view it here.

Subscribe to the newsletter on this page.

Maproulette Newsletter - March 2017

Posted by mvexel on 27 March 2017 in English (English)

Here is the latest from the MapRoulette world! If you want to get this newsletter in your mailbox, you can sign up here!

featured MapRoulette was featured in the JOSM message of the day!

MapRoulette has seen a lot of activity in the past month! A total of 407 mappers have logged on and fixed more than 32000 tasks. That is really cool.

New and notable challenges

Also, lots of new challenges! 539 to be precise created in the last month. I know that challenges can still be hard to discover (working on that, I would welcome ideas and help there!) so I want to just manually highlight some challenges that look interesting. If you want your challenge highlighted in this newsletter, please write to maproulette@maproulette.org!

shorelinesvsreality

The latest from Github

How to: Task Instructions

Perhaps you have seen that the task instructions do not always fit in the panel.

nofit

That is annoying because it can be unclear what you should do. Usually this is caused by hyperlinks that are too long and can't be broken up. If you are a Challenge creator, you can avoid this by using markdown to wrap the hyperlink. Instead of using the raw link in your instruction (https://long-long-url.org), use this: [description](https://long-long-url.org)and it will be displayed like this: description.

Help Wanted: Challenge Administrators

With so many new challenges coming in, it would be great if a few of you would volunteer to help with Challenge maintenance. That means looking at new Challenges that users create to see if they make sense, and helping the Challenge creators to improve them if needed. If you are interested in helping with this, email maproulette@maproulette.org!

That's it for this month. Please write in if you have suggestions for this newsletter. Happy mapping!

MapRoulette newsletter

Posted by mvexel on 9 March 2017 in English (English)

Here's the latest from the MapRoulette world!

New version released

If you head to MapRoulette.org, you will see that we have a new release out, 2.0.3. This release addresses some annoyances with the keyboard shortcuts, adds a German translation (das freut mich! Thanks nebulon42!) and cleans up the interface in a few places.

keyboard-hints

A new section displaying keyboard hints

Mapping Activity

In the past 30 days, we fixed almost 18000 tasks in MapRoulette.

metrics

Metrics for the last 30 days. You can see these for yourself on MapRoulette.org

The most popular challenges were:

  1. Self-Intersecting building outlines - all done!
  2. Crossing Ways: Highway-Railway, US - 55% done, still more than 17000 tasks available
  3. CHN_BuildingRoadIntersectionCheck, 34% done, still 2800 tasks available
  4. Self-Intersecting landuse outlines (World-wide), all done!
  5. Open Rings, all done!

Looking for something to do?

Some interesting Challenges that still need help:

By the way: I created the abbreviated road name challenges using an Overpass query. Did you know that you can turn any Overpass query into a MapRoulette challenge? Read this to learn more.

And also...

  • Do you want to see your own challenge featured in this newsletter? Need help creating your challenge? Contact us at maproulette@maproulette.org!
  • Do you want to receive this newsletter in your inbox in the future? Sign up here!
  • The MapRoulette Questionnaire is still open if you want to share your opinions on MapRoulette! One lucky participant will receive a mappy prize at SOTM or SOTM US.

MapRoulette newsletter

Posted by mvexel on 9 March 2017 in English (English)

Here's the latest from the MapRoulette world!

New version released

If you head to MapRoulette.org, you will see that we have a new release out, 2.0.3. This release addresses some annoyances with the keyboard shortcuts, adds a German translation (das freut mich! Thanks nebulon42!) and cleans up the interface in a few places.

keyboard-hints

A new section displaying keyboard hints

Mapping Activity

In the past 30 days, we fixed almost 18000 tasks in MapRoulette.

metrics

Metrics for the last 30 days. You can see these for yourself on MapRoulette.org

The most popular challenges were:

  1. Self-Intersecting building outlines - all done!
  2. Crossing Ways: Highway-Railway, US - 55% done, still more than 17000 tasks available
  3. CHN_BuildingRoadIntersectionCheck, 34% done, still 2800 tasks available
  4. Self-Intersecting landuse outlines (World-wide), all done!
  5. Open Rings, all done!

Looking for something to do?

Some interesting Challenges that still need help:

By the way: I created the abbreviated road name challenges using an Overpass query. Did you know that you can turn any Overpass query into a MapRoulette challenge? Read this to learn more.

Do you want to see your own challenge featured in this list? Need help creating your challenge? Contact us at maproulette@maproulette.org!

Adding unknown roads using ImproveOSM

Posted by mvexel on 24 February 2017 in English (English)

When using ImproveOSM, the Telenav tool to find and add missing roads (and other missing things) in OSM, sometimes there is just no detailed aerial imagery to see exactly what road type it is.

example

While you could say, 'Well without being able to see, I won't add anything'. But because ImproveOSM uses actual GPS traces from drivers, you know that people have been driving there. From the image in the example you can see that a fair number of people must have taken this route, so there must be some sort of road there.

So what I suggest is to add the road as simply highway=road (generic road tag). Then you or someone else can improve it later, based on better imagery or local knowledge.

The example in the image is this new way.

Remember that you can select multiple tiles on ImproveOSM by holding Shift while selecting. That way you can more quickly mark multiple tiles as Solved.

markdone

Creating MapRoulette Challenges from GeoJSON

Posted by mvexel on 8 December 2016 in English (English)

This post also appears on the MapRoulette wiki

Creating MapRoulette challenges has never been easier. You can use the API or the brand new Challenge Wizard. I will cover the API in a future post. I have already covered the Wizard in general in a previous post, but right now I want to focus on creating a Challenge with a GeoJSON file.

In my example case, I have a GeoJSON file containing all the bus stop locations in Utah. I downloaded these as a Shapefile from the Utah GIS portal and converted them to GeoJSON using QGIS (there are other methods too, like ogr2ogr but I am too old for that). The result looks like this:

{
    "type": "FeatureCollection",
    "crs": {
        "type": "name",
        "properties": {
            "name": "urn:ogc:def:crs:OGC:1.3:CRS84"
        }
    },
    "features": [
        {
            "type": "Feature",
            "properties": {
                "StopId": 13840.0,
                "StopName": "HILL FIELD RD @ 1891 N",
                "StreetNum": 1891.0,
                "OnStreet": "N HILL FIELD RD",
                "AtStreet": "ANTELOPE DR",
                "City": "LAYTON",
                "InService": 1,
                "Bench": 0,
                "Shelter": 0,
                "Lighting": 0,
                "Garbage": 0,
                "Bicycle": 0,
                "Transfer": 1,
                "LocationUs": "Bus Stop",
                "UTAStopID": "101001"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -111.97361202442978,
                    41.08738199558664
                ]
            }
        },
...

With this GeoJSON, I go to MapRoulette. After logging in, I go to my Project and create a new Challenge. (For more details on this, see my original post on creating challenges.) I enter the basic information for the Challenge:

basic

Then I go to the next screen to choose my GeoJSON file:

geojsonchooser

I choose sensible numbers for the default zoom levels and base map:

basemap

And then I can complete my Challenge.

completebuilding

After a while I hit refresh in the browser and see that my Challenge is complete, with a little over 6000 tasks. This is the same as the Feature count in the original Shapefile, so I am happy. I can now go to a random task:

randomtaskstart

And see what the challenge looks like:

lookslike

Looks pretty good! I can also click on the marker to see the GeoJSON properties:

balloon

Now everyone can help add bus stops near me!

But wait...

A word on the quality of this Challenge. It is really hard to verify that a bus stop actually exists using just Bing imagery. So this Challenge may not be suitable for anyone who does not already know the area very well. Another good source for people who don't know the local area is OpenStreetCam or Mapillary imagery. Without knowing if there is coverage for these sources, the Challenge can be hit or miss.

OSC

Finally, MapRoulette does not (yet) add the location of the new node to JOSM or iD when you click 'Edit', so you end up switching back and forth between MapRoulette and editor to figure out where to add the bus stop.

In the future, we want to extend MapRoulette to support better handling of GeoJSON source data:

  • Automatic adding of the feature to JOSM or iD (if possible) when clicking 'Edit'
  • parsing of GeoJSON properties and adding them as tags for the new feature

So with a properly formatted GeoJSON you could take most of the hard work out of mapper's hands and they would just need to verify and upload. That would be great progress for community-driven human imports. Please help make this happen, contribute to MapRoulette!

Happy Mapping!

`exit_to` --> `destination` in Canada

Posted by mvexel on 24 October 2016 in English (English)

We had a discussion about retiring the use of exit_to in favor of destination for motorway / trunk exit tagging in the US a while ago. Since then, a shift has happened and a majority of exits is now mapped with destination instead of exit_to. This is great for navigation applications that rely on detailed signpost information, such as OsmAnd, maps.me and Scout.

With the situation in the U.S. so much improved, a next obvious target for mapping is Canada :) The situation there has been improved already by the community, but also by organized mapping by Mapbox and Telenav. Still, about 1200 exits remain that are tagged with the 'old' scheme. With the help of my colleagues at Telenav we put these in a MapRoulette challenge.

maproulette

Because both OpenStreetView and Mapillary have good coverage in Canada, I think we should be able to update most of these exits to use the new scheme. Let's give this a go!

Cemeteries in Texas MapRoulette Challenge now powered by Texas Imagery Service

Posted by mvexel on 20 October 2016 in English (English)

Important note: The imagery I use as an example below is different in source from the imagery used for the MapRoulette imagery. The example below shows imagery that was commissioned by Texas itself, and is available in the public domain. The imagery in the MapRoulette challenge is licensed from Google by Texas and made available to MapRoulette specifically. So I can't say positively that it's OK to add this imagery to JOSM or iD, and removed specific instructions to do so.

Have you tried the Texas Cemetery challenge in MapRoulette? If you have not heard about it yet, I posted about it on my diary a few weeks ago. The short version: the friendly folks at TxDOT supplied me with a database of their known cemetery locations, we matched them with existing OSM, and if there was no match, we ask you to map it :)

If you tried it, you may have found though that it can be a bit frustrating :( Bing and Mapbox aerial imagery is often just not detailed enough to see if there is a cemetery or not in the location indicated. I discussed this problem with the friendly folks over at TxDOT, who are very excited about getting more data into OSM. They told me about some of the high resolution imagery that is available to the public through TNRIS, the Texas Natural Resources Information System. Here is an example of some of the amazing data they have:

example

If you compare that with Bing, it means the difference between seeing a vague blur or seeing the presence of a cemetery very clearly! Super exciting stuff. So we set out to create a TMS endpoint that we plugged into MapRoulette. See the difference!

difference

Go give it another try! Thank you and thanks TxDOT!

Spotting Cemeteries in Texas

Posted by mvexel on 23 August 2016 in English (English)

I am collaborating with agencies in Texas to update both OSM and Texas data. The pilot project deals with cemeteries. I received a file with almost 7000 cemetery locations. (Even if the idea that there are more people living today than have died thus far in human history turns out to be a myth, I think that is quite a lot!).

The first phase of this collaboration is to see which cemeteries in the Texas data actually exist. We will use MapRoulette for that. Simply go to the Cemetery challenge at maproulette.org and start looking at tasks.

If you see a cemetery in the aerial image, click 'skip' to go to the next one. If you don't see a cemetery, click 'False Positive'. If you are in doubt, click 'skip'.

How can you tell if there is a cemetery? Sometimes it is hard. Look for fine patterns defining the plots, and usually there will be a service road connecting the cemetery to the road network. Sometimes, in larger cemeteries, you may also see paths inside the cemetery. Finally, the marker may not be right on the cemetery, so look around a bit as well. Below are some examples of cemeteries and non-cemeteries.

Once we complete stage 1, we will turn to mapping all the cemeteries that are not yet in OSM yet!

cem

There is a cemetery here: fine regular pattern indicating plots, some paths.

cem

There is a cemetery here also.

cem

No cemetery here, just some grass.

Introducing OpenStreetView

Posted by mvexel on 15 August 2016 in English (English)

After almost a year of thinking, development and testing, the OSM team at Telenav is ready to present OpenStreetView to all OSM mappers! OpenStreetview (OSV) is the free and open street level imagery platform designed 100% with OSM and mappers in mind.

We officially presented OSV to the OSM community at State of the Map US where we had a 20 minute talk and a booth where we gave away crazy little remote controlled cars to everyone who signed up :). The cars were gone quickly – almost half of the people at SOTM US signed up! - but you can still see the talk thanks to the great SOTM US organizers who had all the sessions professionally recorded. If you have 20 minutes and don't like reading, watching that video is going to be the best way to be introduced to what OSV is and how you can use it to improve OSM. Or if you are coming to SOTM in Brussels, you can come meet our team there (more remote controlled cars? Who knows!) and attend the workshop.

booth

The OpenStreetView booth at SOTM US

If you do prefer reading, read on! I wanted to quickly introduce OSV, what the components are, why we believe it is the #1 choice of street level imagery for OSM, and of course how to contribute and use it.

OpenStreetView components

OSV is a web site, openstreetview.org, free and open source mobile apps for Android and iOS, a specialized Map Editor, a plugin for JOSM, and of course a back end server. Support for iD is also planned.

The web site is where you go to explore imagery from all over the world, see leaderboards and your own profile and trips. To see your personal stuff, of course you will need to sign in. Your OpenStreetView account is linked to your OSM account, so you don't need to create a separate account. All we store when you sign in for the first time is whatever is public on OSM. (If you want to check what that is, go to https://www.openstreetmap.org/api/0.6/user/8909, changing 8909 to whatever your OSM ID is -- unless you want to see my details.)

website

The apps are free to download from the play / app store. For Android, you can also download the APK directly. With the apps you can capture trips. They are optimized for driving but also work well for biking and walking scenarios. Apart from recording trips, you can also upload your trips to OSV. This will happen automatically when you enter WiFi, if you want. Finally you can review your local and server trips and see your profile. Even if you have not logged in on the web site, you can log in to OSV with the apps, also through OSM OAuth. Either way, this will create an account for you on OSV.

android-screenshot

One thing that is really specific to OSV is that you can link the app to an OBD2 dongle in your car. Those are little devices that read from the OBD2 port in your car. Almost every car has one. (Challenge: find yours!). The dongle reads all kinds of diagnostic info from the car and broadcast it over Bluetooth or WiFi. They cost around 20 Euros. A list of OSV compatible ones is on the OSM wiki. (Ehm, a very small list so far. If you have a different model, please add your experience!)

obd2 dongle

OSV will read the speed and curve to improve the accuracy of the GPS signal that is recorded for your trip. It comes in extrememly handy when GPS reception is poor or lost altogether, for example in dense tree cover or in tunnels. The dead reckoning provided by the OBD2 unit will maintain proper alignment to the road. Here you see what that means when you are driving through a tunnel (blue = GPS only, signal lost, red = with OBD2 connection).

deadreckoning

The OSV apps also have sign detection built in. So this is not done on the server but at 60FPS on the client! This means that it will detect speed limit signs, and more to come, in real time and can warn you if you are speeding. This warning feature is almost ready and will be in one of the next builds. (We update the apps very frequently.)

speedlimit

The JOSM plugin is in an early beta stage. Right now it will simply display the locations of images on the OSV server, and you can click on them to show the image in the OSV panel. Basic functionality, but it works :) and we hope that you have ideas (or even code) to improve it.

josmplugin

Speaking of ideas, we already have an active community reporting issues and suggestions on Github. This is the best place to let us know of any bugs and ideas you have about any part of OSV. Github is also where all the source code for all the components mentioned here is located. Almost everything in OSV is open source, and if it is not we are looking at how we can make it open source.

If you do not like Github or do not want to create an account there, you can also write to hello@openstreetview.org with your ideas or bug reports. We are also on Twitter as @openstreetview and we are getting on Facebook and Instagram if you're into that kind of thing.

If you check Github, you will see that we also have upload tools for your existing Virb / GoPro and other action camera images. These are Python scripts, but we also have a GUI tool that you can just drag and drop directories onto to upload. This is in early beta but if you want a copy, let me know.

guiuploadtool

There are two components that I have not mentioned yet: the Map Editor and the back end. I want to save those for a separate post that I will write soon. Here is a screenshot of the map editor:

mapeditor

Why OpenStreetView?

We think that OpenStreetView is the #1 choice for street level imagery for OSM. Not only because it is almost completely open source. You also remain in full control of the data you upload to OSV. You can always delete individual photos, trips or even delete everything and remove your account if at any moment you don't want to be a part of OSV any longer. This option is on the web site, no need to email anyone or submit a request.

Another reason is because we are building a platform that is very tightly integrated with OSM. This is obvious from the way we handle user accounts: log in with OSM, no separate account. But also deeper down the integration with OSM is tight: we map all trips to OSM ways, so we can link back and forth between trips / images and OSM way objects. This opens up all kinds of interesting possibilities, and I want to spend a separate post on that as well.

For now, it would be cool if you would give OpenStreetView a try. Download the app for your phone, sign in and start capturing. Tell us about your experiences. Explore what is already there. And most important please use it to improve OSM!

Note that OpenStreetView is not a project run by OpenStreetMap or the OpenStreetMap Foundation. It is maintained by Telenav, where I work, for the sole benefit of improving OpenStreetMap.

Mapping pedestrian crossings

Posted by mvexel on 10 August 2016 in English (English)

I am on a roll mapping pedestrian crossings (or 'crosswalks' as Americans tend to call them.

First I download a sliver of the map that covers a major road in JOSM:

sliver

I think you could also use 'download along way' in JOSM if the road is not nice and straight, but around here they usually are.

Then I pan along the way and add crossing nodes using Shift-R to quickly copy pedestrian crossing tags from the previous node.

crosswalk-action

This way I can add about 15 pedestrian crossings a minute.

A MapRoulette Update: V2, Pedestrian Safety, GeoJSON, and Floating Ways in China

Posted by mvexel on 15 March 2016 in English (English)

I have been on a bit of a MapRoulette binge lately. MapRoulette 2 is coming along nicely and we are at the point where we can start working on the front end. This is where a lot of your suggestions come in. If you have more ideas about how MapRoulette should (not) work, please take a moment to go to that PiratePad and add them. Thanks!

MapRoulette 1 is still very much alive however! Let's look at what has been happening.

Pedestrian Safety challenges

Last week, I posted new Sidewalk Mapping challenges (Tampa, Salt Lake City, your city?) to help OSM become a better map for getting around as a pedestrian safely in cities in the United States. Given that someone on foot on U.S. streets was hit by a car about every 8 minutes in the past decade, we could use better maps to help prevent accidents.

Speaking of pedestrian safety! Dr. Stefan Keller, a long time OSM enthusiast and founder of the Geometalab at the Hochschule für Technik, Rapperswil launched a really cool initiative to detect missing crosswalks based on analysis of both OSM data and aerial imagery. The results are making their way to MapRoulette, too.

missing-sidewalks

A missing sidewalk example identified by Stefan Keller's work

GeoJSON

While I was researching the sidewalks issue, I figured I needed an easier way to create challenges from an Overpass query. I decided to use existing components as much as possible and came up with geojson2maproulette, a Python script that leverages the MapRoulette Python API wrapper and the GeoJSON export functionality from Overpass Turbo to make this a pretty easy process. I think it is useful, and I hope you do too. Definitely get in touch if you need help with it or have ideas for improvements.

geojson2maproulette-run

Floating Ways in China

The latest addition to MapRoulette is a bunch of 'floating ways' I happened upon when looking at OSM data in China. To see what I mean by 'floating ways', here is a simple example:

floatingway

What appears to happen a lot when folks are mapping in China is that they trace a bit of road roughly from aerial imagery available, but they do not connect the road to the rest of the network. Although I believe that some (correct) data is better than no data at all, I thought it was worth highlighting these ways. I found more than 9000 in my analysis. If you want to do some mapping in China, this is a great way to get started!

Stay tuned to my diary and follow @maproulette on Twitter if you want to stay informed about what is happening in MapRoulette land! We also have a mailing list and a Slack channel, by the way.

ImproveOSM with your own GPS data - a Field Report

Posted by mvexel on 14 March 2016 in English (English)

This diary also appears on the ImproveOSM blog. Follow ImproveOSM there or on Twitter to stay informed of everything we do with ImproveOSM.

See also Wille's post about this (in Portuguese)

We launched ImproveOSM about 6 months ago as a way to turn the vast amounts of GPS data that Scout users give us into useful and actionable hints mappers can use to add turn restrictions, missing roads as well as wrong or missing one-way streets. The response has been incredible -- since we launched, more than 26 thousand hints have been processed, leading to more than 16 thousand improvements to the map worldwide. I think that is a fantastic result, and we will keep working to make ImproveOSM better based on your feedback.

Initially, we just used our own GPS data to generate the hints. But there is no reason why we couldn't process any GPS data we can get from other sources. So I was really excited when long time Brazil mapper Wille Marcel got in touch with a cool idea. He worked with the Brazilian Environment Ministry, which collects GPS data of the vehicles that work in environmental monitoring. Most of the data are in rural areas where OSM is much less complete. So this was a perfect fit for ImproveOSM's missing roads tool.

After getting the proper permissions from the agency, Wille sent us the GPS data and we started analyzing it.

sparse-missing-roads

We quickly realized that the GPS data is much less dense than what we are used to working with. Some missing roads were only driven once. Our algorithm, tuned to higher density data, initially only detected a few tiles. We decided to loosen the detection threshold significantly for this particular dataset. After a few iterations of tweaking and testing, we ended up with more than 5000 tiles containing missing roads based on Wille's GPS data.

overview

The missing roads in Brazil are on ImproveOSM now, so why not go to the web site or fire up the ImproveOSM JOSM plugin and help the Brazilian community out by adding some missing roads?

If you are in a similar position as Wille and know of a source of free and open GPS data for your country, please get in touch with me so we can look at the data and see if we can include it in ImproveOSM.

We are already working with a number of other folks who have lots of GPS data. Soon, the number of missing roads, one-ways, and turn restrictions in ImproveOSM will be much, much bigger. We are also working on a host of new features, so I hope you will stay tuned to the ImproveOSM blog to be among the first to hear about what we have up our sleeves for ImproveOSM and other OSM related projects we are working on. And follow us on Twitter at @ImproveOSM!

Help map some sidewalks for cities in the U.S.

Posted by mvexel on 10 March 2016 in English (English)

This post also appears on the ImproveOSM blog

United States cities are built for cars, with very few exceptions. From where I am sitting right now, I see this:

carscarscars

Cars zooming by incessantly at 70kph.

Finding your way in an urban space that is designed this way is tricky - and often dangerous - if you are walking or bicycling. Sidewalks are often not present, crossing streets can be very dangerous or even impossible. OSM has great tagging for bike lanes and sidewalks, but I find that these crucial tags are often missing on ways that need them most: the four or six lane urban arterials that you see in the picture above.

As I was sitting here asking myself how on earth I would get back to my hotel (which is 10 minutes away) safely, I thought to myself: 'we can fix this problem and make the world a bit safer for those who can't or won't drive.'

MapRoulette to the rescue!

I created this challenge highlighting all primary and secondary ways that have no sidewalk tag in Tampa, Florida. (I am actually in Sarasota now, south of Tampa, but I already fixed all the ways there so that would be a boring challenge.) The idea is to look at the aerial image in JOSM or iD, see if there is a sidewalk, and add the appropriate tag. Adding sidewalk=no is actually just as important as adding both, right or left. Here is an example way from this challenge:

exampleway

Even zooming further in there is no sight of a sidewalk:

zoomedin

So let's add that information:

tagging

And upload!

Create a Challenge for your city

The fun part is that you can easily replicate this challenge for your own city. Here's what to do.

Overpass Turbo

First you head over to Overpass Turbo and run the query that highlights all highway=primary and highway=secondary that have no sidewalk tag:

overpassturbo

You can use my query as a template, replacing the GeocodeArea with the name of your city.

Once you have the results, export them to GeoJSON. Let's use a gist:

gist1

gist2

You can now click on the gist link and see the result on GitHub as well:

github

We will need the 'raw' GeoJSON content, so click on the 'Raw' button and copy the link it leads you to.

Geojson2MapRoulette

Next we'll use a little tool I created to easily turn the contents of a GeoJSON file into a MapRoulette challenge. To get it, head over to the Github repository and follow the instructions to install the tool.

The tool takes its configuration from a YAML file. The samples directory contains an example for this sidewalks challenge you can use as a template:

# the base URL for the MapRoulette server API to call
server: http://dev.maproulette.org/api
#server: "http://localhost:5000/api"
#server: http://maproulette.org/api

# server API admin credentials
user: devuser
password: mylittlesony

# source file or URL. You can give a list of URLs too, all data will be gathered and added to the same challenge.
source_url: https://gist.githubusercontent.com/anonymous/310005dc1dcf08f5c4a7/raw/74aa0d6a845a20511d0c450d715dab23d2a6c0d6/overpass.geojson
# source_file: ....

# source geojson property key to use as your task identifier (optional, will use random UUID if not given)
# identifier_property = ...

# Challenge metadata, see https://gist.github.com/mvexel/b5ad1cb0c91ac245ea3f for background
slug: sidewalks-sarasota
title: Add sidewalks to major roads in Sarasota
instruction: This way has no `sidewalk` tag. Usually you can see from the aerial imagery if there is a sidewalk or not. Please add the appropriate `sidewalk` tagging.
help: "Help make OSM be a better resource for safe, walkable streets! Many primary and secondary roads in the US are not safe for pedestrians if there is no sidewalk. This challenge highlights all `primary` and `secondary` ways that have no [`sidewalk`](https://wiki.openstreetmap.org/wiki/Key:sidewalk) tagging whatsoever. You can help by looking at aerial imagery and adding the appropriate `sidewalk` tagging. `sidewalk=no` is just as important to have as the 'positive' values. Thanks for helping make OSM better!"

The only items you would need to change are the source_url (use the raw GeoJSON github link you just copied), the slug (use sidewalks-YOURSTATE-YOURCITY or something similar - this will be the challenge URL component in MapRoulette) and the title (change the city name).

By default this configuration will post to dev.maproulette.org. If you want to post to maproulette.org you would need to get in touch with me to get the credentials.

Once you have the YAML config file in order posting to MapRoulette is as simple as:

$ ./geojson2maproulette.py samples/sidewalks-sarasota.yaml --post --activate
Posting 364 tasks...
server alive: True
Updating challenge...
Reconciling tasks...
Done!

Let me know if you need any help with this or if you want me to create a challenge for you!

Cygnus Field Report

Posted by mvexel on 20 January 2016 in English (English)

It has been a few weeks since I wrote about the public beta release of Cygnus, the Telenav conflation engine for OSM data. Since then, I have since been approached by a few folks who wanted to take it for a spin. One of them is long time OSM contributor MikeN. He is preparing an import for Holt and Atchison counties in the U.S. state of Missouri. We worked together on scaling some technical hurdles. Here's a report of what we (well mostly he) did.

Source Data

Mike obtained the source data from Holt and Atchison counties from their official GIS:

I obtained an updated road network from their official GIS, extracted and translated the tags, and followed up with a review against current aerials as well as checking for connectivity, glomming like road segments, and simplifying geometry. The final goal is to obtain permission to import, go through the import process steps, and merge new data onto the existing OSM data.

The next step was to convert the data into the OSM PBF format that Cygnus requires. This is when Mike got in touch with me to work through some technical difficulties:

Since Cygnus required the PBF format, I used Osmosis to convert. This failed because the nodes did not "have a version attribute as OSM 0.6 are required to have". I have learned from Martijn that OsmConvert works without a version attribute, and was able to verify this on my second county.

The next catch was that PBF doesn't accept negative node numbers. The simple workaround is to just use a text editor to remove the minus sign from id='- and ref='- . This seems a bit dangerous - would that file upload if accidentally selected? If so, many low numbered objects would be corrupted around the world. Hopefully, the conversion from OSM to PBF can be moved to the Cygnus chain so that it can accept zipped OSM since most users will start with .OSM data.

That is a great suggestion. On the one hand, we don't want to make Cygnus too easy to use. (Cygnus is in the end a tool to help with imports. It should never be easy to just import data in OSM. There are strict guidelines, and any tool to help with imports should make the user consider the process very carfully.) On the other hand, handling the conversion from JOSM XML (including the negative IDs) to valid PBF is a mechanical step that most any Cygnus user would need to perform, so I would like to include that in a future version.

With that out of the way, we worked together to produce the Cygnus JOSM XML change file.

The result was that it did pick up the new roads and they appear to connect properly into the existing road network. The cases of modified geometry were also detected. And although the node placement was different, the rest of the roads were properly untouched.

It turned out that the number of changed / new roads was fairly minor. A future import would therefore not be too invasive. Here is the OSM base data versus the updates suggested by Cygnus for one of the counties Mike is working on:

cygnus-updates

In total, Cygnus suggested 68 updates. 31 entirely new geometries, and 37 updated geometries. The updated geometries were mostly caused by connecting the new ways to the existing network, adding a node to the existing way where that happens.

Mike is still working with the counties and the community to move this import forward. Working with Cygnus gave some good insights and will hopefully help prepare the actual import when it happens. This is pretty much an ideal use case for Cygnus, and I hope to see more of them.

Mike also has a wish list for Cygnus:

Future enhancements - In my case, I extracted the surface tags from the GIS source. It would be interesting to have more control over tag merging - such as taking surface tags from the 'new' ways if there is no current surface tag. And in the case of renamed roads, to be able to give the new name a priority for the merge.

I already discussed this with my team and this is high on our list of Cygnus improvements - together with support for POI type nodes.

Get in touch with me if you are ready to give Cygnus a try with local data you have!

Improve OSM Improved - and now with Turn Restrictions

Posted by mvexel on 18 December 2015 in English (English)

I am happy to announce that Improve OSM, the suite of open source Telenav tools that help us fix OSM based on vast amounts of GPS data, is completely redesigned and expanded. We made the web site much easier to use. We combined the existing JOSM plugins into one new plugin. And as of today, we have an entirely new category of fixes: missing turn restrictions. I will talk about all of these changes in this diary entry. A lot to cover so let's get started!

The new Improve OSM web site

We redesigned the Improve OSM web site completely. Instead of separate maps for each type of error, all errors are now displayed on one map. You can turn each type on and off in the layer panel. You can also control the filters for each type there.

new-panel

What's even better (I think) is that you can now perform the entire fixing workflow from the web tool, including resolving the error as fixed or invalid. (You used to need the JOSM plugin to change the status on items.)

edit-flow

The new plugin

We released two Improve OSM JOSM Plugins so far: MissingRoads and TrafficFlowDirection. With more error types in the pipeline, it made sense to combine all Improve OSM functionality into one plugin. This new plugin is simply called ImproveOSM. The functionality is largely the same as the previous separate plugins. Each error type still has its own layer. At low zoom levels, the results are clustered:

lowzoom

When you zoom in, individual errors become visible:

highzoom

Please see the diaries I wrote before on the Missing Roads and Traffic Flow Direction layers to learn more about how to work with them.

If you looked closely, you can see that we added a third layer. More about that next.

New Improve OSM type: Turn Restrictions

If you collect a lot of GPS traces like we do, all sorts of interesting patterns appear. You have seen some results with the Missing Roads and Traffic Flow Direction plugins. It gets even more interesting if you consider trip counts over more than one road segment. For example through an intersection. If you consider the relative amount of trips for each possible turn, it is possible to derive which turns are likely prohibited. Compare that against existing OSM data, and voilà - here is our next Improve OSM project!

(In a future diary I want to go into the technical details of turn restriction detection. Right now I will focus on the functionality of the new layer.)

Mapping Turn Restrictions

If you activate the Turn Restriction layer in either the Improve OSM web site or the JOSM plugin, you will encounter suggested turn restrictions like this one here:

example-tr

I like to edit Turn Restrictions in JOSM myself, with the help of the TurnRestrictions plugin, but it can be done in iD as well. So I click 'edit in JOSM' to load the area in JOSM:

josm

You see that I selected the suggested turn restriction from the Improve OSM Turn Restriction layer. You can also see that I loaded Mapillary data using the Mapillary plugin. Street level imagery can be a great help verifying turn restrictions, and in this case the selected Mapillary image revealed that there is indeed no left turn allowed in this case:

mapillary-image

Image source: Mapillary

While the plugin suggests a no-left-turn restriction, I prefer to map this as an only-straight-on restriction, because that is closest to the actual sign on the road:

trplugin

With the new turn restriction in place I can upload the changeset. I use #improveosm in the changeset comment and I encourage you to do the same:

upload

And finally I can choose to either mark the case as fixed in the web tool or straight in the plugin. I opt for the web tool:

web-tool-close

And that's it!

False positives?

We are still tweaking this algorithm and you may see false positives. Please mark them as 'invalid' if you see them. If you think you see a pattern, don't hesitate to use the feedback links in the plugin or on the web site to let us know. You can also just email me at martijnv@telenav.com.

Sometimes, the algorithm unveils unexpected other map errors as well. Looking at this situation where the algorithm detected a missing left turn restriction:

left-turn-huh

It would seem that this is a viaduct, so there would be no turn to make at all. But it turns out that the bridge and one of the motorway segments passing under it were connected:

connected-bridge

So definitely something you would want to fix!

Happy Holidays and Happy Mapping!

I hope this gives you some interesting things to map over the holidays (in case you celebrate). Let me know what you think! I will be back in the new year with a more detailed post on how we detect turn restrictions. The team will also start looking at any feedback that comes in then.

Missing Roads can now filter paths and water

Posted by mvexel on 14 December 2015 in English (English)

Missing Roads is doing great! Since we launched the plugin a little over two months ago, almost 14 thousand Missing Road tiles were closed.

missingroads

A little over a quarter of those were marked as invalid. We have been looking carefully at those cases, and it turned out the two most occurring invalid tiles were ones involving foot / bike paths and water. So we created filters for those. This should help you use the Missing Roads plugin much more efficiently!

Here is the filter in action for water tiles in the web tool:

filter-water

And in the plugin:

filter-water-josm

Django <3 Overpass API

Posted by mvexel on 13 December 2015 in English (English)

Update After talking to a lot of people about the future of MapRoulette and getting a lot of great feedback, I decided to abandon the Django effort I wrote about here. The New MapRoulette is coming together nicely! See its progress on Github.

I recently started looking into Django, the seminal Python web application framework. I like what I see much more that I thought I would, so much so that I am starting to rewrite MapRoulette as a Django application.

One thing I want to accomplish with a next version of MapRoulette is to make creating challenges much, much easier. For example, you should be able to define a challenge using an Overpass query. (You would be surprised how many annoying errors in OSM can be exposed with a pretty simple Overpass query! For example, here is a query that gives you all highway=tertiary that have no name in my home state Utah - 425 ways.)

So I needed a way for a Django application to access the Overpass API and display and store the results. That should be simple enough to accomplish using the Overpass API Python wrapper project I started a while ago. And it was!

screen

If you are interested in checking it out or trying it for yourself, just clone the Overpass API Python wrapper project. Look for the Django example app in the examples folder. (At the time of writing, the example lives on the branch example that is not merged into master yet.)

Many thanks to all the contributors to the Overpass API Python wrapper project! Let me know if you want to contribute as well, be it code, testing, or documentation. Thanks!

Conflation engine Cygnus now in public beta

Posted by mvexel on 24 November 2015 in English (English)

I wrote about Cygnus, our effort to create an OSM-specific conflation engine, a few months ago. We developed it specifically to aid the import of INEGI road data in Mexico we are preparing together with the community in Mexico. But when used with care, I think it can be very useful in other import scenarios as well. This is why we decided to make it into a web tool anyone can use.

Before I go into details, I should offer a few words of caution.

caution

Firstly, this tool is an early beta. Right now, it only conflates roads, nothing else. We have tested the tool internally, but only on a limited amount of cases, all using open INEGI data from Mexico. With this public beta, I hope to gather more feedback to help us make improvements to it.

Secondly, using this tool requires careful planning and preparation. It is definitely not for casual users. It is useful mostly in import scenarios, so the usual extreme caution and preparation related to data imports apply as well. Anyone considering any import needs to take this warning on the imports wiki page very seriously:

import-warning

If any of what you read below feels confusing or difficult, you should probably not be using Cygnus or attempting a data import in the first place. I do not mean to be discouraging, but importing data into OSM is hard, and you can easily destroy other people's work. So if you become aware of an external dataset that you think would be interesting to incorporate into OSM, study the import guidelines, talk to your local community and discuss the best way forward. Never go at this alone.

What does Cygnus do?

Conflating in GIS is the act of merging two data layers to create one layer containing the features and attributes of both original layers. (A more official sounding definition can be found here.) Cygnus is a tool that conflates external data with OSM. You feed it an external dataset, and Cygnus will compare it against current OSM data. It will give you a result file in JOSM XML format with all the changes. You can load this change file into JOSM and merge it with an OSM data layer. The result could - with extreme caution! see above! - be uploaded to OSM.

Cygnus does its conflation in a non-destructive way. No existing OSM ways are ever deleted or degraded. Existing geometries do get changed where new connections need to be made. Where new and existing ways overlap, Cygnus will honor the original way geometry and attributes.

Let's look at an example so I can go into much more detail.

An INEGI example: Los Morales, Mexico

For this example, I picked the local road data for Los Morales, a hamlet north of the city of Monterrey, Mexico. Looking at Geofabrik's Map Compare, this hamlet is all but non-existent on OSM:

losmorales-geofabrik

Data Source

Mexico has a huge open data initiative, and I wrote about the data from the national Census bureau, INEGI, previously. The data can be found on the INEGI web site as part of the dataset 'Información Vectorial de Localidades Amanzanadas y Números Exteriores' for the administrative region of Salinas Victoria. (More background for the various INEGI road datasets is being compiled on the OpenStreetMap wiki by my colleague Andres.)

Translation

Before we can even start to think about conflation, we need to ensure a proper attibute translation. I purposely picked a fairly uncomplicated example so we can remain focused on the process as a whole. The attributes for this dataset are fairly straightforward:

attributes

I created an OSM file from the data using ogr2osm with a custom, simplified translation file. Because Cygnus requires PBF input, I finally converted the OSM file to PBF using osmosis.

I want to discuss the translation of INEGI data specifically in another blog post in the future. We are working with INEGI to get as much data as we can to ensure a proper mapping from INEGI road types to OSM way types. What I am doing here is a much simplified example for the sake of this demonstration. The result will not actually be uploaded to OSM.

Upload to Cygnus

Now that we have an input file, we can offer it to Cygnus for processing. When you load the Cygnus service page, you see this simple interface:

cygnus-home

There are just two pages: the home page where you add new jobs, and the Job Queue page where you can see your progress and download the result. To add a job to the Cygnus queue, I upload the file I have prepared, and add it to the job queue:

upload

Note that your upload needs to be small-ish: the spatial extent needs to be smaller than 50x50km and the file needs to be 20MB or smaller in size.

Cygnus process

If your input file was uploaded successfully, Cygnus will go to work. Your job will be added to the back of the queue. When it's your turn, Cygnus will read your PBF file, and download the OSM data for the same extent. This is done using the Overpass API. It will then compare your upload with the existing OSM data, and produce the ouptut file. I will spend a separate post on more details about the Cygnus process. For now, the most important thing to remember is that Cygnus will only consider roads. This includes most highway ways in OSM. Any other data will be ignored.

When Cygnus is done processing your file, it will be available to download in the job queue.

queue

You can download and/or remove your file here. Everyone's jobs are visible here, so please be careful not to touch other users' stuff.

Inspect and process in JOSM

The downloaded file is plain JOSM XML:

josm-result

What you see here are the differences between OSM and your uploaded file. This includes new ways added, ways with changed geometries and ways with new tags added. Next we need to inspect the changes carefully against existing OSM data. Cygnus is set to conflate very conservatively by default. The results surely will need manual tweaking.

This is by far the most important, and time consuming, step!

So I load the OSM data in JOSM for the same extent. First, I use the layer panel to get a quick overview of what has been added or changed:

switch

Because there was already a highway=secondary, it is probably a good idea to pay close attention to the data there. While Cygnus does a best effort to connect ways where needed, it acts conservatively so it will not snap ways together that do not belong together.

Here are a few ways that got properly connected to the existing highway=secondary:

good-conflate

But here the distance was too far so Cygnus did not snap:

toofar

In this case, you would need to manually connect the ways if that is appropriate.

You can also inspect what Cygnus proposes by selecting any way of the Cygnus layer and looking for the telenav:graphenhancer tag. This will have the value new for added ways, and changed:geometry for ways that have geometry changes, for example.

(The quality of the result will not only depend on what Cygnus does. 'Garbage In, Garbage Out' also applies. So before you even offer your file to Cygnus for conflating, make sure you have triple / quadruple-checked your atribute translation tables and other pre-processing steps.)

When you are finally satisfied with your manually post-processed conflation result, you can go ahead and merge it with the OSM data in JOSM:

merge

When that is done, you probably want to remove the telenav:graphenhancer tags:

remove

remove-2

After that, the data should be close to ready to be uploaded to OSM. In this case, I am not going to do this because I did not follow import procedures at all, and I wrote a quick, simplified translation file for the attributes. So Los Morales is still just as sadly absent from OSM as it was before. But I hope this will give you an idea of what Cygnus is capable of.

If you have existing import plans that involve road network, and you would like to take Cygnus for a spin, please do. I am happy to help. Email me, ping me on twitter or Skype (mvexel). I am looking forward to hearing from you!

Older Entries | Newer Entries