Recent diary entries
I have been on a bit of a MapRoulette binge lately. MapRoulette 2 is coming along nicely and we are at the point where we can start working on the front end. This is where a lot of your suggestions come in. If you have more ideas about how MapRoulette should (not) work, please take a moment to go to that PiratePad and add them. Thanks!
MapRoulette 1 is still very much alive however! Let's look at what has been happening.
Pedestrian Safety challenges
Last week, I posted new Sidewalk Mapping challenges (Tampa, Salt Lake City, your city?) to help OSM become a better map for getting around as a pedestrian safely in cities in the United States. Given that someone on foot on U.S. streets was hit by a car about every 8 minutes in the past decade, we could use better maps to help prevent accidents.
Speaking of pedestrian safety! Dr. Stefan Keller, a long time OSM enthusiast and founder of the Geometalab at the Hochschule für Technik, Rapperswil launched a really cool initiative to detect missing crosswalks based on analysis of both OSM data and aerial imagery. The results are making their way to MapRoulette, too.
A missing sidewalk example identified by Stefan Keller's work
While I was researching the sidewalks issue, I figured I needed an easier way to create challenges from an Overpass query. I decided to use existing components as much as possible and came up with
geojson2maproulette, a Python script that leverages the MapRoulette Python API wrapper and the GeoJSON export functionality from Overpass Turbo to make this a pretty easy process. I think it is useful, and I hope you do too. Definitely get in touch if you need help with it or have ideas for improvements.
Floating Ways in China
The latest addition to MapRoulette is a bunch of 'floating ways' I happened upon when looking at OSM data in China. To see what I mean by 'floating ways', here is a simple example:
What appears to happen a lot when folks are mapping in China is that they trace a bit of road roughly from aerial imagery available, but they do not connect the road to the rest of the network. Although I believe that some (correct) data is better than no data at all, I thought it was worth highlighting these ways. I found more than 9000 in my analysis. If you want to do some mapping in China, this is a great way to get started!
See also Wille's post about this (in Portuguese)
We launched ImproveOSM about 6 months ago as a way to turn the vast amounts of GPS data that Scout users give us into useful and actionable hints mappers can use to add turn restrictions, missing roads as well as wrong or missing one-way streets. The response has been incredible -- since we launched, more than 26 thousand hints have been processed, leading to more than 16 thousand improvements to the map worldwide. I think that is a fantastic result, and we will keep working to make ImproveOSM better based on your feedback.
Initially, we just used our own GPS data to generate the hints. But there is no reason why we couldn't process any GPS data we can get from other sources. So I was really excited when long time Brazil mapper Wille Marcel got in touch with a cool idea. He worked with the Brazilian Environment Ministry, which collects GPS data of the vehicles that work in environmental monitoring. Most of the data are in rural areas where OSM is much less complete. So this was a perfect fit for ImproveOSM's missing roads tool.
After getting the proper permissions from the agency, Wille sent us the GPS data and we started analyzing it.
We quickly realized that the GPS data is much less dense than what we are used to working with. Some missing roads were only driven once. Our algorithm, tuned to higher density data, initially only detected a few tiles. We decided to loosen the detection threshold significantly for this particular dataset. After a few iterations of tweaking and testing, we ended up with more than 5000 tiles containing missing roads based on Wille's GPS data.
The missing roads in Brazil are on ImproveOSM now, so why not go to the web site or fire up the ImproveOSM JOSM plugin and help the Brazilian community out by adding some missing roads?
If you are in a similar position as Wille and know of a source of free and open GPS data for your country, please get in touch with me so we can look at the data and see if we can include it in ImproveOSM.
We are already working with a number of other folks who have lots of GPS data. Soon, the number of missing roads, one-ways, and turn restrictions in ImproveOSM will be much, much bigger. We are also working on a host of new features, so I hope you will stay tuned to the ImproveOSM blog to be among the first to hear about what we have up our sleeves for ImproveOSM and other OSM related projects we are working on. And follow us on Twitter at @ImproveOSM!
This post also appears on the ImproveOSM blog
United States cities are built for cars, with very few exceptions. From where I am sitting right now, I see this:
Cars zooming by incessantly at 70kph.
Finding your way in an urban space that is designed this way is tricky - and often dangerous - if you are walking or bicycling. Sidewalks are often not present, crossing streets can be very dangerous or even impossible. OSM has great tagging for bike lanes and sidewalks, but I find that these crucial tags are often missing on ways that need them most: the four or six lane urban arterials that you see in the picture above.
As I was sitting here asking myself how on earth I would get back to my hotel (which is 10 minutes away) safely, I thought to myself: 'we can fix this problem and make the world a bit safer for those who can't or won't drive.'
MapRoulette to the rescue!
I created this challenge highlighting all
secondary ways that have no
sidewalk tag in Tampa, Florida. (I am actually in Sarasota now, south of Tampa, but I already fixed all the ways there so that would be a boring challenge.) The idea is to look at the aerial image in JOSM or iD, see if there is a sidewalk, and add the appropriate tag. Adding
sidewalk=no is actually just as important as adding
left. Here is an example way from this challenge:
Even zooming further in there is no sight of a sidewalk:
So let's add that information:
Create a Challenge for your city
The fun part is that you can easily replicate this challenge for your own city. Here's what to do.
First you head over to Overpass Turbo and run the query that highlights all
highway=secondary that have no
You can use my query as a template, replacing the GeocodeArea with the name of your city.
Once you have the results, export them to GeoJSON. Let's use a gist:
You can now click on the gist link and see the result on GitHub as well:
We will need the 'raw' GeoJSON content, so click on the 'Raw' button and copy the link it leads you to.
Next we'll use a little tool I created to easily turn the contents of a GeoJSON file into a MapRoulette challenge. To get it, head over to the Github repository and follow the instructions to install the tool.
The tool takes its configuration from a YAML file. The samples directory contains an example for this sidewalks challenge you can use as a template:
# the base URL for the MapRoulette server API to call server: http://dev.maproulette.org/api #server: "http://localhost:5000/api" #server: http://maproulette.org/api # server API admin credentials user: devuser password: mylittlesony # source file or URL. You can give a list of URLs too, all data will be gathered and added to the same challenge. source_url: https://gist.githubusercontent.com/anonymous/310005dc1dcf08f5c4a7/raw/74aa0d6a845a20511d0c450d715dab23d2a6c0d6/overpass.geojson # source_file: .... # source geojson property key to use as your task identifier (optional, will use random UUID if not given) # identifier_property = ... # Challenge metadata, see https://gist.github.com/mvexel/b5ad1cb0c91ac245ea3f for background slug: sidewalks-sarasota title: Add sidewalks to major roads in Sarasota instruction: This way has no `sidewalk` tag. Usually you can see from the aerial imagery if there is a sidewalk or not. Please add the appropriate `sidewalk` tagging. help: "Help make OSM be a better resource for safe, walkable streets! Many primary and secondary roads in the US are not safe for pedestrians if there is no sidewalk. This challenge highlights all `primary` and `secondary` ways that have no [`sidewalk`](https://wiki.openstreetmap.org/wiki/Key:sidewalk) tagging whatsoever. You can help by looking at aerial imagery and adding the appropriate `sidewalk` tagging. `sidewalk=no` is just as important to have as the 'positive' values. Thanks for helping make OSM better!"
The only items you would need to change are the
source_url (use the raw GeoJSON github link you just copied), the
sidewalks-YOURSTATE-YOURCITY or something similar - this will be the challenge URL component in MapRoulette) and the
title (change the city name).
Once you have the YAML config file in order posting to MapRoulette is as simple as:
$ ./geojson2maproulette.py samples/sidewalks-sarasota.yaml --post --activate Posting 364 tasks... server alive: True Updating challenge... Reconciling tasks... Done!
Let me know if you need any help with this or if you want me to create a challenge for you!
It has been a few weeks since I wrote about the public beta release of Cygnus, the Telenav conflation engine for OSM data. Since then, I have since been approached by a few folks who wanted to take it for a spin. One of them is long time OSM contributor MikeN. He is preparing an import for Holt and Atchison counties in the U.S. state of Missouri. We worked together on scaling some technical hurdles. Here's a report of what we (well mostly he) did.
Mike obtained the source data from Holt and Atchison counties from their official GIS:
I obtained an updated road network from their official GIS, extracted and translated the tags, and followed up with a review against current aerials as well as checking for connectivity, glomming like road segments, and simplifying geometry. The final goal is to obtain permission to import, go through the import process steps, and merge new data onto the existing OSM data.
The next step was to convert the data into the OSM PBF format that Cygnus requires. This is when Mike got in touch with me to work through some technical difficulties:
Since Cygnus required the PBF format, I used Osmosis to convert. This failed because the nodes did not "have a version attribute as OSM 0.6 are required to have". I have learned from Martijn that OsmConvert works without a version attribute, and was able to verify this on my second county.
The next catch was that PBF doesn't accept negative node numbers. The simple workaround is to just use a text editor to remove the minus sign from
ref='-. This seems a bit dangerous - would that file upload if accidentally selected? If so, many low numbered objects would be corrupted around the world. Hopefully, the conversion from OSM to PBF can be moved to the Cygnus chain so that it can accept zipped OSM since most users will start with .OSM data.
That is a great suggestion. On the one hand, we don't want to make Cygnus too easy to use. (Cygnus is in the end a tool to help with imports. It should never be easy to just import data in OSM. There are strict guidelines, and any tool to help with imports should make the user consider the process very carfully.) On the other hand, handling the conversion from JOSM XML (including the negative IDs) to valid PBF is a mechanical step that most any Cygnus user would need to perform, so I would like to include that in a future version.
With that out of the way, we worked together to produce the Cygnus JOSM XML change file.
The result was that it did pick up the new roads and they appear to connect properly into the existing road network. The cases of modified geometry were also detected. And although the node placement was different, the rest of the roads were properly untouched.
It turned out that the number of changed / new roads was fairly minor. A future import would therefore not be too invasive. Here is the OSM base data versus the updates suggested by Cygnus for one of the counties Mike is working on:
In total, Cygnus suggested 68 updates. 31 entirely new geometries, and 37 updated geometries. The updated geometries were mostly caused by connecting the new ways to the existing network, adding a node to the existing way where that happens.
Mike is still working with the counties and the community to move this import forward. Working with Cygnus gave some good insights and will hopefully help prepare the actual import when it happens. This is pretty much an ideal use case for Cygnus, and I hope to see more of them.
Mike also has a wish list for Cygnus:
Future enhancements - In my case, I extracted the surface tags from the GIS source. It would be interesting to have more control over tag merging - such as taking surface tags from the 'new' ways if there is no current surface tag. And in the case of renamed roads, to be able to give the new name a priority for the merge.
I already discussed this with my team and this is high on our list of Cygnus improvements - together with support for POI type nodes.
Get in touch with me if you are ready to give Cygnus a try with local data you have!
I am happy to announce that Improve OSM, the suite of open source Telenav tools that help us fix OSM based on vast amounts of GPS data, is completely redesigned and expanded. We made the web site much easier to use. We combined the existing JOSM plugins into one new plugin. And as of today, we have an entirely new category of fixes: missing turn restrictions. I will talk about all of these changes in this diary entry. A lot to cover so let's get started!
The new Improve OSM web site
We redesigned the Improve OSM web site completely. Instead of separate maps for each type of error, all errors are now displayed on one map. You can turn each type on and off in the layer panel. You can also control the filters for each type there.
What's even better (I think) is that you can now perform the entire fixing workflow from the web tool, including resolving the error as fixed or invalid. (You used to need the JOSM plugin to change the status on items.)
The new plugin
We released two Improve OSM JOSM Plugins so far: MissingRoads and TrafficFlowDirection. With more error types in the pipeline, it made sense to combine all Improve OSM functionality into one plugin. This new plugin is simply called ImproveOSM. The functionality is largely the same as the previous separate plugins. Each error type still has its own layer. At low zoom levels, the results are clustered:
When you zoom in, individual errors become visible:
If you looked closely, you can see that we added a third layer. More about that next.
New Improve OSM type: Turn Restrictions
If you collect a lot of GPS traces like we do, all sorts of interesting patterns appear. You have seen some results with the Missing Roads and Traffic Flow Direction plugins. It gets even more interesting if you consider trip counts over more than one road segment. For example through an intersection. If you consider the relative amount of trips for each possible turn, it is possible to derive which turns are likely prohibited. Compare that against existing OSM data, and voilà - here is our next Improve OSM project!
(In a future diary I want to go into the technical details of turn restriction detection. Right now I will focus on the functionality of the new layer.)
Mapping Turn Restrictions
If you activate the Turn Restriction layer in either the Improve OSM web site or the JOSM plugin, you will encounter suggested turn restrictions like this one here:
I like to edit Turn Restrictions in JOSM myself, with the help of the TurnRestrictions plugin, but it can be done in iD as well. So I click 'edit in JOSM' to load the area in JOSM:
You see that I selected the suggested turn restriction from the Improve OSM Turn Restriction layer. You can also see that I loaded Mapillary data using the Mapillary plugin. Street level imagery can be a great help verifying turn restrictions, and in this case the selected Mapillary image revealed that there is indeed no left turn allowed in this case:
Image source: Mapillary
While the plugin suggests a no-left-turn restriction, I prefer to map this as an only-straight-on restriction, because that is closest to the actual sign on the road:
With the new turn restriction in place I can upload the changeset. I use #improveosm in the changeset comment and I encourage you to do the same:
And finally I can choose to either mark the case as fixed in the web tool or straight in the plugin. I opt for the web tool:
And that's it!
We are still tweaking this algorithm and you may see false positives. Please mark them as 'invalid' if you see them. If you think you see a pattern, don't hesitate to use the feedback links in the plugin or on the web site to let us know. You can also just email me at firstname.lastname@example.org.
Sometimes, the algorithm unveils unexpected other map errors as well. Looking at this situation where the algorithm detected a missing left turn restriction:
It would seem that this is a viaduct, so there would be no turn to make at all. But it turns out that the bridge and one of the motorway segments passing under it were connected:
So definitely something you would want to fix!
Happy Holidays and Happy Mapping!
I hope this gives you some interesting things to map over the holidays (in case you celebrate). Let me know what you think! I will be back in the new year with a more detailed post on how we detect turn restrictions. The team will also start looking at any feedback that comes in then.
Missing Roads is doing great! Since we launched the plugin a little over two months ago, almost 14 thousand Missing Road tiles were closed.
A little over a quarter of those were marked as invalid. We have been looking carefully at those cases, and it turned out the two most occurring invalid tiles were ones involving foot / bike paths and water. So we created filters for those. This should help you use the Missing Roads plugin much more efficiently!
Here is the filter in action for water tiles in the web tool:
And in the plugin:
Update After talking to a lot of people about the future of MapRoulette and getting a lot of great feedback, I decided to abandon the Django effort I wrote about here. The New MapRoulette is coming together nicely! See its progress on Github.
I recently started looking into Django, the seminal Python web application framework. I like what I see much more that I thought I would, so much so that I am starting to rewrite MapRoulette as a Django application.
One thing I want to accomplish with a next version of MapRoulette is to make creating challenges much, much easier. For example, you should be able to define a challenge using an Overpass query. (You would be surprised how many annoying errors in OSM can be exposed with a pretty simple Overpass query! For example, here is a query that gives you all
highway=tertiary that have no name in my home state Utah - 425 ways.)
So I needed a way for a Django application to access the Overpass API and display and store the results. That should be simple enough to accomplish using the Overpass API Python wrapper project I started a while ago. And it was!
If you are interested in checking it out or trying it for yourself, just clone the Overpass API Python wrapper project. Look for the Django example app in the
examples folder. (At the time of writing, the example lives on the branch
example that is not merged into master yet.)
Many thanks to all the contributors to the Overpass API Python wrapper project! Let me know if you want to contribute as well, be it code, testing, or documentation. Thanks!
I wrote about Cygnus, our effort to create an OSM-specific conflation engine, a few months ago. We developed it specifically to aid the import of INEGI road data in Mexico we are preparing together with the community in Mexico. But when used with care, I think it can be very useful in other import scenarios as well. This is why we decided to make it into a web tool anyone can use.
Before I go into details, I should offer a few words of caution.
Firstly, this tool is an early beta. Right now, it only conflates roads, nothing else. We have tested the tool internally, but only on a limited amount of cases, all using open INEGI data from Mexico. With this public beta, I hope to gather more feedback to help us make improvements to it.
Secondly, using this tool requires careful planning and preparation. It is definitely not for casual users. It is useful mostly in import scenarios, so the usual extreme caution and preparation related to data imports apply as well. Anyone considering any import needs to take this warning on the imports wiki page very seriously:
If any of what you read below feels confusing or difficult, you should probably not be using Cygnus or attempting a data import in the first place. I do not mean to be discouraging, but importing data into OSM is hard, and you can easily destroy other people's work. So if you become aware of an external dataset that you think would be interesting to incorporate into OSM, study the import guidelines, talk to your local community and discuss the best way forward. Never go at this alone.
What does Cygnus do?
Conflating in GIS is the act of merging two data layers to create one layer containing the features and attributes of both original layers. (A more official sounding definition can be found here.) Cygnus is a tool that conflates external data with OSM. You feed it an external dataset, and Cygnus will compare it against current OSM data. It will give you a result file in JOSM XML format with all the changes. You can load this change file into JOSM and merge it with an OSM data layer. The result could - with extreme caution! see above! - be uploaded to OSM.
Cygnus does its conflation in a non-destructive way. No existing OSM ways are ever deleted or degraded. Existing geometries do get changed where new connections need to be made. Where new and existing ways overlap, Cygnus will honor the original way geometry and attributes.
Let's look at an example so I can go into much more detail.
An INEGI example: Los Morales, Mexico
For this example, I picked the local road data for Los Morales, a hamlet north of the city of Monterrey, Mexico. Looking at Geofabrik's Map Compare, this hamlet is all but non-existent on OSM:
Mexico has a huge open data initiative, and I wrote about the data from the national Census bureau, INEGI, previously. The data can be found on the INEGI web site as part of the dataset 'Información Vectorial de Localidades Amanzanadas y Números Exteriores' for the administrative region of Salinas Victoria. (More background for the various INEGI road datasets is being compiled on the OpenStreetMap wiki by my colleague Andres.)
Before we can even start to think about conflation, we need to ensure a proper attibute translation. I purposely picked a fairly uncomplicated example so we can remain focused on the process as a whole. The attributes for this dataset are fairly straightforward:
I want to discuss the translation of INEGI data specifically in another blog post in the future. We are working with INEGI to get as much data as we can to ensure a proper mapping from INEGI road types to OSM way types. What I am doing here is a much simplified example for the sake of this demonstration. The result will not actually be uploaded to OSM.
Upload to Cygnus
Now that we have an input file, we can offer it to Cygnus for processing. When you load the Cygnus service page, you see this simple interface:
There are just two pages: the home page where you add new jobs, and the Job Queue page where you can see your progress and download the result. To add a job to the Cygnus queue, I upload the file I have prepared, and add it to the job queue:
Note that your upload needs to be small-ish: the spatial extent needs to be smaller than 50x50km and the file needs to be 20MB or smaller in size.
If your input file was uploaded successfully, Cygnus will go to work. Your job will be added to the back of the queue. When it's your turn, Cygnus will read your PBF file, and download the OSM data for the same extent. This is done using the Overpass API. It will then compare your upload with the existing OSM data, and produce the ouptut file. I will spend a separate post on more details about the Cygnus process. For now, the most important thing to remember is that Cygnus will only consider roads. This includes most
highway ways in OSM. Any other data will be ignored.
When Cygnus is done processing your file, it will be available to download in the job queue.
You can download and/or remove your file here. Everyone's jobs are visible here, so please be careful not to touch other users' stuff.
Inspect and process in JOSM
The downloaded file is plain JOSM XML:
What you see here are the differences between OSM and your uploaded file. This includes new ways added, ways with changed geometries and ways with new tags added. Next we need to inspect the changes carefully against existing OSM data. Cygnus is set to conflate very conservatively by default. The results surely will need manual tweaking.
This is by far the most important, and time consuming, step!
So I load the OSM data in JOSM for the same extent. First, I use the layer panel to get a quick overview of what has been added or changed:
Because there was already a
highway=secondary, it is probably a good idea to pay close attention to the data there. While Cygnus does a best effort to connect ways where needed, it acts conservatively so it will not snap ways together that do not belong together.
Here are a few ways that got properly connected to the existing
But here the distance was too far so Cygnus did not snap:
In this case, you would need to manually connect the ways if that is appropriate.
You can also inspect what Cygnus proposes by selecting any way of the Cygnus layer and looking for the
telenav:graphenhancer tag. This will have the value
new for added ways, and
changed:geometry for ways that have geometry changes, for example.
(The quality of the result will not only depend on what Cygnus does. 'Garbage In, Garbage Out' also applies. So before you even offer your file to Cygnus for conflating, make sure you have triple / quadruple-checked your atribute translation tables and other pre-processing steps.)
When you are finally satisfied with your manually post-processed conflation result, you can go ahead and merge it with the OSM data in JOSM:
When that is done, you probably want to remove the
After that, the data should be close to ready to be uploaded to OSM. In this case, I am not going to do this because I did not follow import procedures at all, and I wrote a quick, simplified translation file for the attributes. So Los Morales is still just as sadly absent from OSM as it was before. But I hope this will give you an idea of what Cygnus is capable of.
If you have existing import plans that involve road network, and you would like to take Cygnus for a spin, please do. I am happy to help. Email me, ping me on twitter or Skype (mvexel). I am looking forward to hearing from you!
You can be more than one kind of geek at the same time, I guess. First and foremost, I am a map geek. This started when I got my first atlas for my 6th birthday. Then came computers, with the arrival of the Commodore 64 I shared with my brothers. Shortly after that came trains. I still have a few thousand slides of stations, trains and railyards in my basement. Also, German and Dutch timetables dating back to 1985.
One of my later railway photos. Berlin Lehrter Stadtbahnhof, summer 1996. This was demolished not too long after to make room for Berlin HBf. Source: Flickr
Roads are a more recent object of geekery for me. Before I moved to the United States, I didn't drive much. As I started driving (and mapping) more in my new home country, I also started to become more interested in the rich history of the vast road network in the United States. And I am not the only one! It doesn't take much online searching to find sites like AARoads and CrossCountryRoads containing 1000s of freeway driving images and videos, as well as Facebook groups dedicated to the celebration of U.S. freeways present and past. (I especially enjoy Freeway Jim for its mix of history and current content. And for Utahns, this Flickr account is pretty cool too.)
I thought it might be interesting to start tapping into this road geek community to help us improve OSM. Even in a well mapped state like Utah cough, most freeway exits still don't have any signpost information except for a
ref. To give you an idea: of 1654 total
motorway_link ways, only 193 have a
destination tag. Even accepting that probably half of these ways are onramps, that's still a big data gap! And one that sites like AAroads and CrossCountryRoads fill perfectly, with images like this one:
(If you're interested, here is the Overpass query for the
motorway_link ways with
After some email exchanges between various US community members and the maintainers of AARoads and CrossCountryRoads, we now have permission to use the images and videos on these sites for mapping. Yay! (I created a wiki page for CrossCountryRoads.com. There is none for AAroads yet, as far as I know.)
I thought I'd try it out myself and share my experiences.
First I pick a region I want to improve. CrossCountryRoads.com has a list of all the Interstates it has coverage for, sorted by state. No Utah, unfortunately :( Let's pick I-68 Westbound in West Virginia. I see a neatly organized list of images, mostly covering exits. Excellent!
Next, I load all
motorway_link ways as well as all
motorway_junction nodes for West Virginia into JOSM. (Perhaps overkill, but I am not an Overpass expert..).
The result looks a little bit like one of those flight tracking web sites :)
The first exit is to the WV Welcome center and a rest area.
I zoom / pan to where I-68 enters West Virginia from Maryland and quickly find the exit, and add the
I also check the
motorway_junction node for any irregularities (such as deprecated
exit_to tags and missing / wrong
I will delete this
exit_to tag. Does anyone know if
noref is a thing?
With the edits done, I upload the changeset. Be sure to attribute the source - in this case CrossCountryRoads:
Then I move on to the next exit and map some more!
I announced our second tool based on Scout GPS data, Traffic Flow Direction, a few days ago. I didn't spend a lot of time on explaining how it works. This post will hopefully make up for that!
The goal of this plugin, and the accompanying web tool, is to make it easy to find and correct OSM ways that we think are missing a
oneway tag, based on billions of GPS points from Scout users. Here is what it looks like in JOSM:
I will walk you through installation, basic operation and some mapping tips in the next paragraphs. Happy mapping!
This is a JOSM plugin so, installation works like any JOSM plugin. Make sure your JOSM is up to date first. Then go to JOSM preferences. Select the Plugins tab and look for the TrafficFlowDirection entry:
Select it and click 'Update Plugins' at the bottom. After JOSM completes installing the plugin (and updating any others that may need updating), restart JOSM and you should see the main components of the plugin appear in the JOSM interface: the overlay on the map and the plugin panel.
You may want to adjust some basic JOSM settings to make optimal use of the plugin as well. The one I would recommend for sure is to enable directional arrows on ways. The direction of the way is crucually important for adding the correct
oneway tagging, so having that direction visible at a glance is really helpful. To enable this, go into JOSM settings and select the Display Settings tab. This is the topmost tab. Within display settings, select the OSM data tab. Here you will find the Segment drawing options:
I recommend selecting both Draw Direction Arrows and Only on head of way. Also make sure Draw oneway arrows below that is selected so you can see at a glance if a way is already tagged with
Like the Missing Roads plugin, the visual components of the Traffic Flow Direction are a map overlay and a settings / information panel. The map overlay shows clusters of suspected errors at low zoom levels, and individual errors at high zoom levels. Here is the colorful scene you get when you enable Missing Roads and Traffic Flow Direction at the same time:
(I recommend hiding the Missing Roads layer when you work on Traffic Flow Direction and vice versa :))
Zoom in on an area you want to work on and you will start seeing individual arrows:
The arrows depict OSM ways that should have a
oneway tag in the indicated direction, according to our GPS data. Click on an arrow to select it and find out more in the Traffic Flow Direction panel:
(If you do not see the panel, you may need to activate from the Window menu.)
The Info tab gives you some basic information on the selected direction error. The most interesting are probably the top two, indicating the % of all drives through that particular way traveling in the indicated direction, and below that the total number of trips that go through that way.
Validating and correcting the errors
Once you are zoomed in to an area where you want to fix some direction errors, make sure you have the best possible aerial imagery layer for your area enabled. Always remember that aerial imagery may be out of date and may not reflect the current reality on the ground!
Now, look for corroborating clues in the aerial image. These may vary by country, but I have found these clues are helpful:
The markings on the road. The suspected one-way street in the image above has a stop line all across the width of the street. Notice how the crossing street has a half stop line indicating that this is a two way street.
Other useful markings are arrows and painted speed limits. Look for these close to intersections.
Another useful clue is the direction of parked cars. In some countries (like the U.K. and most states in the U.S.) it is not legal to park 'against traffic'. So if all cars are parked in one direction, as in the animation above, that gives you a solid clue.
The best clue of course is local knowledge. Start with areas you know well to gain some confidence and experience!
Next, download the data for the area. Before you add
oneway=yes though, check if it applies to the entire way. The OSM way may be longer than the segment we suggest. This has to do with the way we internally process OSM data. We split the ways at each intersection. You may need to split the way before you apply the
(Above) This OSM way may need to be split.
Because we split ways into shorter segments, the opposite may also happen. The
oneway segment may extend beyond what the plugin suggests. This may be because we don't have enough trips through all segments to be sure.
Another thing to watch out for is the directionality of the OSM way. If the direction of the way in OSM (the order of the nodes, basically) is the opposite of the
oneway direction, you either need to tag the way with
oneway=-1 or reverse the direction of the way first.
(Above) The direction of the OSM way (selected, red) is different from the suggested
After you make the
oneway improvements, upload the changeset with
trafficflowdirection in the source.
Finally, mark the issue as Closed. Do this by first making the TrafficFlowDirection layer active and selecting the issue you just solved:
Then click the green lock in the plugin panel, add a comment, and close the issue.
If you find that after inspecting the aerial image or your local knowledge, the way is really not one-way, you can mark the issue as Invalid instead, using the red exclamation mark icon in the plugin panel.
We define three confidence levels for the suspected Traffic Flow Direction errors. These levels are based on different thresholds for the total number of trips and the percentage going of trips going in one direction. Clicking on the 'filter' icon in the Traffic Flow Direction panel will reveal a filter dialog that lets you narrow down the visible errors:
You can also use the filter dialog to switch between Open, Solved and Invalid errors.
You can comment on the selected issue by clicking the blue text bubble icon in the plugin panel:
Perhaps you want to use a different OSM editor. Or you just want to browse around? This is what we created the Traffic Flow Direction Web tool for.
The tool is similar to the Missing Roads web tool. It will show a heatmap at lower zoom levels, and individual errors at higher zoom levels. It allows for filtering by status and confidence level. If you are zoomed in far enough, 'Edit in...' buttons will appear.
I hope you find this useful! I would really like to hear what you think about the tool, this manual, and our efforts to make our data more open. Let me know at email@example.com or comment below.
Thank you and happy mapping!
Last month, we released the Missing Roads plugin and web tool. This quickly became a popular pastime for quite a few mappers - one month later, more than 20% of all the Missing Road tiles have already been resolved :)
For the darker and colder November days -- well, if you are on the Northern hemisphere at least -- we thought we would cook up something new to keep us all busy. We ran another analysis on our GPS data to uncover ways that probably do not have the right directionality. Either they should be
oneway=yes and they are not, or they are
oneway=yes but in the wrong direction.
Here is an example from Karlsruhe, Germany:
The orange arrow points in the direction we think traffic on that street flows based on what we know from our Scout GPS data.
The way we do this is by looking at the directionality of the GPS tracks. If more than, say, 90% of all tracks matched to an OSM way go in a single direction, it is pretty safe to assume that it is a one-way street. We then compare that to how the way is mapped in OSM. If there is no correct
oneway tag on the way, you will see the way in this tool.
Let's open the earlier example up in JOSM:
This looks a bit cluttered because there is so much OSM data, but if we zoom in a bit and make the OSM data layer invisible for a moment, we can see from the direction the cars are parked and the arrows on the road that this is indeed a one-way street:
Aerial imagery in JOSM from Bing
We found more than 140.000 (!) of these cases all over the world waiting for us to fix them. This works almost exactly the same as in our Missing Roads plugin and web tool we released last month. You can look at the Missing Roads manual. (I will write a proper manual for the One-way plugin and tool soon.)
Here is the address for the web tool: http://improve-osm.org/trafficFlowDirection. You can find the JOSM plugin in the JOSM plugin preferences. The source code for the plugin is on Github as always. We started a wiki page for the plugin as well.
Happy mapping and please, as always, let me know what you think :)
I have long wanted to try and do some audio mapping. Especially since I moved to the US and started spending more time in a car.
When you are driving, there is not a lot of ways you can record what you see in a way that makes it easy to map later. One way is to use Mapillary, but the sheer amount of information can be overwhelming. A picture every 5 seconds means 180 images to go through on a short, 15 minute drive. It also means handling over 300 MB of image data. And that's only for a 15 minute drive.
So audio mapping. I have this tiny recorder that weighs almost nothing, has built in space for almost 70 hours (!) of recording and runs weeks on a set of AAA batteries:
So I thought I'd use that. Reading about audio mapping I quickly learned that there are a few strategies. One is to record one long audio track. This way you don't have to push buttons a lot while driving which is nice. Another is to make lots of tiny recordings, one for everything you notice and want to map later.
The second method seemed more appealing to me even if it means pushing the record button a lot while driving. I was envisaging a strategy and result result similar to photo mapping. You record a GPS track as usual, and record audio clips when needed. Once home you would load both into JOSM. You would then see a string of markers along your GPS track. Each marker represents an audio clip. Play - map - play - map!
This is kind of how it turned out, but it turned out I needed to do some thinking and programming.
Hurdles to jump
First, my recorder can only write MP3 files. JOSM will only load WAV files. So I needed a way to easily batch convert the files while preserving the file time stamps.
Second, you have the same offset issues as with photo mapping. Because you use different devices to record the audio and the GPS, there will invariably be some difference between the internal clocks of the two. You need to resolve this before loading GPS and audio clips into JOSM, or the time shift will result in misplaced audio markers. For photo mapping, there is a built in function to adjust this offset in JOSM. For audio clips, there is no such thing.
So I decided to write a small tool that does both these things. In the best tradition of cryptically named OSM tools I named it
mp3towav.py. You can learn more about it at the Github repository. In brief, running this tool on a directory of MP3 files will give you a directory with corresponding WAV files with proper (optionally offset) timestamps.
OSMTracker can also record short audio clips at the press of a button! I used it a lot when I had an Android phone.
The fun part!
With the WAV files handy, all you need to do is load your GPX file into JOSM, and add the audio clips. Curiously this works a little different than with images. The GPX file you just load through the File > Open menu. But then you need to right-click the GPS layer to get access to the 'Import Audio...' function:
You can then select your WAV files, which should show up as markers neatly along your GPS trace:
You can then click on the individual markers and enter the awkward world of listening to your own voice pointing out addresses, businesses, street names and other map-worthy observations.
Once you get in the groove, you can use the audio menu and the shortcut keys F4-F9 to quickly navigate through your clips.
Great, that wasn't too difficult. Let me know how you audio-map, or if this quick walk-through makes you want to try it too!
A few weeks ago, I started something new for the Salt Lake City area OpenStreetMap group: Flash Map Mobs! The idea is to have a group of mappers descend on an under-mapped area and add lots of shops, restaurants etc. in a short amount of time. We do them after work, say from 5-6. Sometimes we get a drink or dinner afterwards.
We have done four so far and I wanted to share some results.
Flash Map Mob 1 - Sugar House
- Meeting Location: Sugar House Coffee - meetup
- Attendees: 3
- Focus: 21st South / Highland Dr area
- POIs added: 88 - Overpass Turbo
The first one! I decided to meet at Sugar House Coffee because I go there a lot. This place can get a bit crowded and loud however. This may turn people off so perhaps not perfect as a meeting place. The Mob itself (then still called 'After Work Map&Meet') was fun. We shared experiences and arrived at Pushpin OSM as the ideal iOS mobile editor for this kind of mapping.
Flash Map Mob 2 - Holladay
- Meeting location: 3 Cups Coffee - meetup
- Attendees: 3
- Focus: Downtown Holladay
- POIs added: 56 - Overpass Turbo
This was our first venture out into the suburbs. Very rewarding, because downtown Holladay has seen a lot of new development in the last couple of years, very little of which had been mapped yet. Met in a coffee shop again, which worked fine this time - this is a large, uncluttered space that is quiet at this time of day (suburbia):
Flash Map Mob 3 - Murray
- Meeting location: The cafe inside Barnes & Noble - meetup
- Attendees: 3
- Focus: Downtown Murray along State Street
- POIs added: 72 - Overpass Turbo
This was perhaps not an ideal location for a Flash Map Mob, but we managed to add over 70 POIs nonetheless. It's just not a very attractive location to walk around - a lot of strip mall type areas. We decided to split up and have some folks drive to different clusters of shops and restaurants. We made it work, but perhaps we should check WalkScore or something similar next time before we settle on a location.
Flash Map Mob 4 - Magna
- Meeting location: Magna Library - meetup
- Attendees: 2
- Focus: Magna Main Street
- POIs added: 62 - Overpass Turbo
The library is a good location to meet. It is public and has great internet. Better than what I have at home! The one downside is that it is supposed to be quiet, so if you want to discuss mapping strategies or meet up after and share experiences, there may be better places. Even though it was just the two of us, this was an interesting Mob, because I had never been to Magna, an old mining town on the western fringe of the Salt Lake City urban area.
Let me know your ideas for how to make the Flash Map Mob more fun / effective! If you want to run your own and don't know where to start, email me at firstname.lastname@example.org or find me on OSM IRC as mvexel and I can give you some pointers.
Last week, we released the Missing Roads JOSM plugin, together with a web tool, the plugin source code, and a manual. More than 700 mappers have already given it a try, and nearly 10% of the missing roads tiles are already resolved. That is fantastic progress!
We have received a lot of great feedback from you and we are looking into every single suggestion. You may have noticed that we have already updated the web app adding a location hash. This means you can now share the URL at any time and it will include the current map view.
We also received a few requests to get access to the data powering the plugin and app. This sounded like a great idea to us! We started publishing daily dumps of the tile data. You can download and use these any way you see fit.
This is what the data looks like:
tileX;tileY;timestamp;numberOfTrips;status;points;type 70009;102823;1443437990.487;5;OPEN;MULTIPOINT(-83.855907 36.129896,-83.85606 36.12981,-83.855953 36.129583,-83.855955 36.129581,-83.85594 36.1296,-83.85592 36.13,-83.855952 36.129584,-83.855907 36.129948,-83.855903 36.129621,-83.85604 36.12973,-83.855935 36.129658,-83.855897 36.130072,-83.855906 36.129732,-83.85598 36.12953,-83.855911 36.129993,-83.855933 36.129608,-83.85595 36.129584,-83.85613 36.13007,-83.85611 36.12993,-83.855914 36.12965,-83.855943 36.129601,-83.85608 36.12987,-83.85592 36.129613,-83.855901 36.130043,-83.855909 36.129836,-83.855952 36.129583,-83.855949 36.129592,-83.85593 36.12932,-83.855904 36.130029,-83.85614 36.13004,-83.8559 36.12995,-83.855946 36.129599,-83.855951 36.129584,-83.855911 36.129781,-83.85614 36.13002,-83.85613 36.12998,-83.855956 36.12958,-83.85601 36.12963,-83.8559 36.1301,-83.85592 36.13005,-83.85611 36.13011);PARKING
Head here to get the daily data dumps: http://improve-osm.org/missingRoadsDumps/
Let me know if you find this useful, or how you would like to see this improved. Email me or go to http://bit.ly/missing-roads-feedback. Thanks!
The most fun I have with OSM is making the map better. Adding nodes and ways and tags. Seeing how my puttering around with JOSM or iD leads to a prettier map that is more useful.
Getting folks together in Salt Lake City where I live to have some OSM fun together. For example. A few weeks ago, I started a new thing called the Flash Map Mob. We do them every other week now in SLC and together we map something like 100 shops and restaurants every time.
The map becomes more useful and folks are having fun. That is what OSM is to me!
I don't need to sit on any board to do these things. Heck, I can probably do more of them when I am not on the board! So why do I do this? What value do I add being on the OSM US Chapter board?
I'll tell you and I'll be brief.
OSM is still very good at adding contributors.
(The graphs come from osmstats.)
What we're not so good at is actually getting folks to go out and map. Look at the flat daily active mappers graph for the US:
This is not a US only problem. But I want to start attacking it where I live.
I don't think putting on big conferences like we have makes more people go out and map. I think we should stop doing that as OSM US.
We should direct that energy towards creating smaller OSM gatherings around the country. Gatherings for mappers. I want us to do 4 of those in the next year. They could be like the fantastic event I attended in Seattle for OSM's 10th anniversary. (More photos). Or something else? We should find out. What does an OSM event look like that you would go to?
We should put local mapping groups first. We should step up funding local groups with mini grants. We should do regular sessions to help people get started organizing their group. Gather more data on what works and what doesn't to help direct our energy. Perhaps it's mapping party kits? Perhaps it's changing the web site to be much more community / mapper oriented? I want us to find out.
I work at Telenav on OSM stuff. So I am in a position where I can get a fairly big organization with lots of talented people to do useful things for mappers. Recently we published Missing Roads. It's stuff like that that gets people mapping. As a board we should encourage companies to enable mappers by having them build useful things and share data.
All of this should lead to us getting from ~250 daily active mappers in the US to 2x that in the next year. This is what I want to accomplish with the board on my next term.
Earlier this week, we released the Missing Roads project. Using its tools, every OSM mapper can now easily find roads that are not yet in OSM, based on our years of collected global GPS data from Scout users.
A lot of you have already had some fun with the JOSM plugin and the web tool. I received interesting reports about important roads you have been able to add. But also some questions about how to optimally make use of the Missing Roads tools. So I thought I might write up a Missing Roads Manual of sorts.
Missing Roads consists of two tools: a web tool and a JOSM plugin.
The Web Tool
The web tool is a convenient way to locate missing roads in an area. You can quickly get a sense of the distribution of the missing roads data.
If you zoom in far enough, you can also see the individual tiles.
Colors of tiles and traces
You will notice that there are different colors for both the traces and the tiles themselves.
The blue tiles haven't been touched yet. These are the 'open' tiles.
The green tiles are the ones that you marked as solved. Yay!
The red tiles are marked as invalid.
Looking at the colors of the traces, you will notice purple, yellow and black trace colors.
The purple color means these are probably actual roads.
The yellow trace color means we think it is people looking for parking.
The black traces are 'mixed': some people were driving faster, some slower, so we are not so sure.
You can filter the results both by tile status and by (probable) type.
The web tool is mainly meant for browsing. You cannot change the status of the tiles from the web tool at this time. This can for now only be done in the JOSM plugin. We do however provide convenient links to edit the current map extent in JOSM and iD.
To quickly check if there are actually roads in the area, you can switch between the default OSM layer and an aerial imagery layer, courtesy of ESRI.
The main interface to the Missing Roads data is our JOSM plugin. It offers a similar browsing functionality as the web tool does, but with a slightly different visualization:
The red dots represent clusters of missing road tiles at lower zooms. When you zoom in you see the actual tiles and the point clouds.
Installation and activation
Install the Missing Roads plugin the familiar way, through the JOSM plugin preference pane. When installed, and after a quick JOSM restart, you should see the MissingRoads layer and panel.
Missing Roads Layer
The layer shows up like any other JOSM layer in the layer panel, and of course on the main map canvas showing you the actual missing road tiles / clusters.
As with any layer in JOSM, it needs to be active if you want to interact with it. So if you want to select tile(s) you will need to activate the MissingRoads layer first.
Missing Roads Panel
In the panel, you can interact with the currently selected tile(s). If you don't see the panel, you should be able to reveal it using ctrl-F3 / cmd-F3.
The panel has three tabs with various bits of information about the selected tile. If you have more than one tile selected, you'll see info about the last tile you selected.
The Tile tab shows basic information about the selected tile.
The History tab shows a history of status changes and comments.
The Have a new idea? tab has a link to the Missing Roads ideas forum. Submit your ideas and bugs there please!
The panel also has a number of action buttons on the bottom. These are for filtering, adding comments, and resolving tiles. I will discuss those features in the next sections!
Similar to the web tool, you can decide which tiles you want to see based on their status and (probable) type.
If you want to clear all filters, you can click Reset.
You can only filter on one status or type at a time.
As a bonus, you can also set a trip count threshold. This allows you to filter out tiles that have a low number of trips passing through them. You can see the number of trips for the selected tile in the Tile tab.
Clicking on the comment button opens the Add Comment dialog allowing you to add a comment to the currently selected tiles for your fellow mappers to see.
If you have multiple tiles selected (using Shift while selecting), the comment will be applied to each tile.
Finally, there are three buttons to resolve the selected tile(s).
The 'lock' button solves a tile and marks it as done.
The 'unlock' button marks the tile(s) as un-done or open again.
The '!' button marks the tile(s) as invalid. Use this if there is not actually a road there.
The tiles and the traces have the same color coding as in the web app. To summarize:
- Blue = Open
- Green = Solved
- Red = Invalid
- Purple = Road
- Yellow = Parking
- Black = Mixed
Obviously you shouldn't add roads solely based on the traces. You need a secondary source of validation. Most of the time, this will be an aerial image. The default aerial layer in JOSM is Bing. Bing imagery can be a few years old. For some regions, more recent imagery may be available. Look for aerial layers in the JOSM imagery menu. Note that most imagery layers are location-aware: they will only appear in the menu if they actually have imagery for the area you are currently mapping. So be sure to check the imagery menu again if you're editing in an unfamiliar area.
Here is an example where Bing shows no sign of a new road, but a local imagery layer does show it:
We have spent a lot of time looking over the results and tweaking the algorithm to get rid of as many irrelevant traces as possible. But we know there are still strange traces out there. I have been getting interesting reports already of traces from trains, airplanes, and even cranes.
Source: Flickr Commons
I know this can be frustrating. If you find strange cases, please take a minute to report them through the feedback page. And mark them as invalid. Thank you!
The code for the plugin can be found on GitHub if you're interested.
There are links in the web tool as well as in the JOSM plugin to give feedback and submit ideas. Both links go to the Missing Roads feedback web site. You can also vote on other mapper's ideas there. I look at all incoming ideas together with my colleagues on our OSM team, so be sure your input is heard and very much appreciated!
Also, don't hesitate to just email me at email@example.com or tweet at me (mvexel). Talk to you soon!
Have fun adding missing roads :)
New roads are built and opened for traffic around the world every single day. In many places, these are added to OSM by watchful mappers right away. Not everywhere though. There are still many places where there are few local mappers, and new construction goes unseen for a while. Available aerial imagery can be pretty outdated, so armchair mappers are not always able to help out either.
Lots of people drive on these new roads from day 1. And we're in luck - a bunch of them are usually Scout users :) This means that they contribute GPS traces of the missing roads to us. Lots. Of. Traces. Enough for us to come up with a good guess about where new roads might be that are not on OSM yet.
We have done just that! I am excited to announce the Missing Roads project, opening up the aggregated, processed GPS data pointing at missing roads as a convenient JOSM plugin.
Here's how it works. First you download the Missing Roads plugin from right within JOSM. After a restart, you will notice an extra Missing Roads layer. It looks like small and big dots if you're zoomed out, and a grid with point clouds if you're zoomed in.
You will notice different color grid cells and point clouds once you're zoomed further in. The grid colors indicate whether someone has marked that cell as completed or invalid. You can do that from within the plugin. If you find tiles that are done or not actually missing roads, please mark them and help your fellow mappers out.
One thing we noticed pretty soon once we started digging into the results is that lots of people leave Scout on when they're looking for parking. This means that lots of 'results' are actually people driving around parking lots. Interesting, but not as interesting as real new roads. So you can filter those out. We distinguish between the two types based mostly on average speed, so it's not always spot on. But it helps.
Finally, we also have a web map where you can easily browse and explore the data, and check for unfinished cells in your region. It links back to
This is version 1. Let us know what you think. How we can improve the plugin and the web app. Other ways you would like to see this data exposed. Email me at firstname.lastname@example.org or talk to me directly on OSM IRC channels, skype (mvexel) or twitter. There's also a little feedback box in the web app where you can submit ideas.
(Source of historic photo: Flickr Commons)
K1wi's recent diary showed us how the use of
destination is catching up with
exit_to in the US for exit sign tagging:
I am excited about this! You may recall that I am very much in favor of
destination on the
_link way to tag exit signs rather than
exit_to on the junction node.
I've come up with an idea to squash the remaining
exit_to-tagged nodes in the US and move the exit sign info to a
No scripting. No funny stuff.
A MapRoulette Challenge!
This is what it would look like:
What do you think? How can the instructions be improved?
You can try it out on dev.maproulette.org now.
Looking for new ways to get people out to map, I started a new initiative in our local Salt Lake City OpenStreetMap group: The Flash Map Mob. Inspired by the Flash Mob, the idea is to descend on a local commercial area, spread out and map all the businesses in an hour or less. We tyically meet at a coffee shop, divide up the area, and go out and map! Most people use their smart phones or tablets with apps like Pushpin, Vespucci or OSMAnd. This way, everything you map gets added to OSM right away and there is no work to be done after. But you could also use pen and paper, or GPS + camera.
Here we are checking out Pushpin on an iPad:
Here are the results of the first Flash Map Mob we did a few weeks ago, where we mapped 80+ businesses with three people:
(Here is the Overpass Turbo request that I used to create this result overview.)
For now we do these Flash Map Mobs on a weekday after work. The next one will be in the Murray Historic Downtown area, just south of Salt Lake City. Previous ones were in the suburb of Holladay and in the Sugar House area of Salt Lake City. So far it's only been 2-3 of us, but it is a lot of fun so I expect more people at future Flash Map Mobs!
It's really easy to organize a Flash Map Mob. Just go to OSM, pick an area that could use some POI love, announce on Meetup or whatever channel you use to announce your local OSM events, and off you go!
I don't do a lot of 'on the go' mapping. I have tried a lot of the tools out there that should help me do that: KeypadMapper, Go Map!!, Vespucci, Pushpin, OsmAnd. Most of these are way too convoluted for me. When I am out and about, I just want to make a quick edit - usually a POI of sorts - and move on.
Pushpin and Go Map!! come the closest to what I want. Both are available for iOS only, I think. (Do Android users not like simple editing tools?) They let me quickly add a node and upload it to OSM. Or add some tags to an existing one.
Still, what I end up doing most is just taking pictures with my phone. Lots of them. I try to get both overview and detail. I end up with a collection of images like this one here from right before I moved to the US.
When I get home, I copy all the images to my computer and load the entire directory into JOSM. It will put the image locations on a map. I can cycle through them easily and map what's on there using all the convenience of JOSM.
The only thing you need to verify is that your phone saves the location with the image. This is on by default on most phones these days.
How do you prefer to map while on the go?