Recent diary entries
We had a discussion about retiring the use of
exit_to in favor of
destination for motorway / trunk exit tagging in the US a while ago. Since then, a shift has happened and a majority of exits is now mapped with
destination instead of
exit_to. This is great for navigation applications that rely on detailed signpost information, such as OsmAnd, maps.me and Scout.
With the situation in the U.S. so much improved, a next obvious target for mapping is Canada :) The situation there has been improved already by the community, but also by organized mapping by Mapbox and Telenav. Still, about 1200 exits remain that are tagged with the 'old' scheme. With the help of my colleagues at Telenav we put these in a MapRoulette challenge.
Because both OpenStreetView and Mapillary have good coverage in Canada, I think we should be able to update most of these exits to use the new scheme. Let's give this a go!
Important note: The imagery I use as an example below is different in source from the imagery used for the MapRoulette imagery. The example below shows imagery that was commissioned by Texas itself, and is available in the public domain. The imagery in the MapRoulette challenge is licensed from Google by Texas and made available to MapRoulette specifically. So I can't say positively that it's OK to add this imagery to JOSM or iD, and removed specific instructions to do so.
Have you tried the Texas Cemetery challenge in MapRoulette? If you have not heard about it yet, I posted about it on my diary a few weeks ago. The short version: the friendly folks at TxDOT supplied me with a database of their known cemetery locations, we matched them with existing OSM, and if there was no match, we ask you to map it :)
If you tried it, you may have found though that it can be a bit frustrating :( Bing and Mapbox aerial imagery is often just not detailed enough to see if there is a cemetery or not in the location indicated. I discussed this problem with the friendly folks over at TxDOT, who are very excited about getting more data into OSM. They told me about some of the high resolution imagery that is available to the public through TNRIS, the Texas Natural Resources Information System. Here is an example of some of the amazing data they have:
If you compare that with Bing, it means the difference between seeing a vague blur or seeing the presence of a cemetery very clearly! Super exciting stuff. So we set out to create a TMS endpoint that we plugged into MapRoulette. See the difference!
Go give it another try! Thank you and thanks TxDOT!
I am collaborating with agencies in Texas to update both OSM and Texas data. The pilot project deals with cemeteries. I received a file with almost 7000 cemetery locations. (Even if the idea that there are more people living today than have died thus far in human history turns out to be a myth, I think that is quite a lot!).
The first phase of this collaboration is to see which cemeteries in the Texas data actually exist. We will use MapRoulette for that. Simply go to the Cemetery challenge at maproulette.org and start looking at tasks.
If you see a cemetery in the aerial image, click 'skip' to go to the next one. If you don't see a cemetery, click 'False Positive'. If you are in doubt, click 'skip'.
How can you tell if there is a cemetery? Sometimes it is hard. Look for fine patterns defining the plots, and usually there will be a service road connecting the cemetery to the road network. Sometimes, in larger cemeteries, you may also see paths inside the cemetery. Finally, the marker may not be right on the cemetery, so look around a bit as well. Below are some examples of cemeteries and non-cemeteries.
Once we complete stage 1, we will turn to mapping all the cemeteries that are not yet in OSM yet!
There is a cemetery here: fine regular pattern indicating plots, some paths.
There is a cemetery here also.
No cemetery here, just some grass.
After almost a year of thinking, development and testing, the OSM team at Telenav is ready to present OpenStreetView to all OSM mappers! OpenStreetview (OSV) is the free and open street level imagery platform designed 100% with OSM and mappers in mind.
We officially presented OSV to the OSM community at State of the Map US where we had a 20 minute talk and a booth where we gave away crazy little remote controlled cars to everyone who signed up :). The cars were gone quickly – almost half of the people at SOTM US signed up! - but you can still see the talk thanks to the great SOTM US organizers who had all the sessions professionally recorded. If you have 20 minutes and don't like reading, watching that video is going to be the best way to be introduced to what OSV is and how you can use it to improve OSM. Or if you are coming to SOTM in Brussels, you can come meet our team there (more remote controlled cars? Who knows!) and attend the workshop.
The OpenStreetView booth at SOTM US
If you do prefer reading, read on! I wanted to quickly introduce OSV, what the components are, why we believe it is the #1 choice of street level imagery for OSM, and of course how to contribute and use it.
OSV is a web site, openstreetview.org, free and open source mobile apps for Android and iOS, a specialized Map Editor, a plugin for JOSM, and of course a back end server. Support for iD is also planned.
The web site is where you go to explore imagery from all over the world, see leaderboards and your own profile and trips. To see your personal stuff, of course you will need to sign in. Your OpenStreetView account is linked to your OSM account, so you don't need to create a separate account. All we store when you sign in for the first time is whatever is public on OSM. (If you want to check what that is, go to
8909 to whatever your OSM ID is -- unless you want to see my details.)
The apps are free to download from the play / app store. For Android, you can also download the APK directly. With the apps you can capture trips. They are optimized for driving but also work well for biking and walking scenarios. Apart from recording trips, you can also upload your trips to OSV. This will happen automatically when you enter WiFi, if you want. Finally you can review your local and server trips and see your profile. Even if you have not logged in on the web site, you can log in to OSV with the apps, also through OSM OAuth. Either way, this will create an account for you on OSV.
One thing that is really specific to OSV is that you can link the app to an OBD2 dongle in your car. Those are little devices that read from the OBD2 port in your car. Almost every car has one. (Challenge: find yours!). The dongle reads all kinds of diagnostic info from the car and broadcast it over Bluetooth or WiFi. They cost around 20 Euros. A list of OSV compatible ones is on the OSM wiki. (Ehm, a very small list so far. If you have a different model, please add your experience!)
OSV will read the speed and curve to improve the accuracy of the GPS signal that is recorded for your trip. It comes in extrememly handy when GPS reception is poor or lost altogether, for example in dense tree cover or in tunnels. The dead reckoning provided by the OBD2 unit will maintain proper alignment to the road. Here you see what that means when you are driving through a tunnel (blue = GPS only, signal lost, red = with OBD2 connection).
The OSV apps also have sign detection built in. So this is not done on the server but at 60FPS on the client! This means that it will detect speed limit signs, and more to come, in real time and can warn you if you are speeding. This warning feature is almost ready and will be in one of the next builds. (We update the apps very frequently.)
The JOSM plugin is in an early beta stage. Right now it will simply display the locations of images on the OSV server, and you can click on them to show the image in the OSV panel. Basic functionality, but it works :) and we hope that you have ideas (or even code) to improve it.
Speaking of ideas, we already have an active community reporting issues and suggestions on Github. This is the best place to let us know of any bugs and ideas you have about any part of OSV. Github is also where all the source code for all the components mentioned here is located. Almost everything in OSV is open source, and if it is not we are looking at how we can make it open source.
If you do not like Github or do not want to create an account there, you can also write to firstname.lastname@example.org with your ideas or bug reports. We are also on Twitter as @openstreetview and we are getting on Facebook and Instagram if you're into that kind of thing.
If you check Github, you will see that we also have upload tools for your existing Virb / GoPro and other action camera images. These are Python scripts, but we also have a GUI tool that you can just drag and drop directories onto to upload. This is in early beta but if you want a copy, let me know.
There are two components that I have not mentioned yet: the Map Editor and the back end. I want to save those for a separate post that I will write soon. Here is a screenshot of the map editor:
We think that OpenStreetView is the #1 choice for street level imagery for OSM. Not only because it is almost completely open source. You also remain in full control of the data you upload to OSV. You can always delete individual photos, trips or even delete everything and remove your account if at any moment you don't want to be a part of OSV any longer. This option is on the web site, no need to email anyone or submit a request.
Another reason is because we are building a platform that is very tightly integrated with OSM. This is obvious from the way we handle user accounts: log in with OSM, no separate account. But also deeper down the integration with OSM is tight: we map all trips to OSM ways, so we can link back and forth between trips / images and OSM way objects. This opens up all kinds of interesting possibilities, and I want to spend a separate post on that as well.
For now, it would be cool if you would give OpenStreetView a try. Download the app for your phone, sign in and start capturing. Tell us about your experiences. Explore what is already there. And most important please use it to improve OSM!
Note that OpenStreetView is not a project run by OpenStreetMap or the OpenStreetMap Foundation. It is maintained by Telenav, where I work, for the sole benefit of improving OpenStreetMap.
I am on a roll mapping pedestrian crossings (or 'crosswalks' as Americans tend to call them.
First I download a sliver of the map that covers a major road in JOSM:
I think you could also use 'download along way' in JOSM if the road is not nice and straight, but around here they usually are.
Then I pan along the way and add crossing nodes using
Shift-R to quickly copy pedestrian crossing tags from the previous node.
This way I can add about 15 pedestrian crossings a minute.
I have been on a bit of a MapRoulette binge lately. MapRoulette 2 is coming along nicely and we are at the point where we can start working on the front end. This is where a lot of your suggestions come in. If you have more ideas about how MapRoulette should (not) work, please take a moment to go to that PiratePad and add them. Thanks!
MapRoulette 1 is still very much alive however! Let's look at what has been happening.
Pedestrian Safety challenges
Last week, I posted new Sidewalk Mapping challenges (Tampa, Salt Lake City, your city?) to help OSM become a better map for getting around as a pedestrian safely in cities in the United States. Given that someone on foot on U.S. streets was hit by a car about every 8 minutes in the past decade, we could use better maps to help prevent accidents.
Speaking of pedestrian safety! Dr. Stefan Keller, a long time OSM enthusiast and founder of the Geometalab at the Hochschule für Technik, Rapperswil launched a really cool initiative to detect missing crosswalks based on analysis of both OSM data and aerial imagery. The results are making their way to MapRoulette, too.
A missing sidewalk example identified by Stefan Keller's work
While I was researching the sidewalks issue, I figured I needed an easier way to create challenges from an Overpass query. I decided to use existing components as much as possible and came up with
geojson2maproulette, a Python script that leverages the MapRoulette Python API wrapper and the GeoJSON export functionality from Overpass Turbo to make this a pretty easy process. I think it is useful, and I hope you do too. Definitely get in touch if you need help with it or have ideas for improvements.
Floating Ways in China
The latest addition to MapRoulette is a bunch of 'floating ways' I happened upon when looking at OSM data in China. To see what I mean by 'floating ways', here is a simple example:
What appears to happen a lot when folks are mapping in China is that they trace a bit of road roughly from aerial imagery available, but they do not connect the road to the rest of the network. Although I believe that some (correct) data is better than no data at all, I thought it was worth highlighting these ways. I found more than 9000 in my analysis. If you want to do some mapping in China, this is a great way to get started!
See also Wille's post about this (in Portuguese)
We launched ImproveOSM about 6 months ago as a way to turn the vast amounts of GPS data that Scout users give us into useful and actionable hints mappers can use to add turn restrictions, missing roads as well as wrong or missing one-way streets. The response has been incredible -- since we launched, more than 26 thousand hints have been processed, leading to more than 16 thousand improvements to the map worldwide. I think that is a fantastic result, and we will keep working to make ImproveOSM better based on your feedback.
Initially, we just used our own GPS data to generate the hints. But there is no reason why we couldn't process any GPS data we can get from other sources. So I was really excited when long time Brazil mapper Wille Marcel got in touch with a cool idea. He worked with the Brazilian Environment Ministry, which collects GPS data of the vehicles that work in environmental monitoring. Most of the data are in rural areas where OSM is much less complete. So this was a perfect fit for ImproveOSM's missing roads tool.
After getting the proper permissions from the agency, Wille sent us the GPS data and we started analyzing it.
We quickly realized that the GPS data is much less dense than what we are used to working with. Some missing roads were only driven once. Our algorithm, tuned to higher density data, initially only detected a few tiles. We decided to loosen the detection threshold significantly for this particular dataset. After a few iterations of tweaking and testing, we ended up with more than 5000 tiles containing missing roads based on Wille's GPS data.
The missing roads in Brazil are on ImproveOSM now, so why not go to the web site or fire up the ImproveOSM JOSM plugin and help the Brazilian community out by adding some missing roads?
If you are in a similar position as Wille and know of a source of free and open GPS data for your country, please get in touch with me so we can look at the data and see if we can include it in ImproveOSM.
We are already working with a number of other folks who have lots of GPS data. Soon, the number of missing roads, one-ways, and turn restrictions in ImproveOSM will be much, much bigger. We are also working on a host of new features, so I hope you will stay tuned to the ImproveOSM blog to be among the first to hear about what we have up our sleeves for ImproveOSM and other OSM related projects we are working on. And follow us on Twitter at @ImproveOSM!
This post also appears on the ImproveOSM blog
United States cities are built for cars, with very few exceptions. From where I am sitting right now, I see this:
Cars zooming by incessantly at 70kph.
Finding your way in an urban space that is designed this way is tricky - and often dangerous - if you are walking or bicycling. Sidewalks are often not present, crossing streets can be very dangerous or even impossible. OSM has great tagging for bike lanes and sidewalks, but I find that these crucial tags are often missing on ways that need them most: the four or six lane urban arterials that you see in the picture above.
As I was sitting here asking myself how on earth I would get back to my hotel (which is 10 minutes away) safely, I thought to myself: 'we can fix this problem and make the world a bit safer for those who can't or won't drive.'
MapRoulette to the rescue!
I created this challenge highlighting all
secondary ways that have no
sidewalk tag in Tampa, Florida. (I am actually in Sarasota now, south of Tampa, but I already fixed all the ways there so that would be a boring challenge.) The idea is to look at the aerial image in JOSM or iD, see if there is a sidewalk, and add the appropriate tag. Adding
sidewalk=no is actually just as important as adding
left. Here is an example way from this challenge:
Even zooming further in there is no sight of a sidewalk:
So let's add that information:
Create a Challenge for your city
The fun part is that you can easily replicate this challenge for your own city. Here's what to do.
First you head over to Overpass Turbo and run the query that highlights all
highway=secondary that have no
You can use my query as a template, replacing the GeocodeArea with the name of your city.
Once you have the results, export them to GeoJSON. Let's use a gist:
You can now click on the gist link and see the result on GitHub as well:
We will need the 'raw' GeoJSON content, so click on the 'Raw' button and copy the link it leads you to.
Next we'll use a little tool I created to easily turn the contents of a GeoJSON file into a MapRoulette challenge. To get it, head over to the Github repository and follow the instructions to install the tool.
The tool takes its configuration from a YAML file. The samples directory contains an example for this sidewalks challenge you can use as a template:
# the base URL for the MapRoulette server API to call server: http://dev.maproulette.org/api #server: "http://localhost:5000/api" #server: http://maproulette.org/api # server API admin credentials user: devuser password: mylittlesony # source file or URL. You can give a list of URLs too, all data will be gathered and added to the same challenge. source_url: https://gist.githubusercontent.com/anonymous/310005dc1dcf08f5c4a7/raw/74aa0d6a845a20511d0c450d715dab23d2a6c0d6/overpass.geojson # source_file: .... # source geojson property key to use as your task identifier (optional, will use random UUID if not given) # identifier_property = ... # Challenge metadata, see https://gist.github.com/mvexel/b5ad1cb0c91ac245ea3f for background slug: sidewalks-sarasota title: Add sidewalks to major roads in Sarasota instruction: This way has no `sidewalk` tag. Usually you can see from the aerial imagery if there is a sidewalk or not. Please add the appropriate `sidewalk` tagging. help: "Help make OSM be a better resource for safe, walkable streets! Many primary and secondary roads in the US are not safe for pedestrians if there is no sidewalk. This challenge highlights all `primary` and `secondary` ways that have no [`sidewalk`](https://wiki.openstreetmap.org/wiki/Key:sidewalk) tagging whatsoever. You can help by looking at aerial imagery and adding the appropriate `sidewalk` tagging. `sidewalk=no` is just as important to have as the 'positive' values. Thanks for helping make OSM better!"
The only items you would need to change are the
source_url (use the raw GeoJSON github link you just copied), the
sidewalks-YOURSTATE-YOURCITY or something similar - this will be the challenge URL component in MapRoulette) and the
title (change the city name).
Once you have the YAML config file in order posting to MapRoulette is as simple as:
$ ./geojson2maproulette.py samples/sidewalks-sarasota.yaml --post --activate Posting 364 tasks... server alive: True Updating challenge... Reconciling tasks... Done!
Let me know if you need any help with this or if you want me to create a challenge for you!
It has been a few weeks since I wrote about the public beta release of Cygnus, the Telenav conflation engine for OSM data. Since then, I have since been approached by a few folks who wanted to take it for a spin. One of them is long time OSM contributor MikeN. He is preparing an import for Holt and Atchison counties in the U.S. state of Missouri. We worked together on scaling some technical hurdles. Here's a report of what we (well mostly he) did.
Mike obtained the source data from Holt and Atchison counties from their official GIS:
I obtained an updated road network from their official GIS, extracted and translated the tags, and followed up with a review against current aerials as well as checking for connectivity, glomming like road segments, and simplifying geometry. The final goal is to obtain permission to import, go through the import process steps, and merge new data onto the existing OSM data.
The next step was to convert the data into the OSM PBF format that Cygnus requires. This is when Mike got in touch with me to work through some technical difficulties:
Since Cygnus required the PBF format, I used Osmosis to convert. This failed because the nodes did not "have a version attribute as OSM 0.6 are required to have". I have learned from Martijn that OsmConvert works without a version attribute, and was able to verify this on my second county.
The next catch was that PBF doesn't accept negative node numbers. The simple workaround is to just use a text editor to remove the minus sign from
ref='-. This seems a bit dangerous - would that file upload if accidentally selected? If so, many low numbered objects would be corrupted around the world. Hopefully, the conversion from OSM to PBF can be moved to the Cygnus chain so that it can accept zipped OSM since most users will start with .OSM data.
That is a great suggestion. On the one hand, we don't want to make Cygnus too easy to use. (Cygnus is in the end a tool to help with imports. It should never be easy to just import data in OSM. There are strict guidelines, and any tool to help with imports should make the user consider the process very carfully.) On the other hand, handling the conversion from JOSM XML (including the negative IDs) to valid PBF is a mechanical step that most any Cygnus user would need to perform, so I would like to include that in a future version.
With that out of the way, we worked together to produce the Cygnus JOSM XML change file.
The result was that it did pick up the new roads and they appear to connect properly into the existing road network. The cases of modified geometry were also detected. And although the node placement was different, the rest of the roads were properly untouched.
It turned out that the number of changed / new roads was fairly minor. A future import would therefore not be too invasive. Here is the OSM base data versus the updates suggested by Cygnus for one of the counties Mike is working on:
In total, Cygnus suggested 68 updates. 31 entirely new geometries, and 37 updated geometries. The updated geometries were mostly caused by connecting the new ways to the existing network, adding a node to the existing way where that happens.
Mike is still working with the counties and the community to move this import forward. Working with Cygnus gave some good insights and will hopefully help prepare the actual import when it happens. This is pretty much an ideal use case for Cygnus, and I hope to see more of them.
Mike also has a wish list for Cygnus:
Future enhancements - In my case, I extracted the surface tags from the GIS source. It would be interesting to have more control over tag merging - such as taking surface tags from the 'new' ways if there is no current surface tag. And in the case of renamed roads, to be able to give the new name a priority for the merge.
I already discussed this with my team and this is high on our list of Cygnus improvements - together with support for POI type nodes.
Get in touch with me if you are ready to give Cygnus a try with local data you have!
I am happy to announce that Improve OSM, the suite of open source Telenav tools that help us fix OSM based on vast amounts of GPS data, is completely redesigned and expanded. We made the web site much easier to use. We combined the existing JOSM plugins into one new plugin. And as of today, we have an entirely new category of fixes: missing turn restrictions. I will talk about all of these changes in this diary entry. A lot to cover so let's get started!
The new Improve OSM web site
We redesigned the Improve OSM web site completely. Instead of separate maps for each type of error, all errors are now displayed on one map. You can turn each type on and off in the layer panel. You can also control the filters for each type there.
What's even better (I think) is that you can now perform the entire fixing workflow from the web tool, including resolving the error as fixed or invalid. (You used to need the JOSM plugin to change the status on items.)
The new plugin
We released two Improve OSM JOSM Plugins so far: MissingRoads and TrafficFlowDirection. With more error types in the pipeline, it made sense to combine all Improve OSM functionality into one plugin. This new plugin is simply called ImproveOSM. The functionality is largely the same as the previous separate plugins. Each error type still has its own layer. At low zoom levels, the results are clustered:
When you zoom in, individual errors become visible:
If you looked closely, you can see that we added a third layer. More about that next.
New Improve OSM type: Turn Restrictions
If you collect a lot of GPS traces like we do, all sorts of interesting patterns appear. You have seen some results with the Missing Roads and Traffic Flow Direction plugins. It gets even more interesting if you consider trip counts over more than one road segment. For example through an intersection. If you consider the relative amount of trips for each possible turn, it is possible to derive which turns are likely prohibited. Compare that against existing OSM data, and voilà - here is our next Improve OSM project!
(In a future diary I want to go into the technical details of turn restriction detection. Right now I will focus on the functionality of the new layer.)
Mapping Turn Restrictions
If you activate the Turn Restriction layer in either the Improve OSM web site or the JOSM plugin, you will encounter suggested turn restrictions like this one here:
I like to edit Turn Restrictions in JOSM myself, with the help of the TurnRestrictions plugin, but it can be done in iD as well. So I click 'edit in JOSM' to load the area in JOSM:
You see that I selected the suggested turn restriction from the Improve OSM Turn Restriction layer. You can also see that I loaded Mapillary data using the Mapillary plugin. Street level imagery can be a great help verifying turn restrictions, and in this case the selected Mapillary image revealed that there is indeed no left turn allowed in this case:
Image source: Mapillary
While the plugin suggests a no-left-turn restriction, I prefer to map this as an only-straight-on restriction, because that is closest to the actual sign on the road:
With the new turn restriction in place I can upload the changeset. I use #improveosm in the changeset comment and I encourage you to do the same:
And finally I can choose to either mark the case as fixed in the web tool or straight in the plugin. I opt for the web tool:
And that's it!
We are still tweaking this algorithm and you may see false positives. Please mark them as 'invalid' if you see them. If you think you see a pattern, don't hesitate to use the feedback links in the plugin or on the web site to let us know. You can also just email me at email@example.com.
Sometimes, the algorithm unveils unexpected other map errors as well. Looking at this situation where the algorithm detected a missing left turn restriction:
It would seem that this is a viaduct, so there would be no turn to make at all. But it turns out that the bridge and one of the motorway segments passing under it were connected:
So definitely something you would want to fix!
Happy Holidays and Happy Mapping!
I hope this gives you some interesting things to map over the holidays (in case you celebrate). Let me know what you think! I will be back in the new year with a more detailed post on how we detect turn restrictions. The team will also start looking at any feedback that comes in then.
Missing Roads is doing great! Since we launched the plugin a little over two months ago, almost 14 thousand Missing Road tiles were closed.
A little over a quarter of those were marked as invalid. We have been looking carefully at those cases, and it turned out the two most occurring invalid tiles were ones involving foot / bike paths and water. So we created filters for those. This should help you use the Missing Roads plugin much more efficiently!
Here is the filter in action for water tiles in the web tool:
And in the plugin:
Update After talking to a lot of people about the future of MapRoulette and getting a lot of great feedback, I decided to abandon the Django effort I wrote about here. The New MapRoulette is coming together nicely! See its progress on Github.
I recently started looking into Django, the seminal Python web application framework. I like what I see much more that I thought I would, so much so that I am starting to rewrite MapRoulette as a Django application.
One thing I want to accomplish with a next version of MapRoulette is to make creating challenges much, much easier. For example, you should be able to define a challenge using an Overpass query. (You would be surprised how many annoying errors in OSM can be exposed with a pretty simple Overpass query! For example, here is a query that gives you all
highway=tertiary that have no name in my home state Utah - 425 ways.)
So I needed a way for a Django application to access the Overpass API and display and store the results. That should be simple enough to accomplish using the Overpass API Python wrapper project I started a while ago. And it was!
If you are interested in checking it out or trying it for yourself, just clone the Overpass API Python wrapper project. Look for the Django example app in the
examples folder. (At the time of writing, the example lives on the branch
example that is not merged into master yet.)
Many thanks to all the contributors to the Overpass API Python wrapper project! Let me know if you want to contribute as well, be it code, testing, or documentation. Thanks!
I wrote about Cygnus, our effort to create an OSM-specific conflation engine, a few months ago. We developed it specifically to aid the import of INEGI road data in Mexico we are preparing together with the community in Mexico. But when used with care, I think it can be very useful in other import scenarios as well. This is why we decided to make it into a web tool anyone can use.
Before I go into details, I should offer a few words of caution.
Firstly, this tool is an early beta. Right now, it only conflates roads, nothing else. We have tested the tool internally, but only on a limited amount of cases, all using open INEGI data from Mexico. With this public beta, I hope to gather more feedback to help us make improvements to it.
Secondly, using this tool requires careful planning and preparation. It is definitely not for casual users. It is useful mostly in import scenarios, so the usual extreme caution and preparation related to data imports apply as well. Anyone considering any import needs to take this warning on the imports wiki page very seriously:
If any of what you read below feels confusing or difficult, you should probably not be using Cygnus or attempting a data import in the first place. I do not mean to be discouraging, but importing data into OSM is hard, and you can easily destroy other people's work. So if you become aware of an external dataset that you think would be interesting to incorporate into OSM, study the import guidelines, talk to your local community and discuss the best way forward. Never go at this alone.
What does Cygnus do?
Conflating in GIS is the act of merging two data layers to create one layer containing the features and attributes of both original layers. (A more official sounding definition can be found here.) Cygnus is a tool that conflates external data with OSM. You feed it an external dataset, and Cygnus will compare it against current OSM data. It will give you a result file in JOSM XML format with all the changes. You can load this change file into JOSM and merge it with an OSM data layer. The result could - with extreme caution! see above! - be uploaded to OSM.
Cygnus does its conflation in a non-destructive way. No existing OSM ways are ever deleted or degraded. Existing geometries do get changed where new connections need to be made. Where new and existing ways overlap, Cygnus will honor the original way geometry and attributes.
Let's look at an example so I can go into much more detail.
An INEGI example: Los Morales, Mexico
For this example, I picked the local road data for Los Morales, a hamlet north of the city of Monterrey, Mexico. Looking at Geofabrik's Map Compare, this hamlet is all but non-existent on OSM:
Mexico has a huge open data initiative, and I wrote about the data from the national Census bureau, INEGI, previously. The data can be found on the INEGI web site as part of the dataset 'Información Vectorial de Localidades Amanzanadas y Números Exteriores' for the administrative region of Salinas Victoria. (More background for the various INEGI road datasets is being compiled on the OpenStreetMap wiki by my colleague Andres.)
Before we can even start to think about conflation, we need to ensure a proper attibute translation. I purposely picked a fairly uncomplicated example so we can remain focused on the process as a whole. The attributes for this dataset are fairly straightforward:
I want to discuss the translation of INEGI data specifically in another blog post in the future. We are working with INEGI to get as much data as we can to ensure a proper mapping from INEGI road types to OSM way types. What I am doing here is a much simplified example for the sake of this demonstration. The result will not actually be uploaded to OSM.
Upload to Cygnus
Now that we have an input file, we can offer it to Cygnus for processing. When you load the Cygnus service page, you see this simple interface:
There are just two pages: the home page where you add new jobs, and the Job Queue page where you can see your progress and download the result. To add a job to the Cygnus queue, I upload the file I have prepared, and add it to the job queue:
Note that your upload needs to be small-ish: the spatial extent needs to be smaller than 50x50km and the file needs to be 20MB or smaller in size.
If your input file was uploaded successfully, Cygnus will go to work. Your job will be added to the back of the queue. When it's your turn, Cygnus will read your PBF file, and download the OSM data for the same extent. This is done using the Overpass API. It will then compare your upload with the existing OSM data, and produce the ouptut file. I will spend a separate post on more details about the Cygnus process. For now, the most important thing to remember is that Cygnus will only consider roads. This includes most
highway ways in OSM. Any other data will be ignored.
When Cygnus is done processing your file, it will be available to download in the job queue.
You can download and/or remove your file here. Everyone's jobs are visible here, so please be careful not to touch other users' stuff.
Inspect and process in JOSM
The downloaded file is plain JOSM XML:
What you see here are the differences between OSM and your uploaded file. This includes new ways added, ways with changed geometries and ways with new tags added. Next we need to inspect the changes carefully against existing OSM data. Cygnus is set to conflate very conservatively by default. The results surely will need manual tweaking.
This is by far the most important, and time consuming, step!
So I load the OSM data in JOSM for the same extent. First, I use the layer panel to get a quick overview of what has been added or changed:
Because there was already a
highway=secondary, it is probably a good idea to pay close attention to the data there. While Cygnus does a best effort to connect ways where needed, it acts conservatively so it will not snap ways together that do not belong together.
Here are a few ways that got properly connected to the existing
But here the distance was too far so Cygnus did not snap:
In this case, you would need to manually connect the ways if that is appropriate.
You can also inspect what Cygnus proposes by selecting any way of the Cygnus layer and looking for the
telenav:graphenhancer tag. This will have the value
new for added ways, and
changed:geometry for ways that have geometry changes, for example.
(The quality of the result will not only depend on what Cygnus does. 'Garbage In, Garbage Out' also applies. So before you even offer your file to Cygnus for conflating, make sure you have triple / quadruple-checked your atribute translation tables and other pre-processing steps.)
When you are finally satisfied with your manually post-processed conflation result, you can go ahead and merge it with the OSM data in JOSM:
When that is done, you probably want to remove the
After that, the data should be close to ready to be uploaded to OSM. In this case, I am not going to do this because I did not follow import procedures at all, and I wrote a quick, simplified translation file for the attributes. So Los Morales is still just as sadly absent from OSM as it was before. But I hope this will give you an idea of what Cygnus is capable of.
If you have existing import plans that involve road network, and you would like to take Cygnus for a spin, please do. I am happy to help. Email me, ping me on twitter or Skype (mvexel). I am looking forward to hearing from you!
You can be more than one kind of geek at the same time, I guess. First and foremost, I am a map geek. This started when I got my first atlas for my 6th birthday. Then came computers, with the arrival of the Commodore 64 I shared with my brothers. Shortly after that came trains. I still have a few thousand slides of stations, trains and railyards in my basement. Also, German and Dutch timetables dating back to 1985.
One of my later railway photos. Berlin Lehrter Stadtbahnhof, summer 1996. This was demolished not too long after to make room for Berlin HBf. Source: Flickr
Roads are a more recent object of geekery for me. Before I moved to the United States, I didn't drive much. As I started driving (and mapping) more in my new home country, I also started to become more interested in the rich history of the vast road network in the United States. And I am not the only one! It doesn't take much online searching to find sites like AARoads and CrossCountryRoads containing 1000s of freeway driving images and videos, as well as Facebook groups dedicated to the celebration of U.S. freeways present and past. (I especially enjoy Freeway Jim for its mix of history and current content. And for Utahns, this Flickr account is pretty cool too.)
I thought it might be interesting to start tapping into this road geek community to help us improve OSM. Even in a well mapped state like Utah cough, most freeway exits still don't have any signpost information except for a
ref. To give you an idea: of 1654 total
motorway_link ways, only 193 have a
destination tag. Even accepting that probably half of these ways are onramps, that's still a big data gap! And one that sites like AAroads and CrossCountryRoads fill perfectly, with images like this one:
(If you're interested, here is the Overpass query for the
motorway_link ways with
After some email exchanges between various US community members and the maintainers of AARoads and CrossCountryRoads, we now have permission to use the images and videos on these sites for mapping. Yay! (I created a wiki page for CrossCountryRoads.com. There is none for AAroads yet, as far as I know.)
I thought I'd try it out myself and share my experiences.
First I pick a region I want to improve. CrossCountryRoads.com has a list of all the Interstates it has coverage for, sorted by state. No Utah, unfortunately :( Let's pick I-68 Westbound in West Virginia. I see a neatly organized list of images, mostly covering exits. Excellent!
Next, I load all
motorway_link ways as well as all
motorway_junction nodes for West Virginia into JOSM. (Perhaps overkill, but I am not an Overpass expert..).
The result looks a little bit like one of those flight tracking web sites :)
The first exit is to the WV Welcome center and a rest area.
I zoom / pan to where I-68 enters West Virginia from Maryland and quickly find the exit, and add the
I also check the
motorway_junction node for any irregularities (such as deprecated
exit_to tags and missing / wrong
I will delete this
exit_to tag. Does anyone know if
noref is a thing?
With the edits done, I upload the changeset. Be sure to attribute the source - in this case CrossCountryRoads:
Then I move on to the next exit and map some more!
I announced our second tool based on Scout GPS data, Traffic Flow Direction, a few days ago. I didn't spend a lot of time on explaining how it works. This post will hopefully make up for that!
The goal of this plugin, and the accompanying web tool, is to make it easy to find and correct OSM ways that we think are missing a
oneway tag, based on billions of GPS points from Scout users. Here is what it looks like in JOSM:
I will walk you through installation, basic operation and some mapping tips in the next paragraphs. Happy mapping!
This is a JOSM plugin so, installation works like any JOSM plugin. Make sure your JOSM is up to date first. Then go to JOSM preferences. Select the Plugins tab and look for the TrafficFlowDirection entry:
Select it and click 'Update Plugins' at the bottom. After JOSM completes installing the plugin (and updating any others that may need updating), restart JOSM and you should see the main components of the plugin appear in the JOSM interface: the overlay on the map and the plugin panel.
You may want to adjust some basic JOSM settings to make optimal use of the plugin as well. The one I would recommend for sure is to enable directional arrows on ways. The direction of the way is crucually important for adding the correct
oneway tagging, so having that direction visible at a glance is really helpful. To enable this, go into JOSM settings and select the Display Settings tab. This is the topmost tab. Within display settings, select the OSM data tab. Here you will find the Segment drawing options:
I recommend selecting both Draw Direction Arrows and Only on head of way. Also make sure Draw oneway arrows below that is selected so you can see at a glance if a way is already tagged with
Like the Missing Roads plugin, the visual components of the Traffic Flow Direction are a map overlay and a settings / information panel. The map overlay shows clusters of suspected errors at low zoom levels, and individual errors at high zoom levels. Here is the colorful scene you get when you enable Missing Roads and Traffic Flow Direction at the same time:
(I recommend hiding the Missing Roads layer when you work on Traffic Flow Direction and vice versa :))
Zoom in on an area you want to work on and you will start seeing individual arrows:
The arrows depict OSM ways that should have a
oneway tag in the indicated direction, according to our GPS data. Click on an arrow to select it and find out more in the Traffic Flow Direction panel:
(If you do not see the panel, you may need to activate from the Window menu.)
The Info tab gives you some basic information on the selected direction error. The most interesting are probably the top two, indicating the % of all drives through that particular way traveling in the indicated direction, and below that the total number of trips that go through that way.
Validating and correcting the errors
Once you are zoomed in to an area where you want to fix some direction errors, make sure you have the best possible aerial imagery layer for your area enabled. Always remember that aerial imagery may be out of date and may not reflect the current reality on the ground!
Now, look for corroborating clues in the aerial image. These may vary by country, but I have found these clues are helpful:
The markings on the road. The suspected one-way street in the image above has a stop line all across the width of the street. Notice how the crossing street has a half stop line indicating that this is a two way street.
Other useful markings are arrows and painted speed limits. Look for these close to intersections.
Another useful clue is the direction of parked cars. In some countries (like the U.K. and most states in the U.S.) it is not legal to park 'against traffic'. So if all cars are parked in one direction, as in the animation above, that gives you a solid clue.
The best clue of course is local knowledge. Start with areas you know well to gain some confidence and experience!
Next, download the data for the area. Before you add
oneway=yes though, check if it applies to the entire way. The OSM way may be longer than the segment we suggest. This has to do with the way we internally process OSM data. We split the ways at each intersection. You may need to split the way before you apply the
(Above) This OSM way may need to be split.
Because we split ways into shorter segments, the opposite may also happen. The
oneway segment may extend beyond what the plugin suggests. This may be because we don't have enough trips through all segments to be sure.
Another thing to watch out for is the directionality of the OSM way. If the direction of the way in OSM (the order of the nodes, basically) is the opposite of the
oneway direction, you either need to tag the way with
oneway=-1 or reverse the direction of the way first.
(Above) The direction of the OSM way (selected, red) is different from the suggested
After you make the
oneway improvements, upload the changeset with
trafficflowdirection in the source.
Finally, mark the issue as Closed. Do this by first making the TrafficFlowDirection layer active and selecting the issue you just solved:
Then click the green lock in the plugin panel, add a comment, and close the issue.
If you find that after inspecting the aerial image or your local knowledge, the way is really not one-way, you can mark the issue as Invalid instead, using the red exclamation mark icon in the plugin panel.
We define three confidence levels for the suspected Traffic Flow Direction errors. These levels are based on different thresholds for the total number of trips and the percentage going of trips going in one direction. Clicking on the 'filter' icon in the Traffic Flow Direction panel will reveal a filter dialog that lets you narrow down the visible errors:
You can also use the filter dialog to switch between Open, Solved and Invalid errors.
You can comment on the selected issue by clicking the blue text bubble icon in the plugin panel:
Perhaps you want to use a different OSM editor. Or you just want to browse around? This is what we created the Traffic Flow Direction Web tool for.
The tool is similar to the Missing Roads web tool. It will show a heatmap at lower zoom levels, and individual errors at higher zoom levels. It allows for filtering by status and confidence level. If you are zoomed in far enough, 'Edit in...' buttons will appear.
I hope you find this useful! I would really like to hear what you think about the tool, this manual, and our efforts to make our data more open. Let me know at firstname.lastname@example.org or comment below.
Thank you and happy mapping!
Last month, we released the Missing Roads plugin and web tool. This quickly became a popular pastime for quite a few mappers - one month later, more than 20% of all the Missing Road tiles have already been resolved :)
For the darker and colder November days -- well, if you are on the Northern hemisphere at least -- we thought we would cook up something new to keep us all busy. We ran another analysis on our GPS data to uncover ways that probably do not have the right directionality. Either they should be
oneway=yes and they are not, or they are
oneway=yes but in the wrong direction.
Here is an example from Karlsruhe, Germany:
The orange arrow points in the direction we think traffic on that street flows based on what we know from our Scout GPS data.
The way we do this is by looking at the directionality of the GPS tracks. If more than, say, 90% of all tracks matched to an OSM way go in a single direction, it is pretty safe to assume that it is a one-way street. We then compare that to how the way is mapped in OSM. If there is no correct
oneway tag on the way, you will see the way in this tool.
Let's open the earlier example up in JOSM:
This looks a bit cluttered because there is so much OSM data, but if we zoom in a bit and make the OSM data layer invisible for a moment, we can see from the direction the cars are parked and the arrows on the road that this is indeed a one-way street:
Aerial imagery in JOSM from Bing
We found more than 140.000 (!) of these cases all over the world waiting for us to fix them. This works almost exactly the same as in our Missing Roads plugin and web tool we released last month. You can look at the Missing Roads manual. (I will write a proper manual for the One-way plugin and tool soon.)
Here is the address for the web tool: http://improve-osm.org/trafficFlowDirection. You can find the JOSM plugin in the JOSM plugin preferences. The source code for the plugin is on Github as always. We started a wiki page for the plugin as well.
Happy mapping and please, as always, let me know what you think :)
I have long wanted to try and do some audio mapping. Especially since I moved to the US and started spending more time in a car.
When you are driving, there is not a lot of ways you can record what you see in a way that makes it easy to map later. One way is to use Mapillary, but the sheer amount of information can be overwhelming. A picture every 5 seconds means 180 images to go through on a short, 15 minute drive. It also means handling over 300 MB of image data. And that's only for a 15 minute drive.
So audio mapping. I have this tiny recorder that weighs almost nothing, has built in space for almost 70 hours (!) of recording and runs weeks on a set of AAA batteries:
So I thought I'd use that. Reading about audio mapping I quickly learned that there are a few strategies. One is to record one long audio track. This way you don't have to push buttons a lot while driving which is nice. Another is to make lots of tiny recordings, one for everything you notice and want to map later.
The second method seemed more appealing to me even if it means pushing the record button a lot while driving. I was envisaging a strategy and result result similar to photo mapping. You record a GPS track as usual, and record audio clips when needed. Once home you would load both into JOSM. You would then see a string of markers along your GPS track. Each marker represents an audio clip. Play - map - play - map!
This is kind of how it turned out, but it turned out I needed to do some thinking and programming.
Hurdles to jump
First, my recorder can only write MP3 files. JOSM will only load WAV files. So I needed a way to easily batch convert the files while preserving the file time stamps.
Second, you have the same offset issues as with photo mapping. Because you use different devices to record the audio and the GPS, there will invariably be some difference between the internal clocks of the two. You need to resolve this before loading GPS and audio clips into JOSM, or the time shift will result in misplaced audio markers. For photo mapping, there is a built in function to adjust this offset in JOSM. For audio clips, there is no such thing.
So I decided to write a small tool that does both these things. In the best tradition of cryptically named OSM tools I named it
mp3towav.py. You can learn more about it at the Github repository. In brief, running this tool on a directory of MP3 files will give you a directory with corresponding WAV files with proper (optionally offset) timestamps.
OSMTracker can also record short audio clips at the press of a button! I used it a lot when I had an Android phone.
The fun part!
With the WAV files handy, all you need to do is load your GPX file into JOSM, and add the audio clips. Curiously this works a little different than with images. The GPX file you just load through the File > Open menu. But then you need to right-click the GPS layer to get access to the 'Import Audio...' function:
You can then select your WAV files, which should show up as markers neatly along your GPS trace:
You can then click on the individual markers and enter the awkward world of listening to your own voice pointing out addresses, businesses, street names and other map-worthy observations.
Once you get in the groove, you can use the audio menu and the shortcut keys F4-F9 to quickly navigate through your clips.
Great, that wasn't too difficult. Let me know how you audio-map, or if this quick walk-through makes you want to try it too!
A few weeks ago, I started something new for the Salt Lake City area OpenStreetMap group: Flash Map Mobs! The idea is to have a group of mappers descend on an under-mapped area and add lots of shops, restaurants etc. in a short amount of time. We do them after work, say from 5-6. Sometimes we get a drink or dinner afterwards.
We have done four so far and I wanted to share some results.
Flash Map Mob 1 - Sugar House
- Meeting Location: Sugar House Coffee - meetup
- Attendees: 3
- Focus: 21st South / Highland Dr area
- POIs added: 88 - Overpass Turbo
The first one! I decided to meet at Sugar House Coffee because I go there a lot. This place can get a bit crowded and loud however. This may turn people off so perhaps not perfect as a meeting place. The Mob itself (then still called 'After Work Map&Meet') was fun. We shared experiences and arrived at Pushpin OSM as the ideal iOS mobile editor for this kind of mapping.
Flash Map Mob 2 - Holladay
- Meeting location: 3 Cups Coffee - meetup
- Attendees: 3
- Focus: Downtown Holladay
- POIs added: 56 - Overpass Turbo
This was our first venture out into the suburbs. Very rewarding, because downtown Holladay has seen a lot of new development in the last couple of years, very little of which had been mapped yet. Met in a coffee shop again, which worked fine this time - this is a large, uncluttered space that is quiet at this time of day (suburbia):
Flash Map Mob 3 - Murray
- Meeting location: The cafe inside Barnes & Noble - meetup
- Attendees: 3
- Focus: Downtown Murray along State Street
- POIs added: 72 - Overpass Turbo
This was perhaps not an ideal location for a Flash Map Mob, but we managed to add over 70 POIs nonetheless. It's just not a very attractive location to walk around - a lot of strip mall type areas. We decided to split up and have some folks drive to different clusters of shops and restaurants. We made it work, but perhaps we should check WalkScore or something similar next time before we settle on a location.
Flash Map Mob 4 - Magna
- Meeting location: Magna Library - meetup
- Attendees: 2
- Focus: Magna Main Street
- POIs added: 62 - Overpass Turbo
The library is a good location to meet. It is public and has great internet. Better than what I have at home! The one downside is that it is supposed to be quiet, so if you want to discuss mapping strategies or meet up after and share experiences, there may be better places. Even though it was just the two of us, this was an interesting Mob, because I had never been to Magna, an old mining town on the western fringe of the Salt Lake City urban area.
Let me know your ideas for how to make the Flash Map Mob more fun / effective! If you want to run your own and don't know where to start, email me at email@example.com or find me on OSM IRC as mvexel and I can give you some pointers.
Last week, we released the Missing Roads JOSM plugin, together with a web tool, the plugin source code, and a manual. More than 700 mappers have already given it a try, and nearly 10% of the missing roads tiles are already resolved. That is fantastic progress!
We have received a lot of great feedback from you and we are looking into every single suggestion. You may have noticed that we have already updated the web app adding a location hash. This means you can now share the URL at any time and it will include the current map view.
We also received a few requests to get access to the data powering the plugin and app. This sounded like a great idea to us! We started publishing daily dumps of the tile data. You can download and use these any way you see fit.
This is what the data looks like:
tileX;tileY;timestamp;numberOfTrips;status;points;type 70009;102823;1443437990.487;5;OPEN;MULTIPOINT(-83.855907 36.129896,-83.85606 36.12981,-83.855953 36.129583,-83.855955 36.129581,-83.85594 36.1296,-83.85592 36.13,-83.855952 36.129584,-83.855907 36.129948,-83.855903 36.129621,-83.85604 36.12973,-83.855935 36.129658,-83.855897 36.130072,-83.855906 36.129732,-83.85598 36.12953,-83.855911 36.129993,-83.855933 36.129608,-83.85595 36.129584,-83.85613 36.13007,-83.85611 36.12993,-83.855914 36.12965,-83.855943 36.129601,-83.85608 36.12987,-83.85592 36.129613,-83.855901 36.130043,-83.855909 36.129836,-83.855952 36.129583,-83.855949 36.129592,-83.85593 36.12932,-83.855904 36.130029,-83.85614 36.13004,-83.8559 36.12995,-83.855946 36.129599,-83.855951 36.129584,-83.855911 36.129781,-83.85614 36.13002,-83.85613 36.12998,-83.855956 36.12958,-83.85601 36.12963,-83.8559 36.1301,-83.85592 36.13005,-83.85611 36.13011);PARKING
Head here to get the daily data dumps: http://improve-osm.org/missingRoadsDumps/
Let me know if you find this useful, or how you would like to see this improved. Email me or go to http://bit.ly/missing-roads-feedback. Thanks!
The most fun I have with OSM is making the map better. Adding nodes and ways and tags. Seeing how my puttering around with JOSM or iD leads to a prettier map that is more useful.
Getting folks together in Salt Lake City where I live to have some OSM fun together. For example. A few weeks ago, I started a new thing called the Flash Map Mob. We do them every other week now in SLC and together we map something like 100 shops and restaurants every time.
The map becomes more useful and folks are having fun. That is what OSM is to me!
I don't need to sit on any board to do these things. Heck, I can probably do more of them when I am not on the board! So why do I do this? What value do I add being on the OSM US Chapter board?
I'll tell you and I'll be brief.
OSM is still very good at adding contributors.
(The graphs come from osmstats.)
What we're not so good at is actually getting folks to go out and map. Look at the flat daily active mappers graph for the US:
This is not a US only problem. But I want to start attacking it where I live.
I don't think putting on big conferences like we have makes more people go out and map. I think we should stop doing that as OSM US.
We should direct that energy towards creating smaller OSM gatherings around the country. Gatherings for mappers. I want us to do 4 of those in the next year. They could be like the fantastic event I attended in Seattle for OSM's 10th anniversary. (More photos). Or something else? We should find out. What does an OSM event look like that you would go to?
We should put local mapping groups first. We should step up funding local groups with mini grants. We should do regular sessions to help people get started organizing their group. Gather more data on what works and what doesn't to help direct our energy. Perhaps it's mapping party kits? Perhaps it's changing the web site to be much more community / mapper oriented? I want us to find out.
I work at Telenav on OSM stuff. So I am in a position where I can get a fairly big organization with lots of talented people to do useful things for mappers. Recently we published Missing Roads. It's stuff like that that gets people mapping. As a board we should encourage companies to enable mappers by having them build useful things and share data.
All of this should lead to us getting from ~250 daily active mappers in the US to 2x that in the next year. This is what I want to accomplish with the board on my next term.