Recent diary entries
OpenStreetMap in 2007 and today...
Mexico released a huge amount of open data not too long ago.
A lot of this data is geospatial, so I say yummie! Alex Barth wrote about this data, that comes from the Mexican national statistical agency INEGI, on his diary before, with a nifty map to show how rich this data is:
(My mediocre animated GIF skills really don't do it justice - check out Alex's blog post to see an interactive map.)
So now the question becomes: how do we get some (or all?) of this data into OSM? This is not straightforward - OSM already has rich data in many places in Mexico we would definitely want to keep.
Here at the Telenav OSM team, we have come up with an answer to this question. We call it Cygnus - The Bringer of Balance. Let me explain in a few visuals what Cygnus does.
Consider this area in the Aguascalientes region. There is some OSM data there:
If we look at the Bing aerial images, we can clearly see that there's an entire village there that is not mapped though!
But INEGI has most if not all the roads in this village in their now open dataset Conjunto de Datos Vectoriales de Carreteras y Vialidades Urbanas.
After converting the original data attributes mapped to OSM-appropriate tagging, saving the result as an OSM file, and loading it into JOSM, it looks like this:
OK, that is nice, but we still have two separate layers that are unconnected, and even if we merge them, we will still have to manually resolve duplicate ways and connections between the ways from OSM and those from the INEGI data.
This is where Cygnus comes in - a new conflation technology we developed at Telenav specifically to tackle this.
Cygnus, as the Bringer of Balance, takes as its input a base OSM file in PBF format, as well as what we call an 'enhancing' file, also in PBF format. It will then compare and conflate the two inputs and output one JOSM XML file that can be merged with OSM base data straight away. This detail shows the merged layers with all the INEGI 'imported' ways connected to the pre-existing OSM way:
Even though Cygnus does a pretty amazing job merging OSM data with an 'enhancing' layer, you will still need to check the result before you upload. Take this example here:
highway=secondary was the pre-existing OSM data, and the
highway=residential; oneway=yes came from INEGI data. It is clear from Bing imagery that the two ways should be connected, yet they are not. Cygnus has a (tweakable) distance threshold it uses when it decides if two ways should be connected or not. In this case, the INEGI way was too far away, so it remained disconnected.
There are a few other things to consider when you work with a Cygnus-produced change file:
- Cygnus will never degrade existing OSM data
- When both OSM and the enhancing layer have the same way, Cygnus will always keep the original OSM geometry, but it will optionally import
nameand other useful tags.
We are currently working on an import plan for INEGI data that will make heavy use of this new technology. More about this very soon! In the mean time, watch our SOTM US talk on the INEGI data and OSM in Mexico.
UPDATE We are working on getting the source code out as Open Source as well. I will post with more details when I have them.
ScoutSigns is a JOSM plugin developed by my colleagues here at Telenav that displays speed limit signs (and a few other types) as a layer in JOSM. These signs are detected by users of our global Scout app if they opted in and have their phone mounted on the windshield. This is a pretty good source of sign information ready to be mapped!
JOSM showing Scout detected signs in ScoutSigns in Hamburg, Germany
Recently, Mapillary started detecting all kinds of signs in their millions of user-uploaded street view images. What would make more sense than collaborating with them to display their detected traffic signs right along with ours in Scout Signs? That's exactly what we did!
JOSM showing Scout (red) and Mapillary (blue) detected signs in ScoutSigns in Hamburg, Germany
With this new release, JOSM mappers can unleash the power of the fast growing Mapillary imagery right in your favorite OSM editor! I have updated the ScoutSigns manual with some details about the new Mapillary functionality.
I am excited to have worked with Mapillary to expose their fantastic number of recognized signs to the power JOSM mapping community! I am going to make this a brief blog post because I want to do some more speed limit mapping :)
If you have ScoutSigns installed already, it will update automatically the next time you start JOSM. If you haven't tried ScoutSigns yet, look for it in the JOSM Plugins preference pane.
I think I have mentioned before that the Telenav OSM team is always thinking of more ways to help improve OpenStreetMap. Something we have recently started looking into is supplying proposed changes directly as JOSM compatible XML files. This way, we could propose changes and additions to the map in chunks, instead of one by one like we have done for MapRoulette thus far.
So looking into the JOSM file format, we created a small (hypothetical) test file that just adds some ways that connect to the current network. Shown below is the process of merging that test file with existing OSM data in three stages: Base OSM layer, change layer overlaid, and finally the merged result.
Did you just notice something STRANGE here? When I made the change layer visible, some but not all the new ways appeared on the map - but after merging, the ways all appear. What makes this even weirder is that I can select the invisible way in the change layer, see and change tags etc. Is this a JOSM bug or is there something about the JOSM file format I do not understand?
If you want to try it for yourself, here is a zip file with both the base OSM data and the proposed change file.
Dual carriageways, divided highways... Whatever you call them, this is what they look like:
In OSM however, they are often mapped like this:
Screenshot from JOSM, showing Bing imagery
This is surely better than nothing, but what we really want to see is an accurate representation of the road as it is in reality. Not only because we're perfectionistic OSM mappers :) but also because there are practical implications to this simplified representation.
Here is an obvious example. Say you need to be at the place by the pink arrow, and you're coming from the southeast there:
Annotated screenshot from JOSM, showing Bing imagery
If this road were mapped as a single way, any navigation app would tell you to just turn left, which is clearly not an option. Also, turning on to the main road, navigation could tell you to turn left which is equally impossible.
There are still a lot of roads in the United States that are mapped as single ways but are in reality divided highways. Here at Telenav, where we use OSM for the Scout navigation app, we want all these roads to be properly mapped as dual ways. This is pretty tedious work (perhaps I will spend a separate diary entry on some of the mappy-grappy details) which is probably why it is not done in the first place, so our internal editing team has taken this on themselves.
fixme=dual carriageway around Memphis, Tenn. (Overpass Turbo). Map ©OpenStreetMap contributors
We don't use any magic to identify the candidate ways. We just search the OSM database for
fixme tags that have a value of
dual carrigeway or some variation thereof. This gave us a total of around 6,000 ways to look at. Prioritizing
primary, we were able to limit this to around 1,500 ways to look at for now.
We expect to complete this work by mid-February, and once we do, anyone using OSM data for navigation will see much improved results for the affected areas!
As always, if you encounter our work on this project in your area, please share your feedback, using the great new changeset comment feature for example. Or just drop me a line!
The Battlegrid had been down for a while and I have had no time to look at it really. Until today! The Battlegrid is back up. And it now compares with the latest TIGER 2014 data, so bright cells should more often be a treasure trove for real problems.
If this is the first you hear about the Battlegrid, read much more about it in the original announcement blog post.
What are you waiting for!? Fix some cells, head over to battlegrid.us!
Image credit: AARoads
Here at Telenav, our editing team has been working tirelessly on adding signposts to OSM for the past months. Today, I am happy and excited to report that we have completed this major editing project! We added signpost information to over 9,500 exits across the United States, covering more than 25 major metropolitan areas and a few important connecting freeways as well:
Wait - that is not 'complete', is it? Well, you are right. We haven't covered every exit in the United States, not by a long shot. We have only a limited number of in house mappers, and a lot of things we want to achieve. So we prioritize the areas where we believe most people will benefit from our work. (On a personal note: that leaves me empty-handed out here in Salt Lake City - I guess I have to go out and map myself!)
We have learned a lot in the process, and we hope to contribute some of these learnings back as well. We already made quite a few improvements to the wiki pages for the
destination tags. Now that this project is complete, we will sit down and revisit these pages and add more clarification where needed. If you are interested in more tagging specifics, please see my earlier blog post on
destination. (And don't forget to read through the comments!)
I want to thank a few folks for making this happen. First and foremost our Telenav Mappers team, with lots of guidance from our resident road geek Robert. But also all of you in the community for giving feedback, calling us out on the occasional errors, and working with us to improve signpost mapping conventions. I feel proud to be a member of a great mapping community!
Scout (USA) has been powered by OpenStreetMap data for about 5 months now. From day one, Scout users have been able to report navigation errors to us:
We have since received thousands of reports and we have worked hard to review them and learn from the feedback we get from our users. We have manually submitted OSM notes where we could not resolve the issue ourselves, and we have fixed dozens of map issues based on the incoming reports. But we felt that we could do a better job closing the feedback loop between Scout users and the OSM community. We have now taken another big step towards this goal with direct OSM Notes creation from Scout US.
Some Scout Feedback will automatically become OSM Notes
As of last week, certain kinds of feedback we get from our Scout US community will be posted to OSM Notes automatically. We apply a smart filter to make sure that only reports that we think the OSM community can fix get posted. For example, 'Destination Incorrect' reports will not make it to OSM, because Scout does not use OSM data to locate addresses and businesses. And we will only forward reports where the Scout user entered a sensible looking comment.
What happens when a Scout user posts a note that passes our filter? Let me walk you through the process. The screenshots are from an actual note posted by a Scout user a few days ago.
Post to OSM
The report is translated into an OSM note and posted to OSM immediately. We take the location of the report as the note location, and add the report comments as the note text.
The note will also contain a link to another page that contains more information that may help you resolve the issue. We include the a part of the GPS trace of the user before and after they submitted the report (anonymized to remove any personally identifiable information of the user) and the OSM map data version used when the Scout driver submitted their report:
As soon as the report is posted to OSM, we send the Scout user an email telling them that their report is being reviewed by the OSM community. Of course, we also encourage them to fix the problem themselves and include a link to the note.
The Scout report system monitors the OSM notes it posted and kicks in as soon as they are closed. (In this case, I resolved the problem myself by adding a turn restriction and tagging a link road as such.) We will send another email to the Scout user telling them that their note has been closed.
Our objective with this new feedback loop is for the Scout and OpenStreetMap communities to connect. With Scout's switch to OSM last April, all of a sudden we gained many more eyes on the OpenStreetMap data. This is an important step in feeding the intelligence we get out of that back to the OSM community. We plan to continue down this road and find more ways for the Scout community to help out in keeping OSM in great shape!
Making a MapRoulette challenge has always been a bit of a, well, challenge. I wrote a tutorial in an attempt to explain the process, but in the end, it is still more work than I would like it to be, and more technical than I would like it to be, too.
So I made something new: the Magical MapRoulette Machine!
The MMM lets you create a MapRoulette challenge interactively. Just answer a few simple questions and your own challenge can be live in minutes. The key component, the part that actually defines which OSM ways or nodes you want to have fixed, is an Overpass API query. If you are not familiar with writing queries in the Overpass Query Language, that would be the only thing you would need to learn a little bit about.
How do I use the Machine?
First download and install it using the instructions on the project page.
Once you have the Machine on your machine, you can just run it in interactive mode by calling it like this:
./mmm.py. It will then ask you a series of questions about things like the Title, Instruction, Help text, etcetera. All those bits are explained in the MapRoulette Tutorial.
Next, it will ask you which MapRoulette server to use. It will default to the dev server, which is a great place to start. It is publicly accessible so you can use it to 'battle-test' your Challenge idea. It will also ask if this is a new Challenge or not. Unless you are uploading your Challenge for the first time, you should accept the default, which is to update an existing challenge.
(By now you will have noticed that there are default values for each question. These represent an example challenge for shops and restaurants without opening hours around Salt Lake City, Utah, USA. They are just meant to give you some more guidance.)
Next up is the most interesting question: which OSM nodes / ways will actually represent your tasks? You tell the Machine by way of an Overpass query.
Let's look at that a little more closely. Consider this simple example. Say I am a Dutch mapper and I want to create a challenge to fill in all missing
lanes on motorways in and around the Netherlands. The Overpass query for that is fairly straightforward.
The Overpass QL looks intimidating at first, but the basics are easy to learn. And for the Magical MapRoulette Machine, you only need the part that actually queries the nodes (or ways). The rest is taken care of internally.
You only need the highlighted part
Once you have the Overpass Query, you are 95% of the way there!
The Machine will now talk to the Overpass API and retrieve your tasks. Depending on the size of the area you are querying, and the complexity of the query, this could take a little while. The machine will eventually respond, and ask you to confirm one more time:
At the bottom it will give you the link to your new challenge! Yay! (The example one is online as well to prove that it really works!
Give it a try and let me know what you think!
I am headed to Buenos Aires for SOTM 2014 tomorrow, and reading up on money and general travel tips. I found this page on bringing and exchanging money. Some of the tips in there:
- Bring cash, preferably in US$50 / US$100 bills
- Pay in US$ whenever you can.
- There is an unofficial 'blue dollar' rate that is much better than the official US$ / Peso rate.
I have never been to Argentina before, and I have no idea if these are good tips. Perhaps an Argentinean mapper can shed some light on this?
See you at SOTM!
Today is another being-part-of-the-solution kind of day. We (a few colleages at Telenav were involved in this) took the output of the Battle Grid comparison we use to generate the Battle Grid cells, divided them into more and less severe cases based on the road class and amount of difference between TIGER and OSM, and created a MapRoulette challenge out of the more interesting cases. This will make it easier for anyone to help fix antiquated, bad TIGER data anywhere.
This is what a typical case would look like in MapRoulette:
Loading this case in iD, it's plain to see that this road is mis-aligned pretty badly:
Easy to fix! Just drag the nodes that make up the misaligned way to match the aerial image:
To point out these errors, we compare TIGER 2014 against OSM data. This gets us mostly good results, but it's not perfect. In most cases where MapRoulette will point out a non-error, the TIGER data is still wrong. This is the case in Alaska more than anywhere else, it seems.
TIGER is wrong here, even though it's hard to see with the low resolution satellite imagery...
The number of non-errors is small enough to not be too annoying, I hope. If it is let me know, and I can see what more we can do to improve this challenge.
Not all at once
There are a lot of Mismatched Ways. A lot. Most of them, way more than 90%, are local and low speed roads, mapped to
residential in OSM when TIGER was imported. Of course, these all need to be fixed eventually, but let's focus on the more important roads first. Based on how it goes, I will gradually add more cases.
I look forward to hearing from you about this new challenge! I am pretty excited about it!
Together, we resolved more than 100 thousand suspected 'sharp angle' errors in the Ways Needing Smoothing challenge, and just under 25 thousand crossing ways in the Crossing Ways challenge.
So at least for the United States, we now have much less of this:
.. and many less situations where two ways cross but should really have an intersection node. (Sorry, in this case an image does not say more than a thousand words..)
Not all suspected cases turned out to be real errors. For the crossing ways, about two thirds of the cases were eventually marked as 'fixed' by MapRoulette users:
For the Ways Needing Smoothing, the fixed rate was much lower, about 40%:
This low rate can be explained in part by MapRoulette users fixing more than they are asked to fix! What I heard a lot is that mappers would load a smoothing task in their browser, and ending up spending a lot of time cleaning up the area around the error - often resolving other MapRoulette tasks without even knowing it. So it's not as bad as it looks, and I received mostly really positive feedback about this challenge.
I have a new challenge in the pipeline that also has to do with TIGER fixup - a separate diary is forthcoming about that!
As always, if you have a good idea for a challenge, don't hesitate to get in touch with me so I can help you get started! Better still, subscribe to the MapRoulette mailing list and share your ideas with the MapRoulette community!
There is a well-established definition for tagging National Parks in OpenStreetMap.
Hooge Veluwe National Park Image credit: Wikipedia
A national park is a relatively large area of land declared by a government (just as
boundary=administrativeare declared/recognised by governments), to be set aside for human recreation and enjoyment, animal and environmental protection
would also apply to the many State Parks in the United States, and perhaps provincial and state parks in other countries as well. However, the
boundary=national_park page does not give any specific guidance. I am about to map some state parks in my home state Utah, and I will tag them with
boundary=national_park for now because they fit the definition.
How do you map state or provincial parks in your area?
Playing MapRoulette, you get dropped in all these random locations on earth. This really makes my mind wander sometimes.
Here is Cambridge, Ohio:
Image credit: Wikipedia
A nice enough town. Probably friendly people. BUT NOT VERY MANY MAPPERS AMONG THEM:
A fellow mapper suggested this simple & brilliant idea: OSM Sister Towns. Why don't we team up towns (and cities) in need of mapping help with towns elsewhere in the world that have mapping forces to spare? They could use the Tasking Manager, MapRoulette, or just iD / JOSM with aerial imagery to help out the town in need. I foresee face to face meet-ups at SOTM and new global bonds of mapping friendship!
Sometimes I wonder if those TIGER surveyors were just having a grand old time making up things as they drove past and casually looked out of the window of their cars...
This whole area needs lots of work - if you want to help out, that would be great!
(I have no personal or other connection to the area, but got dropped there by the MapRoulette 'Ways Needing Smoothing' challenge - freshly cleaned up, so most cases you get should now be real things to fix!
I joined OpenStreetMap in June 2007. A lot has changed since then!
I'm seeing this all the time now when trying to use iD on my Mac!
What's up? There's a recent bug that seems related. If you're experiencing the same thing, see if you can weigh in there.
It looks like there's a more recent version of iD up here, give it a try if you get crashes as well.
MapRoulette silently added a new feature a while ago: the ability to select an area you want to work in. This is one of the most requested features in MapRoulette and I am pretty excited to publicly announce it!
Here is how it works. Go to MapRoulette and select a new challenge. You will see this screen:
Notice the button at the top? It says what it does on the tin - it lets you select a work area that will be remembered until you cancel it. (I will show you how to do that later.) When you click the button, the challenge selection window disappears and you will get this notification:
It tells you to click on the map to set the center of your work area. It will then draw a circle that defines your work area. The initial radius of the circle is determined by the current map zoom, so if you want to define a large area, try zooming out first. Here, I zoomed out to the United States and clicked somewhere in Utah. This gives me a work area of about the size of the state:
Right now, only circular areas are supported.
You can change the radius using the + and - keys on your keyboard. You can click anywhere else on the map to set a new center for your area.
Once you're satisfied with the area you have set, click the notification in the top left to confirm your selection. The challenge selection window will re-appear, but will show only challenges that actually have work in your area, and tell you how much:
(Right now, there is an annoying little bug that will cause MapRoulette to first tell you that everything is done and you need to pick a different challenge. I hope to squish that really soon. Just click OK.)
As you see, there are only two challenges in my area that need work! Yay. If I select one of them, sure enough the next task I get is in Utah:
Resetting your work area is pretty much the same process. Instead of selecting an area like I described before, just click the notification without doing anything on the map, and you will be back to the old behavior where your work area is the entire world.
I hope you enjoy this new feature! Let me know if you have any questions.
Let me update you on some of the goings on at Telenav related to OSM.
Scout OSM data update
First off, we released new OSM data for the US Scout app on July 15th. The OSM data is from July 3rd. No app update is required; just launch Scout and it will automatically use the latest data. We are currently updating the OSM data that drives Scout about every other week.
Note that the Scout International app (formerly the Skobbler GPS Navigation app) has its own data update regimen that is independent from the US version of Scout.
We continue to improve OSM data for the United States as well. We are focusing on signpost information right now (the information that you see on the freeway exit signs).
Image from AARoads
We are adopting the well-established
destination= convention instead of the US-specific, limited and poorly documented
exit_to convention. We are using AARoads as our primary source, for which they kindly gave permission. We are starting in Texas. Let me know if you have any questions about the editing work we are doing!
screenshot from the awesome achavi diff viewer
If you are interested in the background of
destination, please see my previous diary entry about this topic.
Alright, that's it for now. I will write up updates like this one more often going forward!