Recent diary entries
Image credit: AARoads
Here at Telenav, our editing team has been working tirelessly on adding signposts to OSM for the past months. Today, I am happy and excited to report that we have completed this major editing project! We added signpost information to over 9,500 exits across the United States, covering more than 25 major metropolitan areas and a few important connecting freeways as well:
Wait - that is not 'complete', is it? Well, you are right. We haven't covered every exit in the United States, not by a long shot. We have only a limited number of in house mappers, and a lot of things we want to achieve. So we prioritize the areas where we believe most people will benefit from our work. (On a personal note: that leaves me empty-handed out here in Salt Lake City - I guess I have to go out and map myself!)
We have learned a lot in the process, and we hope to contribute some of these learnings back as well. We already made quite a few improvements to the wiki pages for the
destination tags. Now that this project is complete, we will sit down and revisit these pages and add more clarification where needed. If you are interested in more tagging specifics, please see my earlier blog post on
destination. (And don't forget to read through the comments!)
I want to thank a few folks for making this happen. First and foremost our Telenav Mappers team, with lots of guidance from our resident road geek Robert. But also all of you in the community for giving feedback, calling us out on the occasional errors, and working with us to improve signpost mapping conventions. I feel proud to be a member of a great mapping community!
Scout (USA) has been powered by OpenStreetMap data for about 5 months now. From day one, Scout users have been able to report navigation errors to us:
We have since received thousands of reports and we have worked hard to review them and learn from the feedback we get from our users. We have manually submitted OSM notes where we could not resolve the issue ourselves, and we have fixed dozens of map issues based on the incoming reports. But we felt that we could do a better job closing the feedback loop between Scout users and the OSM community. We have now taken another big step towards this goal with direct OSM Notes creation from Scout US.
Some Scout Feedback will automatically become OSM Notes
As of last week, certain kinds of feedback we get from our Scout US community will be posted to OSM Notes automatically. We apply a smart filter to make sure that only reports that we think the OSM community can fix get posted. For example, 'Destination Incorrect' reports will not make it to OSM, because Scout does not use OSM data to locate addresses and businesses. And we will only forward reports where the Scout user entered a sensible looking comment.
What happens when a Scout user posts a note that passes our filter? Let me walk you through the process. The screenshots are from an actual note posted by a Scout user a few days ago.
Post to OSM
The report is translated into an OSM note and posted to OSM immediately. We take the location of the report as the note location, and add the report comments as the note text.
The note will also contain a link to another page that contains more information that may help you resolve the issue. We include the a part of the GPS trace of the user before and after they submitted the report (anonymized to remove any personally identifiable information of the user) and the OSM map data version used when the Scout driver submitted their report:
As soon as the report is posted to OSM, we send the Scout user an email telling them that their report is being reviewed by the OSM community. Of course, we also encourage them to fix the problem themselves and include a link to the note.
The Scout report system monitors the OSM notes it posted and kicks in as soon as they are closed. (In this case, I resolved the problem myself by adding a turn restriction and tagging a link road as such.) We will send another email to the Scout user telling them that their note has been closed.
Our objective with this new feedback loop is for the Scout and OpenStreetMap communities to connect. With Scout's switch to OSM last April, all of a sudden we gained many more eyes on the OpenStreetMap data. This is an important step in feeding the intelligence we get out of that back to the OSM community. We plan to continue down this road and find more ways for the Scout community to help out in keeping OSM in great shape!
Making a MapRoulette challenge has always been a bit of a, well, challenge. I wrote a tutorial in an attempt to explain the process, but in the end, it is still more work than I would like it to be, and more technical than I would like it to be, too.
So I made something new: the Magical MapRoulette Machine!
The MMM lets you create a MapRoulette challenge interactively. Just answer a few simple questions and your own challenge can be live in minutes. The key component, the part that actually defines which OSM ways or nodes you want to have fixed, is an Overpass API query. If you are not familiar with writing queries in the Overpass Query Language, that would be the only thing you would need to learn a little bit about.
How do I use the Machine?
First download and install it using the instructions on the project page.
Once you have the Machine on your machine, you can just run it in interactive mode by calling it like this:
./mmm.py. It will then ask you a series of questions about things like the Title, Instruction, Help text, etcetera. All those bits are explained in the MapRoulette Tutorial.
Next, it will ask you which MapRoulette server to use. It will default to the dev server, which is a great place to start. It is publicly accessible so you can use it to 'battle-test' your Challenge idea. It will also ask if this is a new Challenge or not. Unless you are uploading your Challenge for the first time, you should accept the default, which is to update an existing challenge.
(By now you will have noticed that there are default values for each question. These represent an example challenge for shops and restaurants without opening hours around Salt Lake City, Utah, USA. They are just meant to give you some more guidance.)
Next up is the most interesting question: which OSM nodes / ways will actually represent your tasks? You tell the Machine by way of an Overpass query.
Let's look at that a little more closely. Consider this simple example. Say I am a Dutch mapper and I want to create a challenge to fill in all missing
lanes on motorways in and around the Netherlands. The Overpass query for that is fairly straightforward.
The Overpass QL looks intimidating at first, but the basics are easy to learn. And for the Magical MapRoulette Machine, you only need the part that actually queries the nodes (or ways). The rest is taken care of internally.
You only need the highlighted part
Once you have the Overpass Query, you are 95% of the way there!
The Machine will now talk to the Overpass API and retrieve your tasks. Depending on the size of the area you are querying, and the complexity of the query, this could take a little while. The machine will eventually respond, and ask you to confirm one more time:
At the bottom it will give you the link to your new challenge! Yay! (The example one is online as well to prove that it really works!
Give it a try and let me know what you think!
I am headed to Buenos Aires for SOTM 2014 tomorrow, and reading up on money and general travel tips. I found this page on bringing and exchanging money. Some of the tips in there:
- Bring cash, preferably in US$50 / US$100 bills
- Pay in US$ whenever you can.
- There is an unofficial 'blue dollar' rate that is much better than the official US$ / Peso rate.
I have never been to Argentina before, and I have no idea if these are good tips. Perhaps an Argentinean mapper can shed some light on this?
See you at SOTM!
Today is another being-part-of-the-solution kind of day. We (a few colleages at Telenav were involved in this) took the output of the Battle Grid comparison we use to generate the Battle Grid cells, divided them into more and less severe cases based on the road class and amount of difference between TIGER and OSM, and created a MapRoulette challenge out of the more interesting cases. This will make it easier for anyone to help fix antiquated, bad TIGER data anywhere.
This is what a typical case would look like in MapRoulette:
Loading this case in iD, it's plain to see that this road is mis-aligned pretty badly:
Easy to fix! Just drag the nodes that make up the misaligned way to match the aerial image:
To point out these errors, we compare TIGER 2014 against OSM data. This gets us mostly good results, but it's not perfect. In most cases where MapRoulette will point out a non-error, the TIGER data is still wrong. This is the case in Alaska more than anywhere else, it seems.
TIGER is wrong here, even though it's hard to see with the low resolution satellite imagery...
The number of non-errors is small enough to not be too annoying, I hope. If it is let me know, and I can see what more we can do to improve this challenge.
Not all at once
There are a lot of Mismatched Ways. A lot. Most of them, way more than 90%, are local and low speed roads, mapped to
residential in OSM when TIGER was imported. Of course, these all need to be fixed eventually, but let's focus on the more important roads first. Based on how it goes, I will gradually add more cases.
I look forward to hearing from you about this new challenge! I am pretty excited about it!
Together, we resolved more than 100 thousand suspected 'sharp angle' errors in the Ways Needing Smoothing challenge, and just under 25 thousand crossing ways in the Crossing Ways challenge.
So at least for the United States, we now have much less of this:
.. and many less situations where two ways cross but should really have an intersection node. (Sorry, in this case an image does not say more than a thousand words..)
Not all suspected cases turned out to be real errors. For the crossing ways, about two thirds of the cases were eventually marked as 'fixed' by MapRoulette users:
For the Ways Needing Smoothing, the fixed rate was much lower, about 40%:
This low rate can be explained in part by MapRoulette users fixing more than they are asked to fix! What I heard a lot is that mappers would load a smoothing task in their browser, and ending up spending a lot of time cleaning up the area around the error - often resolving other MapRoulette tasks without even knowing it. So it's not as bad as it looks, and I received mostly really positive feedback about this challenge.
I have a new challenge in the pipeline that also has to do with TIGER fixup - a separate diary is forthcoming about that!
As always, if you have a good idea for a challenge, don't hesitate to get in touch with me so I can help you get started! Better still, subscribe to the MapRoulette mailing list and share your ideas with the MapRoulette community!
There is a well-established definition for tagging National Parks in OpenStreetMap.
Hooge Veluwe National Park Image credit: Wikipedia
A national park is a relatively large area of land declared by a government (just as
boundary=administrativeare declared/recognised by governments), to be set aside for human recreation and enjoyment, animal and environmental protection
would also apply to the many State Parks in the United States, and perhaps provincial and state parks in other countries as well. However, the
boundary=national_park page does not give any specific guidance. I am about to map some state parks in my home state Utah, and I will tag them with
boundary=national_park for now because they fit the definition.
How do you map state or provincial parks in your area?
Playing MapRoulette, you get dropped in all these random locations on earth. This really makes my mind wander sometimes.
Here is Cambridge, Ohio:
Image credit: Wikipedia
A nice enough town. Probably friendly people. BUT NOT VERY MANY MAPPERS AMONG THEM:
A fellow mapper suggested this simple & brilliant idea: OSM Sister Towns. Why don't we team up towns (and cities) in need of mapping help with towns elsewhere in the world that have mapping forces to spare? They could use the Tasking Manager, MapRoulette, or just iD / JOSM with aerial imagery to help out the town in need. I foresee face to face meet-ups at SOTM and new global bonds of mapping friendship!
Sometimes I wonder if those TIGER surveyors were just having a grand old time making up things as they drove past and casually looked out of the window of their cars...
This whole area needs lots of work - if you want to help out, that would be great!
(I have no personal or other connection to the area, but got dropped there by the MapRoulette 'Ways Needing Smoothing' challenge - freshly cleaned up, so most cases you get should now be real things to fix!
I joined OpenStreetMap in June 2007. A lot has changed since then!
I'm seeing this all the time now when trying to use iD on my Mac!
What's up? There's a recent bug that seems related. If you're experiencing the same thing, see if you can weigh in there.
It looks like there's a more recent version of iD up here, give it a try if you get crashes as well.
MapRoulette silently added a new feature a while ago: the ability to select an area you want to work in. This is one of the most requested features in MapRoulette and I am pretty excited to publicly announce it!
Here is how it works. Go to MapRoulette and select a new challenge. You will see this screen:
Notice the button at the top? It says what it does on the tin - it lets you select a work area that will be remembered until you cancel it. (I will show you how to do that later.) When you click the button, the challenge selection window disappears and you will get this notification:
It tells you to click on the map to set the center of your work area. It will then draw a circle that defines your work area. The initial radius of the circle is determined by the current map zoom, so if you want to define a large area, try zooming out first. Here, I zoomed out to the United States and clicked somewhere in Utah. This gives me a work area of about the size of the state:
Right now, only circular areas are supported.
You can change the radius using the + and - keys on your keyboard. You can click anywhere else on the map to set a new center for your area.
Once you're satisfied with the area you have set, click the notification in the top left to confirm your selection. The challenge selection window will re-appear, but will show only challenges that actually have work in your area, and tell you how much:
(Right now, there is an annoying little bug that will cause MapRoulette to first tell you that everything is done and you need to pick a different challenge. I hope to squish that really soon. Just click OK.)
As you see, there are only two challenges in my area that need work! Yay. If I select one of them, sure enough the next task I get is in Utah:
Resetting your work area is pretty much the same process. Instead of selecting an area like I described before, just click the notification without doing anything on the map, and you will be back to the old behavior where your work area is the entire world.
I hope you enjoy this new feature! Let me know if you have any questions.
Let me update you on some of the goings on at Telenav related to OSM.
Scout OSM data update
First off, we released new OSM data for the US Scout app on July 15th. The OSM data is from July 3rd. No app update is required; just launch Scout and it will automatically use the latest data. We are currently updating the OSM data that drives Scout about every other week.
Note that the Scout International app (formerly the Skobbler GPS Navigation app) has its own data update regimen that is independent from the US version of Scout.
We continue to improve OSM data for the United States as well. We are focusing on signpost information right now (the information that you see on the freeway exit signs).
Image from AARoads
We are adopting the well-established
destination= convention instead of the US-specific, limited and poorly documented
exit_to convention. We are using AARoads as our primary source, for which they kindly gave permission. We are starting in Texas. Let me know if you have any questions about the editing work we are doing!
screenshot from the awesome achavi diff viewer
If you are interested in the background of
destination, please see my previous diary entry about this topic.
Alright, that's it for now. I will write up updates like this one more often going forward!
Mapping opening hours correctly can be a pain; the format for the
opening_hours key can be hard to remember. I didn't use to do it much for that reason. Until I recently discovered that there's a JOSM plugin for that! It works really well, let me show you how I added hours of a local restaurant in seconds.
First, of course, make sure you have the plugin installed:
Then, select the thing you want to add opening hours for:
And select 'Edit Opening Hours' from the Data menu (or use the keyboard shortcut).
If the plugin detects an existing
opening_hours tag, it will offer to edit that, or you can create it. It will then offer a graphical interface for setting the hours:
You can move and drag blocks as you would expect. The plugin will parse it in the expected format. A nice little extra is that it supports the 'open end' hours in the format
17:00+ - useful for places that don't have a set closing hour but instead say 'open till late' or something similar.
Perhaps old news to y'all, but it made me happy :)
A Word Of Caution
Several people have pointed out (link is to the German forum) that the plugin is limited in what it can do. It will not cover most complex cases, like exceptions for public holidays and month to month variations) and does not seem to properly cover some of the simpler cases.
I still feel that it is good for a lot of basic opening hours tagging, but please review the opening_hours specification.
There is also a validator tool that I don't quite understand how to use, perhaps someone can explain in the comments?
I like the Watchlist on the wiki - it helps me keep track of changes to pages I care about. When you are logged in, you will see a link to your Watchlist in the top right corner:
If you click on it, you get an overview of changes to the pages you added to your watchlist:
You can filter the output in various ways to make the result less cluttered.
Adding pages to your watchlist is simple - just click the star at the top of a page:
The biggest drawback is that you actually have to go to the page. I would rather be notified of changes on my phone.
This is where the Atom feed comes in. On your Watchlist page, you will find it in the left margin:
The Atom link provides the latest items on your watchlist as a feed, meant for software to do something useful with. Much like RSS. Actually, if you use an RSS reader, you can probably add the Atom link to your reader.
I don't use RSS readers, but I do use IFTTT, or If This Than That. IFTTT is a very smart web site that you can feed all kinds of information from the internet, and perform actions on it based on triggers. For example, it can send you a text message whenever it is going to rain, based on weather data it gathers. IFTTT calls these things Recipes. People have come up with tons of useful recipes, and IFTTT is constantly adding interesting input options.
Let me show you how I created a simple IFTTT Recipe to get a notification on my phone every time someone updates an OSM wiki page I am watching.
First, if you don't have an account with IFTTT yet, create one.
Once you are logged in, you will be able to create new Recipes:
The first step in creating a Recipe is defining your 'this' or input:
Click 'this' and choose 'Feed' from the list:
You can choose to pull this trigger on each new Feed item or just one that matches a certain string.
Choose 'new feed item'. In the next step, you paste the Atom URL:
Then click to Create the Trigger.
This is the 'this' step that takes care of the input. IFTTT will monitor your Watchlist through the Atom feed and pull this Trigger every time it sees something new. I am not quite sure how often it goes and looks, but it's about every 10-15 minutes in my experience.
Next is the 'that' step, or what do you want to do when the Trigger is pulled?
There are a lot of options, most of which you will need to 'activate' by downloading an app for your phone, linking another account, or something similar so that IFTTT knows what to talk to on your end. IFTTT has native apps for iOS and Android that it can talk to to send you notifications. This is probably your easiest option if you want notifications on your phone.
You activate these 'Channels' by downloading the app from your app store and linking it to your IFTTT account. I use a third party notification app called Pushover myself, because it has an API that can receive notifications from elsewhere as well. If you don't care about that, go with the IFTTT apps or even the SMS Channel option. The example here assumes Pushover, but the steps are the same.
Once you have selected a 'that' Channel, you may need to choose between various output options. Pushover for example has two, normal and 'high priority'. In the app, you can have these two types trigger different alert types / sounds.
In the next step, you get to craft what your notification looks like:
The text with a grey background will be replaced by text from the feed when the notification gets sent out. There are some more of these placeholders available under the + button:
Unfortunately, there is no 'preview' available, so sometimes it takes a little guess work to see what the results will look like.
I ended up with this:
The final step is to create the Recipe.
Now, when someone edits a wiki page on the OSM wiki that is on your watchlist, you will know right away!
(Note that these notifications are not about editing wiki pages :) I did not take a screen shot of those yet. The notifications you see here show whenever someone edits OSM in your area - something I wrote about on my old blog a while ago.
Here's an example of a Mapillary image and map context:
I have created an account and did some test shooting from my car, using the iPhone app Mapillary provides. This is OK, but I want better quality images. So I am looking at an action camera I can mount in my car and on my bike.
The Garmin VIRB (Elite edition shown):
The GoPro Hero3 (White shown):
- The VIRB seems to have better battery life (important! but could not find any side by side comparisons)
- The basic GoPro Hero3 is less expensive than the basic VIRB
- The Elite edition of the Garmin VIRB has lots of built in sensors: GPS, altimeter, accelerometer - nice, but I am already carrying a GPS wherever I go anyway.
- Both can do automatic time lapse photos at set intervals, which is crucial.
- The VIRB can do 16MP stills, the most expensive Hero3 can do 12MP. This does not necessarily mean better pictures, of course.
- The most expensive Hero3 comes with a remote, which can be convenient if you don't want to time-lapse. For example on long freeway drives where you just want to capture the signs. The VIRB can be remote controlled by certain Garmin devices like the Fenix 2 watch. Both come with smartphone apps that can control the camera as well. Convenience factor is much less though.
- The VIRB has a built in screen. Not sure if that is actually an advantage. What would you use it for?
- Both have wide angle lenses which is important to capture as much in one shot as possible. Specs are sketchy but it looks like the GoPros have a slightly wider angle.
- GoPro has been at this for a lot longer than Garmin.
Here's a comparison of the video side by side. I wouldn't use video mode as much so this is of limited value, but anyway:
iframe embedding does not seem to work - is there a way?)
From what I can see the GoPro wins out by a small margin: sharper and better colors.
What do you use? Any other cameras to consider?
Here in the U.S., we have been using the convention of tagging exit information on the
highway=junction node using the poorly documented
exit_to tag. The rest of the world has been using
destination=, and now that I start to think about this problem more, I can see why.
Let's take this example of the junction of I-70 and the Baltimore beltway I-695.
(Have you used Mapillary? It's pretty awesome and unlike Google Streetview, OpenStreetMap explicitly does have permission to map from the images. It's pretty easy and fun to contribute your own, too.)
How to tag this so that navigation and other software could make sense of it?
Currently, the only relevant tags on the junction node are
ref:right=91A indicating the exit / junction numbers. There is nothing on the `_link' ways to indicate destination.
One way I have seen people map this information in the US is roughly
exit_to:left=I 695 N;New York;Towson and
exit_to:right=I 695 S;Baltimore;Glen Burnie. This is not great for a number of reasons:
- There is no way to distinguish the
I 695 Setc) from the names reliably.
- The US seems to be the only jurisdiction where
exit_toon the junction node is used, the rest of the world uses
If we were to adopt
destination= in the US like everywhere else, most of the signpost information would be on the
_link ways after the junction node. Here is the situation in OSM:
According to the
destination documentation, the first way would get
destination=New York;Towson and the second way would get
So far so good. But there's a few pieces of information that are not captured yet:
1) The exit (junction) numbers 91A and 91B. Oh wait - these were already on the junction node as
ref:right=91A. This works, but intuitively it would make more sense to have this information on the
_link ways as well. I am not sure how and the wiki does not give any guidance. (Actually it suggests using
ref on the junction node for exit numbers.)
2) The road numbers the exits connect to, in this case I-695 N/S, leading to I-95 N/S.
destination:ref seems to be used for that quite a bit. That would work for I-695 but I don't know how to capture the connection to I-95.
Following the documentation as best I can, this is what I come up with for the highlighted _link ways:
I think this system makes more sense than the
exit_to convention that is poorly documented, doesn't scale well to cover complex cases, and is used nowhere else in the world. The only good reason I can think of why people use it is because it's used much more (about 18,000 times) than
destination= in the U.S. Elsewhere, from what I can see in TagInfo,
destination= is more popular.
We created a new MapRoulette challenge over at Telenav: Crossing Ways. As always with the Telenav-generated challenges, this one is US only, because we currently only process OpenStreetMap data for the United States. We are working on extending support for other regions.
TL;DR --> On to MapRoulette!!
The Crossing Ways challenge is pretty straightforward and should make for an easy challenge for novice mappers looking for something to fix. It detects ways that cross, but neither way is tagged as a bridge or tunnel. Here is an example:
If you open this up in iD or JOSM, you will be able to tell the required resolution from the aerial imagery:
In this case, the ways should be connected as there is clearly an intersection. Another possibility is that there actually should be a bridge or tunnel tag on either of the ways.
In this particular case, you can see that there actually is a node at the intersection, but it does not connect both ways. In JOSM, it displays as a 'skinny node':
In this case, you can use JOSM's 'Join' function (shortcut 'J') to connect both ways. In iD, a node connecting two ways will be grey, and a regular node will be white:
If you use iD, you can just drag an existing node to the intersection to snap the two ways together, or drag a virtual node to the intersection to create a new intersection node. (Virtual nodes appear between two existing nodes and allow you to quickly create a new node on a way, an example shown below.)
There are currently around 19,000 cases in the US alone, so let's get on fixing these annoying bugs in OSM!
From time to time someone will ask me: 'How do I find out who my fellow local mappers are?'
That can be a surprisingly hard question to answer. You can go to your user page and see a few people who have indicated their 'home location' to be near yours. That does not work so well for a variety of reasons:
- Not all people set their home locations. My guess is most people don't.
- People don't update their home locations when they move.
- The home location is not necessarily where that person is mapping.
You can also have a look at Pascal Neis's wonderful 'Overview of OpenStreetMap Contributors aka Who's around me?' map, which gives you a visual impression of mappers in your area based on actual mapping activity. This is much better! But it still does not always give you the complete picture. For example, if I navigate to Salt Lake City, where I have lived and mapped for three years now, I can't find myself on that map. I guess I am mapping too much in other places for Pascal's algorithm to detect me as a local mapper.
Dennis Zielstra's engaging talk 'Where Are The US Mappers At' at State Of The Map US 2014 made me revisit this problem (and dig up some old code, more on that in a bit). He presented an in-depth analysis on user engagement and community activity for top US cities. Following that talk, I had a few interesting discussions focusing on questions like
What to do about those cities that do not have an active local community?
How can we make it easier for people to start organizing their community locally?
If we're going to address these questions, we need a better way to identify who active mappers in our area are, and contact them.
For contacting, we will need the groups functionality on OSM.org that has been under development for a while. Being at SOTM US and catching up with lots of mappers from all over made me eager to get this project going again.
For identifying, I think I can contribute something. I wrote some code on top of osmjs a couple of years ago that I dusted off. It processes a full history planet file for an area and outputs stats for all users who have ever contributed in that area, things like:
- First and last edit
- Total number of edits
- Number of edits still visible in today's data
I ran it on Los Angeles county and this is what came out. As a list, it is already pretty insightful, but you can roll nice visualizations with it as well. Enter the Brave Mappers Of Wherever You Are project that was sitting under the same layer of dust. I ran it on the LA county output with this result.
Brave Mappers shows a timeline of mappers for your area based on historical OSM data. A long bar is a mapper who has been active for a long time, like techlady here:
The users are sorted by how long they have been active in this area. Scrolling down, there are some interesting outliers to be seen. Here is DaveHansenTiger, the original TIGER import account:
As expected, active for a short time right at the beginning of the timeline. The green color of their bar means that they mostly created things (rather than modifying or deleting). The transparency indicates most of their work is no longer in the current data. In this case that is probably a good thing!
Finally, clicking on a user's bar gives some more details on their contributions:
If you want a Brave Mappers page (or just the spreadsheet) for your area, that can be arranged! If you feel up to it, you can clone the OSMQualityMetrics repo and run
UserStats.js yourself on a full history planet for your area, then process it with the script in the bravemappers repo. Or you can send me an OSM POLY file and I will try and make some time to do it for you!