Recent diary entries
Today is another being-part-of-the-solution kind of day. We (a few colleages at Telenav were involved in this) took the output of the Battle Grid comparison we use to generate the Battle Grid cells, divided them into more and less severe cases based on the road class and amount of difference between TIGER and OSM, and created a MapRoulette challenge out of the more interesting cases. This will make it easier for anyone to help fix antiquated, bad TIGER data anywhere.
This is what a typical case would look like in MapRoulette:
Loading this case in iD, it's plain to see that this road is mis-aligned pretty badly:
Easy to fix! Just drag the nodes that make up the misaligned way to match the aerial image:
To point out these errors, we compare TIGER 2014 against OSM data. This gets us mostly good results, but it's not perfect. In most cases where MapRoulette will point out a non-error, the TIGER data is still wrong. This is the case in Alaska more than anywhere else, it seems.
TIGER is wrong here, even though it's hard to see with the low resolution satellite imagery...
The number of non-errors is small enough to not be too annoying, I hope. If it is let me know, and I can see what more we can do to improve this challenge.
Not all at once
There are a lot of Mismatched Ways. A lot. Most of them, way more than 90%, are local and low speed roads, mapped to
residential in OSM when TIGER was imported. Of course, these all need to be fixed eventually, but let's focus on the more important roads first. Based on how it goes, I will gradually add more cases.
I look forward to hearing from you about this new challenge! I am pretty excited about it!
Together, we resolved more than 100 thousand suspected 'sharp angle' errors in the Ways Needing Smoothing challenge, and just under 25 thousand crossing ways in the Crossing Ways challenge.
So at least for the United States, we now have much less of this:
.. and many less situations where two ways cross but should really have an intersection node. (Sorry, in this case an image does not say more than a thousand words..)
Not all suspected cases turned out to be real errors. For the crossing ways, about two thirds of the cases were eventually marked as 'fixed' by MapRoulette users:
For the Ways Needing Smoothing, the fixed rate was much lower, about 40%:
This low rate can be explained in part by MapRoulette users fixing more than they are asked to fix! What I heard a lot is that mappers would load a smoothing task in their browser, and ending up spending a lot of time cleaning up the area around the error - often resolving other MapRoulette tasks without even knowing it. So it's not as bad as it looks, and I received mostly really positive feedback about this challenge.
I have a new challenge in the pipeline that also has to do with TIGER fixup - a separate diary is forthcoming about that!
As always, if you have a good idea for a challenge, don't hesitate to get in touch with me so I can help you get started! Better still, subscribe to the MapRoulette mailing list and share your ideas with the MapRoulette community!
There is a well-established definition for tagging National Parks in OpenStreetMap.
Hooge Veluwe National Park Image credit: Wikipedia
A national park is a relatively large area of land declared by a government (just as
boundary=administrativeare declared/recognised by governments), to be set aside for human recreation and enjoyment, animal and environmental protection
would also apply to the many State Parks in the United States, and perhaps provincial and state parks in other countries as well. However, the
boundary=national_park page does not give any specific guidance. I am about to map some state parks in my home state Utah, and I will tag them with
boundary=national_park for now because they fit the definition.
How do you map state or provincial parks in your area?
Playing MapRoulette, you get dropped in all these random locations on earth. This really makes my mind wander sometimes.
Here is Cambridge, Ohio:
Image credit: Wikipedia
A nice enough town. Probably friendly people. BUT NOT VERY MANY MAPPERS AMONG THEM:
A fellow mapper suggested this simple & brilliant idea: OSM Sister Towns. Why don't we team up towns (and cities) in need of mapping help with towns elsewhere in the world that have mapping forces to spare? They could use the Tasking Manager, MapRoulette, or just iD / JOSM with aerial imagery to help out the town in need. I foresee face to face meet-ups at SOTM and new global bonds of mapping friendship!
Sometimes I wonder if those TIGER surveyors were just having a grand old time making up things as they drove past and casually looked out of the window of their cars...
This whole area needs lots of work - if you want to help out, that would be great!
(I have no personal or other connection to the area, but got dropped there by the MapRoulette 'Ways Needing Smoothing' challenge - freshly cleaned up, so most cases you get should now be real things to fix!
I joined OpenStreetMap in June 2007. A lot has changed since then!
I'm seeing this all the time now when trying to use iD on my Mac!
What's up? There's a recent bug that seems related. If you're experiencing the same thing, see if you can weigh in there.
It looks like there's a more recent version of iD up here, give it a try if you get crashes as well.
MapRoulette silently added a new feature a while ago: the ability to select an area you want to work in. This is one of the most requested features in MapRoulette and I am pretty excited to publicly announce it!
Here is how it works. Go to MapRoulette and select a new challenge. You will see this screen:
Notice the button at the top? It says what it does on the tin - it lets you select a work area that will be remembered until you cancel it. (I will show you how to do that later.) When you click the button, the challenge selection window disappears and you will get this notification:
It tells you to click on the map to set the center of your work area. It will then draw a circle that defines your work area. The initial radius of the circle is determined by the current map zoom, so if you want to define a large area, try zooming out first. Here, I zoomed out to the United States and clicked somewhere in Utah. This gives me a work area of about the size of the state:
Right now, only circular areas are supported.
You can change the radius using the + and - keys on your keyboard. You can click anywhere else on the map to set a new center for your area.
Once you're satisfied with the area you have set, click the notification in the top left to confirm your selection. The challenge selection window will re-appear, but will show only challenges that actually have work in your area, and tell you how much:
(Right now, there is an annoying little bug that will cause MapRoulette to first tell you that everything is done and you need to pick a different challenge. I hope to squish that really soon. Just click OK.)
As you see, there are only two challenges in my area that need work! Yay. If I select one of them, sure enough the next task I get is in Utah:
Resetting your work area is pretty much the same process. Instead of selecting an area like I described before, just click the notification without doing anything on the map, and you will be back to the old behavior where your work area is the entire world.
I hope you enjoy this new feature! Let me know if you have any questions.
Let me update you on some of the goings on at Telenav related to OSM.
Scout OSM data update
First off, we released new OSM data for the US Scout app on July 15th. The OSM data is from July 3rd. No app update is required; just launch Scout and it will automatically use the latest data. We are currently updating the OSM data that drives Scout about every other week.
Note that the Scout International app (formerly the Skobbler GPS Navigation app) has its own data update regimen that is independent from the US version of Scout.
We continue to improve OSM data for the United States as well. We are focusing on signpost information right now (the information that you see on the freeway exit signs).
Image from AARoads
We are adopting the well-established
destination= convention instead of the US-specific, limited and poorly documented
exit_to convention. We are using AARoads as our primary source, for which they kindly gave permission. We are starting in Texas. Let me know if you have any questions about the editing work we are doing!
screenshot from the awesome achavi diff viewer
If you are interested in the background of
destination, please see my previous diary entry about this topic.
Alright, that's it for now. I will write up updates like this one more often going forward!
Mapping opening hours correctly can be a pain; the format for the
opening_hours key can be hard to remember. I didn't use to do it much for that reason. Until I recently discovered that there's a JOSM plugin for that! It works really well, let me show you how I added hours of a local restaurant in seconds.
First, of course, make sure you have the plugin installed:
Then, select the thing you want to add opening hours for:
And select 'Edit Opening Hours' from the Data menu (or use the keyboard shortcut).
If the plugin detects an existing
opening_hours tag, it will offer to edit that, or you can create it. It will then offer a graphical interface for setting the hours:
You can move and drag blocks as you would expect. The plugin will parse it in the expected format. A nice little extra is that it supports the 'open end' hours in the format
17:00+ - useful for places that don't have a set closing hour but instead say 'open till late' or something similar.
Perhaps old news to y'all, but it made me happy :)
A Word Of Caution
Several people have pointed out (link is to the German forum) that the plugin is limited in what it can do. It will not cover most complex cases, like exceptions for public holidays and month to month variations) and does not seem to properly cover some of the simpler cases.
I still feel that it is good for a lot of basic opening hours tagging, but please review the opening_hours specification.
There is also a validator tool that I don't quite understand how to use, perhaps someone can explain in the comments?
I like the Watchlist on the wiki - it helps me keep track of changes to pages I care about. When you are logged in, you will see a link to your Watchlist in the top right corner:
If you click on it, you get an overview of changes to the pages you added to your watchlist:
You can filter the output in various ways to make the result less cluttered.
Adding pages to your watchlist is simple - just click the star at the top of a page:
The biggest drawback is that you actually have to go to the page. I would rather be notified of changes on my phone.
This is where the Atom feed comes in. On your Watchlist page, you will find it in the left margin:
The Atom link provides the latest items on your watchlist as a feed, meant for software to do something useful with. Much like RSS. Actually, if you use an RSS reader, you can probably add the Atom link to your reader.
I don't use RSS readers, but I do use IFTTT, or If This Than That. IFTTT is a very smart web site that you can feed all kinds of information from the internet, and perform actions on it based on triggers. For example, it can send you a text message whenever it is going to rain, based on weather data it gathers. IFTTT calls these things Recipes. People have come up with tons of useful recipes, and IFTTT is constantly adding interesting input options.
Let me show you how I created a simple IFTTT Recipe to get a notification on my phone every time someone updates an OSM wiki page I am watching.
First, if you don't have an account with IFTTT yet, create one.
Once you are logged in, you will be able to create new Recipes:
The first step in creating a Recipe is defining your 'this' or input:
Click 'this' and choose 'Feed' from the list:
You can choose to pull this trigger on each new Feed item or just one that matches a certain string.
Choose 'new feed item'. In the next step, you paste the Atom URL:
Then click to Create the Trigger.
This is the 'this' step that takes care of the input. IFTTT will monitor your Watchlist through the Atom feed and pull this Trigger every time it sees something new. I am not quite sure how often it goes and looks, but it's about every 10-15 minutes in my experience.
Next is the 'that' step, or what do you want to do when the Trigger is pulled?
There are a lot of options, most of which you will need to 'activate' by downloading an app for your phone, linking another account, or something similar so that IFTTT knows what to talk to on your end. IFTTT has native apps for iOS and Android that it can talk to to send you notifications. This is probably your easiest option if you want notifications on your phone.
You activate these 'Channels' by downloading the app from your app store and linking it to your IFTTT account. I use a third party notification app called Pushover myself, because it has an API that can receive notifications from elsewhere as well. If you don't care about that, go with the IFTTT apps or even the SMS Channel option. The example here assumes Pushover, but the steps are the same.
Once you have selected a 'that' Channel, you may need to choose between various output options. Pushover for example has two, normal and 'high priority'. In the app, you can have these two types trigger different alert types / sounds.
In the next step, you get to craft what your notification looks like:
The text with a grey background will be replaced by text from the feed when the notification gets sent out. There are some more of these placeholders available under the + button:
Unfortunately, there is no 'preview' available, so sometimes it takes a little guess work to see what the results will look like.
I ended up with this:
The final step is to create the Recipe.
Now, when someone edits a wiki page on the OSM wiki that is on your watchlist, you will know right away!
(Note that these notifications are not about editing wiki pages :) I did not take a screen shot of those yet. The notifications you see here show whenever someone edits OSM in your area - something I wrote about on my old blog a while ago.
Here's an example of a Mapillary image and map context:
I have created an account and did some test shooting from my car, using the iPhone app Mapillary provides. This is OK, but I want better quality images. So I am looking at an action camera I can mount in my car and on my bike.
The Garmin VIRB (Elite edition shown):
The GoPro Hero3 (White shown):
- The VIRB seems to have better battery life (important! but could not find any side by side comparisons)
- The basic GoPro Hero3 is less expensive than the basic VIRB
- The Elite edition of the Garmin VIRB has lots of built in sensors: GPS, altimeter, accelerometer - nice, but I am already carrying a GPS wherever I go anyway.
- Both can do automatic time lapse photos at set intervals, which is crucial.
- The VIRB can do 16MP stills, the most expensive Hero3 can do 12MP. This does not necessarily mean better pictures, of course.
- The most expensive Hero3 comes with a remote, which can be convenient if you don't want to time-lapse. For example on long freeway drives where you just want to capture the signs. The VIRB can be remote controlled by certain Garmin devices like the Fenix 2 watch. Both come with smartphone apps that can control the camera as well. Convenience factor is much less though.
- The VIRB has a built in screen. Not sure if that is actually an advantage. What would you use it for?
- Both have wide angle lenses which is important to capture as much in one shot as possible. Specs are sketchy but it looks like the GoPros have a slightly wider angle.
- GoPro has been at this for a lot longer than Garmin.
Here's a comparison of the video side by side. I wouldn't use video mode as much so this is of limited value, but anyway:
iframe embedding does not seem to work - is there a way?)
From what I can see the GoPro wins out by a small margin: sharper and better colors.
What do you use? Any other cameras to consider?
Here in the U.S., we have been using the convention of tagging exit information on the
highway=junction node using the poorly documented
exit_to tag. The rest of the world has been using
destination=, and now that I start to think about this problem more, I can see why.
Let's take this example of the junction of I-70 and the Baltimore beltway I-695.
(Have you used Mapillary? It's pretty awesome and unlike Google Streetview, OpenStreetMap explicitly does have permission to map from the images. It's pretty easy and fun to contribute your own, too.)
How to tag this so that navigation and other software could make sense of it?
Currently, the only relevant tags on the junction node are
ref:right=91A indicating the exit / junction numbers. There is nothing on the `_link' ways to indicate destination.
One way I have seen people map this information in the US is roughly
exit_to:left=I 695 N;New York;Towson and
exit_to:right=I 695 S;Baltimore;Glen Burnie. This is not great for a number of reasons:
- There is no way to distinguish the
I 695 Setc) from the names reliably.
- The US seems to be the only jurisdiction where
exit_toon the junction node is used, the rest of the world uses
If we were to adopt
destination= in the US like everywhere else, most of the signpost information would be on the
_link ways after the junction node. Here is the situation in OSM:
According to the
destination documentation, the first way would get
destination=New York;Towson and the second way would get
So far so good. But there's a few pieces of information that are not captured yet:
1) The exit (junction) numbers 91A and 91B. Oh wait - these were already on the junction node as
ref:right=91A. This works, but intuitively it would make more sense to have this information on the
_link ways as well. I am not sure how and the wiki does not give any guidance. (Actually it suggests using
ref on the junction node for exit numbers.)
2) The road numbers the exits connect to, in this case I-695 N/S, leading to I-95 N/S.
destination:ref seems to be used for that quite a bit. That would work for I-695 but I don't know how to capture the connection to I-95.
Following the documentation as best I can, this is what I come up with for the highlighted _link ways:
I think this system makes more sense than the
exit_to convention that is poorly documented, doesn't scale well to cover complex cases, and is used nowhere else in the world. The only good reason I can think of why people use it is because it's used much more (about 18,000 times) than
destination= in the U.S. Elsewhere, from what I can see in TagInfo,
destination= is more popular.
We created a new MapRoulette challenge over at Telenav: Crossing Ways. As always with the Telenav-generated challenges, this one is US only, because we currently only process OpenStreetMap data for the United States. We are working on extending support for other regions.
TL;DR --> On to MapRoulette!!
The Crossing Ways challenge is pretty straightforward and should make for an easy challenge for novice mappers looking for something to fix. It detects ways that cross, but neither way is tagged as a bridge or tunnel. Here is an example:
If you open this up in iD or JOSM, you will be able to tell the required resolution from the aerial imagery:
In this case, the ways should be connected as there is clearly an intersection. Another possibility is that there actually should be a bridge or tunnel tag on either of the ways.
In this particular case, you can see that there actually is a node at the intersection, but it does not connect both ways. In JOSM, it displays as a 'skinny node':
In this case, you can use JOSM's 'Join' function (shortcut 'J') to connect both ways. In iD, a node connecting two ways will be grey, and a regular node will be white:
If you use iD, you can just drag an existing node to the intersection to snap the two ways together, or drag a virtual node to the intersection to create a new intersection node. (Virtual nodes appear between two existing nodes and allow you to quickly create a new node on a way, an example shown below.)
There are currently around 19,000 cases in the US alone, so let's get on fixing these annoying bugs in OSM!
From time to time someone will ask me: 'How do I find out who my fellow local mappers are?'
That can be a surprisingly hard question to answer. You can go to your user page and see a few people who have indicated their 'home location' to be near yours. That does not work so well for a variety of reasons:
- Not all people set their home locations. My guess is most people don't.
- People don't update their home locations when they move.
- The home location is not necessarily where that person is mapping.
You can also have a look at Pascal Neis's wonderful 'Overview of OpenStreetMap Contributors aka Who's around me?' map, which gives you a visual impression of mappers in your area based on actual mapping activity. This is much better! But it still does not always give you the complete picture. For example, if I navigate to Salt Lake City, where I have lived and mapped for three years now, I can't find myself on that map. I guess I am mapping too much in other places for Pascal's algorithm to detect me as a local mapper.
Dennis Zielstra's engaging talk 'Where Are The US Mappers At' at State Of The Map US 2014 made me revisit this problem (and dig up some old code, more on that in a bit). He presented an in-depth analysis on user engagement and community activity for top US cities. Following that talk, I had a few interesting discussions focusing on questions like
What to do about those cities that do not have an active local community?
How can we make it easier for people to start organizing their community locally?
If we're going to address these questions, we need a better way to identify who active mappers in our area are, and contact them.
For contacting, we will need the groups functionality on OSM.org that has been under development for a while. Being at SOTM US and catching up with lots of mappers from all over made me eager to get this project going again.
For identifying, I think I can contribute something. I wrote some code on top of osmjs a couple of years ago that I dusted off. It processes a full history planet file for an area and outputs stats for all users who have ever contributed in that area, things like:
- First and last edit
- Total number of edits
- Number of edits still visible in today's data
I ran it on Los Angeles county and this is what came out. As a list, it is already pretty insightful, but you can roll nice visualizations with it as well. Enter the Brave Mappers Of Wherever You Are project that was sitting under the same layer of dust. I ran it on the LA county output with this result.
Brave Mappers shows a timeline of mappers for your area based on historical OSM data. A long bar is a mapper who has been active for a long time, like techlady here:
The users are sorted by how long they have been active in this area. Scrolling down, there are some interesting outliers to be seen. Here is DaveHansenTiger, the original TIGER import account:
As expected, active for a short time right at the beginning of the timeline. The green color of their bar means that they mostly created things (rather than modifying or deleting). The transparency indicates most of their work is no longer in the current data. In this case that is probably a good thing!
Finally, clicking on a user's bar gives some more details on their contributions:
If you want a Brave Mappers page (or just the spreadsheet) for your area, that can be arranged! If you feel up to it, you can clone the OSMQualityMetrics repo and run
UserStats.js yourself on a full history planet for your area, then process it with the script in the bravemappers repo. Or you can send me an OSM POLY file and I will try and make some time to do it for you!
Over at Telenav we have been busy comparing the GPS traces of drivers using our navigation apps against OSM data. This is fun work and sometimes leads to really useful results. One of our recent studies involved comparing what we think are one-way streets to the corresponding OSM ways. The process is quite simple: count how many traces following a given street go in one direction, and how many go in the other. If the number for one direction is close to zero, the street is probably one-way. The outcome depends on the confidence level you apply, but it turns out that there are more than 100 thousand OSM ways that probably should be marked with a oneway tag, but aren't.
One. Hundred. Thousand.
I don't know about you, but I think that would make for a really great MapRoulette challenge. So that's what I did, and here it is.
But how do you know if a street should be one-way without having eyes on the ground? It turns out the aerial imagery provides helpful hints. Let me show you what I look at to verify the one-way-ness of a street using just JOSM and the Bing layer.
Let's look at this case here:
Notice the arrowhead? That is the direction we found the vast majority of trips followed, so we think that is the one-way direction of this street. Let's look in JOSM:
In this case, there is really only one hint that this is in fact a one way street, but it's a pretty convincing one: The markings on the road. See the stop line? It goes all the way across, which it wouldn't if traffic were allowed to enter from the right. Also, it is a little blurry, but there is what appears to be a left arrow painted on the road as well. So all in all, I am convinced that this way should be marked oneway=yes.
Be careful though! OSM ways can be very long, and the one-way restriction may not apply to the entire way. So always be sure to check the entire length of the way before you tag and upload, and split the way as necessary! The Telenav detection works on smaller segments internally, so the suggested one-way-ness does not always apply to the entire OSM way.
Lets' look at another case.
Looks like a residential area. Let's load it up in JOSM:
In this case, it's really hard to see. There's no obvious stopping line that I could see, and there's too much shade to see if there are parked cars on the left side of the street that are parked facing west - which would be another good indication that it is a one-way street, because it is illegal in the United States to park in the opposite direction of the flow of traffic. So this one I would mark as 'Couldn't See'. Perhaps someone else sees something I don't? Here is the permalink to the task in MapRoulette. (Neat eh, permalinks - makes it easy to share a task you want to discuss or highlight. They remain valid even after a task is fixed or even deleted.)
Oh and yes, I am afraid this is another United States only challenge... We are working hard to get more international challenges in, and we already have some interest. I am sure I mentioned this before, but MapRoulette now has an API that lets 'third parties' submit their own challenges. On top of that, we're also making it really easy to deploy your own MapRoulette server. If you're interested in any of that, just let me know. I will be hosting a session on MapRoulette at State Of The Map US this weekend, and Serge and are also planning to run a BoF session for a deeper dive into MapRoulette challenge making. If you don't make it to SOTM US, don't feel left out - there will likely also be MapRoulette sessions at SOTM EU in June!
Those first few MapRoulette challenges did not keep you busy very long: most tasks in the Zorro / Tangled / Wrong One Ways challenges were fixed within the first few days. Dang! We were not prepared for that and had to take MapRoulette down for a little while because there was nothing to do...
But now we're back! This time with Ways Needing Smoothing. We detect ways that have suspiciously sharp angles and ask you to smoothen them out. Here's a great example:
Opening this task up in JOSM reveals that there's more to fix than just the one sharp angle that was detected:
This is typical for the US TIGER data - poorly aligned ways that are either under- or overnoded. And there's still a lot of them! We detected over 100,000 cases, not all of which are loaded for this challenge yet.
This challenge is bound to have some more tasks than usual that are not 'real' errors - that is the nature of challenges of this type. Just skip over them and mark them as 'not an error'. They will not show up again.
Oh and finally - a cool new little feature is the permalink: you can now link to any task you encounter and want to discuss or share. The link for this one is http://maproulette.org/#t=ways_needing_smoothing/527470359108565095470769424325999623764196595999
Happy mapping :)
We are about to launch the new version of MapRoulette! Finally! The most obvious changes are that you now sign in using your OSM account - so you can track your and other mappers' progress, and multiple challenges being available to choose from! We will launch the day after tomorrow with four new challenges to start with: the familiar Zorro Ways, ways with overlapping parts, wrong one way ways (if you know a better name, let me know..) and tangled ways. More to come!
Behind the scenes much more has changed. There is now an API for challenge providers to post their own challenges (let me know if you want access), challenges can have an area associated with them so you will be able to choose a challenge that is local to you, and they also have difficulty levels. The entire application has also been rebuilt from the ground up to be more stable, scalable, and allow more flexibility for different types of challenges in the future. If you want to see how MapRoulette is built, check out MapRoulette on Github.
So, on March 12, head on over to maproulette.org and start fixing!
I keep coming across beautiful OSM-based maps! Here is one for Poland:
Go explore for yourself here.