Last time I tried to do an import, things ended up a bit ugly. I prepared all the data transformations and processes, but I did not receive the permission to use the dataset from Boston GIS, and I was not able to contact anybody from the GIS department. So I gave up. And learned from my mistakes.

After spending some time untangling the lanes on nearby highways in Greater Detroit Area and adding 5 square miles of residential buildings just because why not, I decided to see whether there is any open dataset available to tag these buildings with addresses.

After being told by my city officials that neither they nor the county we are in provide any data, I kept my hopes low.

SEMCOG

Two days ago I discovered SEMCOG, the Southeast Michigan Council of Governments, and today I wrote them for permission to import their data into OpenStreetMap.

Their coverage is huge:

The data provides the addresses, building footprints, even median height of the features, so it will make Detroit area much easier to navigate using only OpenStreetMap.

At some point SEMCOG have hosted a #Maptime event, but I was not able to find any reference of this organization in our Wiki, and it feels like we should know about it.

Now, this post is just me sharing the excitement, so I am going to proceed with posting to the import mailing list, planning, transformations, and scripts only if permission is received from SEMCOG.

I am using various OSM-based applications as a daily driver for navigation purposes (this month I am fascinated by Magic Earth. They have recently added relief data and traffic-based routing, so go check them out).

I’ve been testing routing primarily with demo OSRM and Valhalla servers, but these services take time to be updated, and I want to make sure my changes don’t break things now.

While browsing the Routing I found CartoType Maps, which is a closed-source desktop application that renders and perform routing locally.

A separate makemap application is required to convert from OSM XML to CartoType internal representation.

I found that sometimes the application gives false positives, claiming roads are not connected only because it does not seem to handle turn restrictions properly. Far from ideal, but still better than waiting for the remote servers to catch up.

What do you use to test routing?

As I try to continue writing about various OpenStreetMap-based navigation applications, I’m really struggling to write something about Scout.

It is a weird application.

It is a combination of a navigation engine (offline download costs money), but it only works in the US, ignores service roads, and does not show buildings. It has a chat built in, but I have 5 chat clients already. It supports sending your location to other people, but these other people must have Scout installed as well, and you can’t just say “here, watch me go somewhere”, you MUST provide your destination (unlike Glympse). You don’t see POIs unless you search for them, and the POI database may not have correct location for the addresses. Since there are no buildings (and no satellite imagery), you can’t see you are directed to a shed instead of Walmart Superstore.

Has anybody here used Scout and thinks it is the best thing since sliced bread?

So, I’ve decided to look at GNOME Maps yesterday and found that this happened:

http://open.mapquest.com/ shows the following message:

Beginning July 2016, this version of MapQuest, along with Open.MapQuest.com, will no longer be available. We apologize for any inconvenience. If you’re using a map generated from this site on your website, please visit our Developer blog for information on how to transition your map to our updated tiles.

So now all the tiles displayed by MapQuest are hosted on MapBox servers, and every request requires a key to be provided.

actually we don’t do the maps anymore, we use Mapbox. MapQuest is focusing on some mobile and IoT applications, routing and direction engines rather than maps

– http://stackoverflow.com/a/38356148/340480

The moment I launched Navmii GPS I was presented with 3 words that looked like this:

I started looking for the explanation and found that’s from the service called what3words, that attempts to solve the user-unfriendliness of GPS coordinates by splitting the whole world into 3m x 3m squares and assigning a unique set of 3 words to every location:

Now, I got all excited and thought this is going to be the best thing ever, but my wife was skeptical and made a great point:

what3words address for a Robin Williams bench has an address of “script.export.noble”. Move 3 meters east and you’ll get “votes.began.hours”. So it is allows you to precisely locate a feature. But here’s the problem - you can’t infer anything from these words. Unless you have an access to a device that has a mapping available, you can’t know what continent it is on. This may not be the best use case, but when you have an address of “5 Child Street, Boston, Massachusetts, USA”, you can assume “6 Child Street, Boston, Massachusetts, USA” will be somewhere nearby.

Okay, so I was less hyped and started looking into the way the words are generated, and it turned out the algorithm and the data is not in a public domain, and you need to use their REST API or offline SDK to perform the mapping.

Our goal is for what3words to become a global standard for communicating location. At the moment, the core what3words algorithms and data are not in the public domain. In the future, we may release some or all of our source code – we will continually evaluate the business case for doing this.

— http://developer.what3words.com/pricing/

And then I’ve read this in their API License terms:

You must not pre-fetch, cache, index, copy, re-utilise, extract or store any what3words Data.

Well… you can contact them to get the SDK containing the data, but… I am not sure I like the idea of a company pushing a global addressing standard that you can’t freely use. I’ve recently read w3w is selected by Mongolian Post office to provide addresses for locations that don’t really have an official address. It will be interesting to see how this works in real life.

Another thing I encountered during my search for navigation was Mapcode (used, for example in Here Android app).

Originally created by TomTom, this system does not really rely on any centralized data. The algorithm, however, is patented:

In order to prevent misuse, unauthorised alterations, copying or commercial exploitation, please note that the ideas and algorithms behind the mapcode system have been patented and that the term “mapcode” is a registered trademark of the Stichting Mapcode Foundation. Our mapcode source code is released Apache License Version 2.0.

A good thing about Mapcode is that it shortens the coordinates to manageable tokens, and you can select the representation that better suits your needs. And the best part is that you don’t need any server interaction for it all to work.

[rye@delorean ~]$ python3
Python 3.4.3 (default, Jun 13 2016, 16:33:53)
Type "help", "copyright", "credits" or "license" for more information.
>>> import mapcode
>>> mapcode.encode(42.52838, -83.36005)
[('7S.65B', 'US-MI'), ('S3K.6YD', 'US-MI'), ('CDS1.XWV', 'US-MI'), ... ]
>>> mapcode.decode('US-MI 7S.65B')
(42.528403, -83.360037)

Close enough.

Okay, guess what part of the world “US-MI 7S.65B” is in? How about bugs.bumps.glue? And I am pretty sure it is easy to figure out that “US-MI 7S.659” is nearby.

On the other hand, embedding the country/state into the code can cause the same issues as the country borders in disputed territories. The location above also has an international address of “T8BB9.T204”, which has the same problems as w3w one, except that there are neither bugs nor glue.

So, what do you think? Does a world need a GPS coordinate shortener and if so, what would you use?

As promised before, here’s my review of a navigation application based on OpenStreetMap data, OsmAnd:

OsmAnd: Navigating With OpenStreetMap

I have to say that when I started using OsmAnd+ for surveying location, I saw occasional rendering hickups. It was not as pleasant as the other mapping applications, but it was still doing it’s job.

Things changed dramatically when I tried using it on the roads with 45 mph (76 km/h) speed limit with complex intersections where zoom in/zoom out or turns caused the whole thing to redraw so furiously it was hard to keep track of what was where.

I posted a number of videos to show what I mean and I regret to say that I ended up with a result completely different to what I’ve hoped.

Yes, it is Open Source, and it has a ton of features (and almost awesome OSM Live!), but I just cannot use it for car navigation, sorry.

A set of turns bypassing a roundabout I reconfigured recently was disliked by OSRM, instead sending the cars via the actual roundabout:

Daniel Patterson @ OSRM #2421 directed me towards a debug OSRM map showing that the roundabout links without ‘‘maxspeed’’ tag were categorized with default speeds of ~25 km/h (~16 mph) while the roundabout tagged with 20 mph (32 km/h) provided a faster way.

So, take a look at http://map.project-osrm.org/debug/ if OSRM seems to ignore a better route.

I finally wrote about OpenStreetMap on my personal blog, so here’s the link:

In the Beginning There Was a Node

---

On a related note, I am using OpenStreetMap data for all my navigation needs.

I got quite impressed with MapFactor Navigator Android application which seems to have the most up-to-date maps so far and does not have the redraw issues of OsmAnd. Yes, I know, OsmAnd is opensource, but I am not that versed in Android development. Eraser Map Beta seems to be progressing nicely, even though it likes to repeat the turn-by-turn instruction very quickly and several times. I have a list of the applications I tested in the blog post above, and I am preparing detailed reviews for each one.

Now, here’s the thing that bothers me - OpenStreetMap is updated every second. Any navigation application that lags behind for more than a month or two raises a question of business continuity and infrastructure readiness. If an application vendor claims they release the updated maps only when they release a new version of an application, does that mean they don’t have an automated import process? Do they have tight coupling of map data and business logic? If a vendor does not tell about the update schedule, is that application even worth considering to be used as a daily driver?

On an unrelated note, Boston, MA address import is stuck awaiting for City of Boston response of any sort, and since I am no longer in Boston, I am updating a different town, Farmington Hills, MI.

Since City of Boston data has a CC BY 4.0 license, I am now waiting for them to confirm that a link on Contributors page is sufficient.

In the mean time I decided to play with CartoDB and here’s what I have now:

Magic!

The import hit a roadblock. At the moment I am waiting for a license terms clarification from Boston GIS, since the current one was not considered a license by imports/talk-us mailing list members.

The City of Boston recognizes the value and benefit gained by sharing GIS data. Although the City has made reasonable efforts to provide accurate data, the City makes no representations or guarantees about the accuracy, completeness, or currency of the information provided. The City of Boston provides this data as is and with all faults, and makes no warranty of any kind. Each user is responsible for determining the suitability of the data for their intended use or purpose. Neither the City nor its affiliates, employees, or agents shall be liable for any loss or injury caused in whole or in part by use of any data obtained from this website. The GIS data is updated and modified on a regular basis and users are encouraged to report any errors to the City.

Now, while I am waiting to hear back from Boston GIS, I decided to proceed to the second phase - splitting the buildings that house more than one, well, house number.

And there are a lot of these in Boston.

Original building import was done a while ago, and most of the buildings were left in the original shape. Since then MassGIS came up with a number of updates shapefiles that, if imported originally, would have made the whole process much easier. Since replacing the geometry for every building in Boston was out of the scope of the import, I decided to proceed with manual terracing of the buildings, verifying weird cases via Bing satellite maps, learning how to use the tools and mastering the skills of spotting building differences.

Now, you may ask, why don’t I just use the shapefiles provided by MassGIS? Well, as I already realized, imports are painful for OpenStreetMap, and replacing the whole city would create more pushback than I’d like to handle. I tried to copy a small number of buildings from MassGIS Data - Building Structures (2-D, from Ortho Imagery), but a lot of them grew arcs on the corners/bay windows missing from the real world:

And in the previous version every building had around 8 extra nodes along the perimeter that had no real use. At some point I’ll probably get bored and will probably do an automatic removal of collinear nodes (there must be a tool for that already somewhere).

This is a tedious job, but the results are awesome - I’ve got through all issues in Back Bay, Bay Village, currently progressing through South Boston part, driving the number of unique buildings up. Even if SAM import does not happen now, split buildings will allow for easier number assignment later, and granular automatic geometry updates as well.

It would be so awesome if one could see the result of OSM changes before they go live…

Jamaica Plain

Numbers!

Dorchester

More numbers!

Downtown

Buildings here are not split properly, so numbers appear not as often as in other neighborhoods.

I started producing .pbf and .obf (OsmAnd map) files linked to from Boston Street Address Management (SAM) Import, so you can see what’s going to happen to Boston, MA in OSM once the import is approved.

Last time I wrote here I was adding the addr:housenumber tags to some houses automatically using the data from Property Assessment dataset. Since the license was not clear for that data (it is public and provided via Open Government initiative), I decided to pause whatever I was doing and search for an alternative source of data with an explicit license.

City of Boston provides another dataset called Live Street Address Management (SAM) Addresses and it fits our task perfectly.

At this point I decided that this is going to be a full-scale import instead of street-by-street handpicking. I joined the relevant mailing lists and sent the RFC for wiki:Boston Street Address Management (SAM) Import.

Even without any manual work, we are able to assign the unique addr:housenumber to almost 90% of the existing buildings across all the neighborhoods, which is 100% better than no house numbers at all.

The python source driving the transformation is linked from the wiki page as well as .osc files that are periodically generated as I am refining the algorithm. The files must not be uploaded until the import is approved (and .osn files must not be uploaded at all), but you can load them locally and see what the future holds for Boston.

I’ve been occasionally looking at OpenStreetMap project for some “free/opensource” directions, location information, etc. But usually the results for the places I’ve searched were quite poor. It was making me sad and reverting to a mainstream mapping services.

And then I’ve got some free time, and decided that I should not spend the days home, and explore the world that is immediately around me instead. Pairing this with the low quality of Jamaica Plain map (where I currently am) in general, I decided to try fill in as many blank spots as possible.

Source Data

The first thing I found was the ArcGIS-based service from City Of Boston - Boston Tax Parcel Viewer. After spending an hour trying to learn more about map tile services in general I found that there is a MapServer with WMTS support. Since some of the buildings I knew were shaped quite accurately there, I decided to start using is as a common base layer.

Most of the building shapes in Boston in OSM comes from Orthoimagery paired with LIDAR data from some older days as well as people mapping the buildings from Bing imagery. There is newer data available from 2011-2015 Ortho Imagery. I may later switch to these maps even though the rooftop analysis is error-prone, but it is still much better than retracing the buildings from Bing.

The roads for Boston were imported from MassDOT using the data from 2007 and don’t seem to have been update since. Some of the streets had curves that weren’t there in the Basemap or the real world, so I got new ones dated 2013.

The first thing I noticed a long time ago is that a lot of maps in US don’t show the building numbers. Which is not much helpful when you try to figure out the location of the building you’re looking for while walking around. At some point for complex side streets it turns into bruteforcing, and having seen it is actually possible to obtain the address data from Tax Parcel service, I found that Property Assessment 2015 contains just the information that I need. I converted that data into an sqlite DB and wrote a script that puts the numbers on the buildings. It is NOT supposed to be used for fully automatic numbering, since the data in the Property Assessment table contains errors, but it is a good way to obtain reference points during the actual survey.

Survey

Paired with my Nexus 5 and OsmAnd for GPX tracks and custom maps I went to look at the conflicting buildings, gather local business information, add recreational areas and remove POIs that are no longer valid. This turned out to be harder than I imagined, since you need to go to side streets to find entrances, or look for a building number under a rock, but it is also healthier than sitting in front of a computer according to my fitness tracker. That is still needed after the data is gathered though.

Processing

I am using JOSM to update the data, and the numbering script to assign the addr:housenumbers to the buildings for the streets I’m about to survey. Most of the local businesses can be found on Yelp, but I still try to get information from their website/front door pictures. It is a great feeling to refresh the openstreetmap.org website and find the area matching the real world even though all the other services still show outdated content.

After processing the data I would then upload the modified maps to my phone (OsmAnd provides conversion tool), which makes it easier to refine the data.

Tips

I am pretty new to all this, but I already found that for a successful survey you need these things:

• GPX tracks - get a portable battery charger and make those tracks. They are not that useful in densely populated areas due to signal reflections, but you will be able to retrace your path later when you need it.
• Take pictures of corners, storefronts, and most importantly, signs. Make sure you have GPS coordinates written to the photos. I can’t count the number of times I had dozens of pictures with some important things only to find that I forgot the context and they are unusable.
• If there is something on the ground that is supposedly clearly visible for the satellite/aerial photography, add a waypoint. Trees are surprisingly well at hiding everything below them. Don’t trust the winter trees.
• Don’t hesitate to ask people around if some feature is confusing. You may get a great story out of that, or you can spark an interest in exploring the surroundings.

I used to think that the way does not matter, only the destination. This week of mapping with OSM made me to appreciate the way as well.