Turn Left at People's United BankPosted by blackboxlogic on 16 June 2020 in English (English). Last updated on 24 June 2020.
I just finished the Maine address import and thought I might share a narrative.
In early 2019, I was driving home, getting directions from google maps. I had grown apathetic to fast food restaurants highlighted on the map, but on this trip, the google lady vomited forth: “Turn left at People’s United Bank”. She might have said “turn left on Main Street” or just “turn left” but the application I trusted for navigation went out of its way to shill a business. I was left wondering “what else does it do”. Could advertisers pay to /optimize/ my route, sending me past their venue? That sounds /crazy/, and plausible. So it was time to make a change.
The alternative I found was OsmAnd. But upon trying it, I found Maine was missing nearly all addresses. I was baffled how that could be, and dismayed that I might crawl back to google maps. This wasn’t just impeding me, this would block anyone in Maine from switching, and /I am stubborn/. I realize everyone can contribute to OSM and Maine had a handful of contributors, but at the current rate, Maine may not be navigable within my lifetime. It needed a push.
I had a vague idea of “get data from here” and “put data there”. The APIs had documentation and I had a compiler, how hard could this be? I started developing tools in July of 2019. In the beginning, I had two concerns:
- I could lose interest and leave the work “half done”, and regret that I ever started
- I could do a terrible job… and regret that I ever started
Looking back, this project held my attention the whole way and I’m satisfied with the results.
Side node: Did I look for existing tools? Yes, but I’m a software engineer and when you have a hammer everything looks like a nail.
Using my language of choice, I wrote an Osm API client, a client for Maine’s address API, a E911-to-OSM schema translator, a conflation engine, and a command line interface to tie it together.
Some effects of writing my own tools:
- Good: I am now intimately acquainted with the many challenges of each component
- Good: I had complete control and visibility into every part of the process
- Bad: I never polished the user interface, so it was only realistically usable by me. I couldn’t divide the work between other mappers
- Good: I had hoped the tools would be useful to the community. Some parts are already being used in other projects, some parts are effectively dead
- Good: I was able to contribute fixes to other, related projects
- Good: I have a substantial portfolio of work to show potential employers
With passable tools, I started processing the import in January of 2020. My tools evolved constantly throughout the import handling ever-weirder corner cases. Her are a few of the poor assumptions I made along the way:
- The source data will be accurate and clean
- The state won’t change their API and schema in the middle of my project
- Geometry is easy, simulating the human decision making process for picking things to combine is easy
- House numbers always relate to roads
- There wouldn’t be odd bunches of addresses that need to be handled specially by context (trailer parks and apartment buildings)
- There would never be ambiguity about which municipality something is in, or what a municipality is called (villages, unorganized territories, reservations, plantations)
- There won’t be many data conflicts to manually resolve (Portland took ~20 hours, Sanford took ~30 hours)
- I can use existing OSM road data to validate addresses
- I can use existing OSM municipal boundaries to validate addresses
- Names will be in canonical form (Rose Road, Roses Road, Rose’s Road, Roses’ Road, ROSES ROAD, Roses Rd, Rosesroad, Roses Street, etc)
- And many more…
I was working at a large international company, doing work that I genuinely believed in (we literally saved lives, a few a day) but the work had turned gray. I didn’t notice I was burned out until I found this project, and absolutely loved working on it. It reminded me that programming /is/ fun, and I started to realize work had /stopped/ being fun. There were many reasons, but primarily, I wasn’t learning anything new. I spoke to my manager and asked for a whole lot of unpaid time off, which was declined. So I quit (and got my unpaid time off). Aside from other interests, I spent the next year working this project. It has given me a great sense of accomplishment and contribution. I met many characters long the way and learned a lot about my state. And finally, I can use OsmAnd for navigation.
Did I mention I’m available for hire? I offer this project as my resume while I seek GIS opportunities.
I’m planning Maine roads, or getting a job, whichever comes first.
Comment from CjMalone on 17 June 2020 at 23:16
Awesome work. Sometimes when I trace a building I think the few occupants might look at OSM and because there house is mapped become contributors. You can say that for an entire state! Thank you.
Comment from GHiggins on 21 June 2020 at 23:26
Wow, looks like you really put your head down and went for it. I might know of a job that is available. Ill send you an email.
Comment from stevea on 5 July 2020 at 16:59
More proof that a single dedicated person (though, nothing wrong with sharing the load with others, too) can complete an entire state’s worth of data like this. (Well, a “sub-aspect” of a whole state, addresses). Yes, it’s a great deal of work, real work, hard work, fraught with what appear to be stumbles along the way, but really these are simply the process of learning as you give your efforts to this great project called OSM.
During my decade+ in OSM, I’ve thrown my shoulder into bicycle routing (at the national, regional and local levels), rail networks (at the gargantuan level of the rail-rich state of California) and many other “bigger than bite-size” efforts, and I find this to be especially well-executed and seriously impressive. What a fantastic addition to our map. I offer a deep bow of obeisance to you, sir.