Reflections on 10 years of a changing Open Source Map

Posted by James Derrick on 28 December 2017 in English (English)

Reflections on 10 years of a changing Open Source Map

It was ten years ago today that I registered to map the highway!

With apologies to the Beatles, on 2007-12-28, I registered an account with OSM, started editing the map, and after returning from Falkirk to Cramlington, uploaded my first GPS survey data.

The map has changed rather a lot in that time, and not just in completeness, but in the diversity of the usage of OSM data. I was surprised a few years ago when visiting Lichfield that the Council was using OSM to show car parks, but shouldn't have been surprised at all when preparing for a walk up Simonside when the Forestry Commission linked to an OSM map based in a small way on my previous visit!

Back in 2007, OSM in the North East of England included the major long distance roads only. Pioneers with GPS must have driven blindly into the blank 'Here be Dragons' space probably rushing towards Edinburgh. Back then, my town of 45k souls was little more than the odd through road so my earliest traces are dendritic studies of suburban planning surveys. NW Cramlington dendritic housing Using the classic maze solving technique of 'follow the left hand side of the road', I cycled around local housing estates adding detail and getting progressively fitter. The improvement in my legs became important as survey cycle rides became progressively longer, ending up with an hour's travel to map a distant hamlet in Northumberland before heading back.

Although my heavy commuter bike didn't get any lighter, other tools of the field surveyor did improve. After my first Garmin GPS II+ GPSr fell victim to sea water ingress whilst teaching young dinghy sailors, I moved to a Nokia N800 - no, not the Lumia 800; the Nokia N800 Internet tablet. This was a full Linux device with a colour touch screen and both Bluetooth and Wi-Fi at a time when tablets were only available from pharmacies!

Using the rather amazing (for the time) Maemo Mapper software and a Bluetooth GPSr 'puck' it was possible to both capture a GPX file and more importantly, make field survey notes on the same device.

Interestingly, even back then, proprietary maps cost a lot of money - around £150 for the N800 so OSM was rather useful.

The 640x480 camera was so poor quality, even I could have captured scenes better with crayons, so text notes were the order of the day. GPX files with one point per second didn't fill up the 256Mb SD card, and street names could be added directly to waypoints - something I recorded for posterity in the OSM Wiki with a screenshot back in 2008:

Maemo Mapper Insert Mark

Pecking away at a tiny on-screen keyboard with a resistive stylus was S-L-O-W (remember, this was 2008!), so I experimented with Photo mapping, and Audio mapping using time synchronisation and the several JOSM plugins to work out where a note was captured (as the recording device didn't have a GPSr). Years of field survey suggests mixed methods are best - photos for specific facts and general street scenes, audio for moving commentary, waypoints for signs.

Top Tip: A picture can indeed save a thousand words, which can be a long time when listening back to an Audio mapping recording for the forth time because wind and traffic noise mean you can't quite catch the spelling of that side street name!

Top Tip: Never rely on a single survey method or device. Use more than one GPSr (Garmin models capture inertia differently; phones have poor antenna), take spare batteries, a camera phone, and a pen (and a spare pen :-) ).

Ah - the first mention of JOSM. Like many powerful tools, starting out with JOSM was VERY difficult. I remember finger bending gymnastics trying to trace GPX files and add roads. My 19" CRT monitor must have looked like a sunflower as the screen was surrounded by a halo of yellow sticky notes containing scrawled keyboard shortcuts and tags.

After only ten years practice, I can trace about one building every four seconds in a dense area and probably have deformed fingers from jumping between A to add, S to select, B to add a building, SHIFT to re-align (and no, I can't remember the keys - they just happen). Unfortunately, I can see future archaeologists interpreting the pattern of muscle attachments in my hands as a sign of RITUAL behavior rather than Open Cartography!

After completing the survey of all streets in a 30km radius from my home, Microsoft announced in 2010 that Bing Maps could be used for tracing into OSM. This changed mapping from requiring direct GPSr field survey for ALL activities, to an iterative process of 'armchair mapping' using aerial imagery to rough-out road and building geometry first before capturing ground-truth, road names, actual position data, and gazetteer details with a physical visit.

Comparing the data captured with early Bluetooth GPSr (only capable of tracking 8 satellites) with modern imagery and Garmin GPSr, the results are surprisingly good. Sure, you can show spacial offsets using Trimble or Leica differential survey kit, but the features are mostly within the width of the road. Almost - smaller roads have less GPS data so are less accurate; wider larger roads have more traffic, so more traces, so more averaged centrelines.

Bing still has a lot more aerial imagery than myself (even after a DC-3 survey): Cramlington from a DC-3

With my locality mapped, my pattern of mapping has changed from area survey to maintenance, adding new developments, checking Note questions, and occasionally wondering why my past self interpreted a trace in such an obviously wrong way (with the benefit of 20:20 high-res aerial hindsight!)

Re-visiting early areas based on ground survey before aerial imagery does highlight some common issues:

  • Moving GPSr tend to use inertial averaging giving exaggerated corners. The impact is junctions can have poor geometry, with a straight 90deg T-junction skewed into a more complex Y or L, often reversing the 'give way' priority.

  • Straight roads aren't. Road engineers have big toys which don't like corners, so roads like to be straight as long as the terrain allows it. Early GPSr and parked cars conspire to make roads more curved than reality.

Top Tip: The easiest way to map a straight road is not to put bends in it! Simply - take out unnecessary extra points, and line up the remainder (e.g. the JOSM L-key).

So, after ten years of GPSr field cycle survey, how should you commemorate the milestone?

Well, a year ago I decided to map EVERY DAY for my tenth year - and somewhat to my own surprise, have managed the feat. After almost 9000 edits and 620 traces, Pascal Neis excellent tool shows 366 mapping days in the last year:

My 2016-2017 Mapping History

As well as mapping from a bike, I have also captured images and tracks whilst on a narrowboat on the UK canal system. Using both Garmin GPSr, and a Raspberry Pi complete with camera (much cheaper than a GoPro), I decided to arm-chair map several journeys from Lichfield to Stafford, Stone, Stoke on Trent, etc.

I ended up adding both urban and rural detail from Lichfield, down the S&W past Penkridge, to Autherley Junction, up the Shroppie to Chester, then back to Middlewich, Kidsgrove, then up the Macc mapping Congleton, Macclesfield, Marple, and New Mills (to name but a few), before stopping at Bugsworth (AKA Buxworth - the canal is older than the road signs :-) ).

This has taken some creativity, from using iD on a Linux desktop, OsmAnd on a phone, and even JOSM running fine on a Raspberry Pi 3 with official touch screen and 4G dongle whilst on a train back from a business trip!

JOSM on a Raspberry Pi 3 on a Train

Top tip: Use a scroll-wheel USB mouse as a touch screen is just not precise enough when moving at 200km/h...

Remarcably, for an embedded ARM teaching board, JOSM is quite usable on the RPi3 even when mixing GPX survey data with aerial imagery - as long as you use TAB to hide the sidebar to make the most of the small screen.

Thanks for all the Nodes Ways and Relations!

So, here's to the next ten years of OSM contributions both ground-truth and armchair, and thanks to all those other mappers, surveyors, wiki-fiddlers, sys admins, devs, DBAs, OSM Foundation, organisers, and not least our ex-BDFL who have made it all possible!

Where next? Well, there's three on-going construction sites to check, and a new one reported via a note... better get the GPSr and bike out!


Location: Cramlington, Northumberland, North East England, England, United Kingdom

Comment from nebulon42 on 28 December 2017 at 18:01

One of the true pioneers! :) Thanks for this personal story, a worthwhile read.

Comment from GOwin on 29 December 2017 at 04:29

Another one of the awesome OSM folks! Kudos for staying on, and thank you for your contributions.

Good luck with your map everyday goal.

Comment from dcp on 31 December 2017 at 07:26

Just brilliant what you have achieved. I am a 9 year old OSMer, i.e. started in 2008 but otherwise a very old man, maybe one of the oldest. Having had my own similar OSM odyssey but with different tools and the same results. However your efforts are a quantum jump ahead of mine. You have my respect!

Comment from robbieonsea on 31 December 2017 at 13:42

A great write up for your OSM story. Definitely need a +1 / :ike / thumbs up for this diary entry. Here's to the next 10 years.

Comment from andy mackey on 2 January 2018 at 23:52

I enjoyed that. Thanks and Happy Mapping.

Comment from ChristianA on 3 January 2018 at 19:43

Nice read and very nice work! Cheers!

Comment from cbeddow on 12 January 2018 at 12:30

Awesome read from a true veteran! Are you using Mapillary these days for your photos, or just taking still photos manually?

Comment from James Derrick on 13 January 2018 at 21:15

Crikey - thanks for your kind words one and all!

As for survey images, it depends...

Mapilliary is useful for general survey images using a forward-facing phone camera when cycling or driving, but I still use a separate Garmin Oregon 750 camera for specific details.

For example, an older phone running Mapilliary continuously captured lots of useful images on a survey earlier today with minimal effort, but with a fixed forward view, it could not see a street name sign high up on a building wall 90deg to the road.

When photographing random details, a camera phone produces great images, but the GPS fix can be very poor when first extracted from a pocket. On the other hand, a dedicated GPSr gives a much better geo-location and runs for hours on AA cells, but the camera can be poor in low light.

So, everything is a trade off, but I tend to use Garmin GPSr most of the time as they have secure handlebar mounts, and complain less about heavy rain than I do! :-)

Login to leave a comment