Reflections on 10 years of a changing Open Source Map

It was ten years ago today that I registered to map the highway!

With apologies to the Beatles, on 2007-12-28, I registered an account with OSM, started editing the map, and after returning from Falkirk to Cramlington, uploaded my first GPS survey data.

The map has changed rather a lot in that time, and not just in completeness, but in the diversity of the usage of OSM data. I was surprised a few years ago when visiting Lichfield that the Council was using OSM to show car parks, but shouldn’t have been surprised at all when preparing for a walk up Simonside when the Forestry Commission linked to an OSM map based in a small way on my previous visit!

Back in 2007, OSM in the North East of England included the major long distance roads only. Pioneers with GPS must have driven blindly into the blank ‘Here be Dragons’ space probably rushing towards Edinburgh. Back then, my town of 45k souls was little more than the odd through road so my earliest traces are dendritic studies of suburban planning surveys. Using the classic maze solving technique of ‘follow the left hand side of the road’, I cycled around local housing estates adding detail and getting progressively fitter. The improvement in my legs became important as survey cycle rides became progressively longer, ending up with an hour’s travel to map a distant hamlet in Northumberland before heading back.

Although my heavy commuter bike didn’t get any lighter, other tools of the field surveyor did improve. After my first Garmin GPS II+ GPSr fell victim to sea water ingress whilst teaching young dinghy sailors, I moved to a Nokia N800 - no, not the Lumia 800; the Nokia N800 Internet tablet. This was a full Linux device with a colour touch screen and both Bluetooth and Wi-Fi at a time when tablets were only available from pharmacies!

Using the rather amazing (for the time) Maemo Mapper software and a Bluetooth GPSr ‘puck’ it was possible to both capture a GPX file and more importantly, make field survey notes on the same device.

Interestingly, even back then, proprietary maps cost a lot of money - around £150 for the N800 so OSM was rather useful.

The 640x480 camera was so poor quality, even I could have captured scenes better with crayons, so text notes were the order of the day. GPX files with one point per second didn’t fill up the 256Mb SD card, and street names could be added directly to waypoints - something I recorded for posterity in the OSM Wiki with a screenshot back in 2008:

Pecking away at a tiny on-screen keyboard with a resistive stylus was S-L-O-W (remember, this was 2008!), so I experimented with Photo mapping, and Audio mapping using time synchronisation and the several JOSM plugins to work out where a note was captured (as the recording device didn’t have a GPSr). Years of field survey suggests mixed methods are best - photos for specific facts and general street scenes, audio for moving commentary, waypoints for signs.

Top Tip: A picture can indeed save a thousand words, which can be a long time when listening back to an Audio mapping recording for the forth time because wind and traffic noise mean you can’t quite catch the spelling of that side street name!

Top Tip: Never rely on a single survey method or device. Use more than one GPSr (Garmin models capture inertia differently; phones have poor antenna), take spare batteries, a camera phone, and a pen (and a spare pen :-) ).

Ah - the first mention of JOSM. Like many powerful tools, starting out with JOSM was VERY difficult. I remember finger bending gymnastics trying to trace GPX files and add roads. My 19” CRT monitor must have looked like a sunflower as the screen was surrounded by a halo of yellow sticky notes containing scrawled keyboard shortcuts and tags.

After only ten years practice, I can trace about one building every four seconds in a dense area and probably have deformed fingers from jumping between A to add, S to select, B to add a building, SHIFT to re-align (and no, I can’t remember the keys - they just happen). Unfortunately, I can see future archaeologists interpreting the pattern of muscle attachments in my hands as a sign of RITUAL behavior rather than Open Cartography!

After completing the survey of all streets in a 30km radius from my home, Microsoft announced in 2010 that Bing Maps could be used for tracing into OSM. This changed mapping from requiring direct GPSr field survey for ALL activities, to an iterative process of ‘armchair mapping’ using aerial imagery to rough-out road and building geometry first before capturing ground-truth, road names, actual position data, and gazetteer details with a physical visit.

Comparing the data captured with early Bluetooth GPSr (only capable of tracking 8 satellites) with modern imagery and Garmin GPSr, the results are surprisingly good. Sure, you can show spacial offsets using Trimble or Leica differential survey kit, but the features are mostly within the width of the road. Almost - smaller roads have less GPS data so are less accurate; wider larger roads have more traffic, so more traces, so more averaged centrelines.

Bing still has a lot more aerial imagery than myself (even after a DC-3 survey):

With my locality mapped, my pattern of mapping has changed from area survey to maintenance, adding new developments, checking Note questions, and occasionally wondering why my past self interpreted a trace in such an obviously wrong way (with the benefit of 20:20 high-res aerial hindsight!)

Re-visiting early areas based on ground survey before aerial imagery does highlight some common issues:

• Moving GPSr tend to use inertial averaging giving exaggerated corners. The impact is junctions can have poor geometry, with a straight 90deg T-junction skewed into a more complex Y or L, often reversing the ‘give way’ priority.

• Straight roads aren’t. Road engineers have big toys which don’t like corners, so roads like to be straight as long as the terrain allows it. Early GPSr and parked cars conspire to make roads more curved than reality.

Top Tip: The easiest way to map a straight road is not to put bends in it! Simply - take out unnecessary extra points, and line up the remainder (e.g. the JOSM L-key).

So, after ten years of GPSr field cycle survey, how should you commemorate the milestone?

Well, a year ago I decided to map EVERY DAY for my tenth year - and somewhat to my own surprise, have managed the feat. After almost 9000 edits and 620 traces, Pascal Neis excellent tool shows 366 mapping days in the last year:

My 2016-2017 Mapping History

As well as mapping from a bike, I have also captured images and tracks whilst on a narrowboat on the UK canal system. Using both Garmin GPSr, and a Raspberry Pi complete with camera (much cheaper than a GoPro), I decided to arm-chair map several journeys from Lichfield to Stafford, Stone, Stoke on Trent, etc.

I ended up adding both urban and rural detail from Lichfield, down the S&W past Penkridge, to Autherley Junction, up the Shroppie to Chester, then back to Middlewich, Kidsgrove, then up the Macc mapping Congleton, Macclesfield, Marple, and New Mills (to name but a few), before stopping at Bugsworth (AKA Buxworth - the canal is older than the road signs :-) ).

This has taken some creativity, from using iD on a Linux desktop, OsmAnd on a phone, and even JOSM running fine on a Raspberry Pi 3 with official touch screen and 4G dongle whilst on a train back from a business trip!

JOSM on a Raspberry Pi 3 on a Train

Top tip: Use a scroll-wheel USB mouse as a touch screen is just not precise enough when moving at 200km/h…

Remarcably, for an embedded ARM teaching board, JOSM is quite usable on the RPi3 even when mixing GPX survey data with aerial imagery - as long as you use TAB to hide the sidebar to make the most of the small screen.

Thanks for all the Nodes Ways and Relations!

So, here’s to the next ten years of OSM contributions both ground-truth and armchair, and thanks to all those other mappers, surveyors, wiki-fiddlers, sys admins, devs, DBAs, OSM Foundation, organisers, and not least our ex-BDFL who have made it all possible!

Where next? Well, there’s three on-going construction sites to check, and a new one reported via a note… better get the GPSr and bike out!

James

Another round of Ordnance Survey data has been released and processed by ITO! to produce a very useful comparison against OSM. Once again, their very usable web slippy map, and JOSM layer this has shown up a new building site I didn’t know existed ready for survey.

ITO World Open Data Tools

As well as missing developments, the comparison also showed up a fat-fingered typo from one of my past surveys - quickly fixed!

As usual, don’t assume OSM mappers are any better or worse than the professionals - OS data feeds from local authorities don’t always match the ‘ground truth’ signage and so don’t accept the OS Locator data has to be correct without a check particularly if OSM is marked source=survey.

If there’s no signage out there to contradict or correct, then an attributed name from OS is better than none (source:name=OS Locator). If Locator is wrong, use the tag ‘not:name=’ to show the issue to comparison sites like ITO, and also to OS so they can check themselves.

So once again, raise a glass to OS Open data, ITO for the service, and the local groups of fellow mappers slowly increasing the comparison UK Percentage Complete beyond the current 97.60%.

Help turn all UK counties UK cyan on the overview map - a 100% match!

Hi,

About every six months, ITO process road name data published from the UK Ordnance Survey and run a comparison against OSM. This is an interesting time as the differences show up new building sites you didn’t know existed and are prime for survey.

The ITO service is not the only tool that does this, but I like the colour coding and overall view of both the UK as a whole, and individual differences.

ITO World Open Data Tools

The comparison also shows up fat-fingered mistakes, but don’t assume OSM mappers are any better or worse than the professionals - OS data feeds from local authorities don’t always match the ‘ground truth’ signage actually on the ground!

In particular, this means don’t blindly accept the OS Locator data is correct and add a ‘source:name=OS Locator’ tag, particularly if OSM is marked source=survey. Take the time for a ground survey, and if there’s no signage out there to contradict or correct, then an attributed name from OS is better than none. If Locator is wrong, use the tag ‘not:name=' to show the issue to comparison sites like ITO, and also to OS so they can check themselves.

The indeterminate cases are harder - North Tyneside has a street with four signs on walls next to each other saying both ‘EDWINS AVENUE’ and ‘EDWIN’S AVENUE’. As the apostrophe signage is newer, I’ve added it to the highway, with alt_name for the version without as the ‘ground truth’ really is both versions!

Looking at my local areas, Wansbeck had a couple of additions which are already surveyed so will update to a 100% match in a few days. Blyth Valley survey locations will be an easy cycle tomorrow to Seaton Delaval, leaving Castle Morpeth and North Tyneside to catch up over the next few weeks.

It is interesting watching the statistics tick closer and closer to 100% as local mappers fix issues, only for the stats to go back down in 6 months. Still, the whole of the UK is only just under 98% in agreement with OS (including where we agree to disagree) so not only is OSM comprehensive, it is keeping up with changes over time.

So, raise a glass to ITO for the service, and the local groups of fellow mappers! You will see the number of differences reduce sharply over the next few weeks as GPSr are turned on and notebooks written in.

Help turn all UK counties UK cyan on the overview map - a 100% match!

Over the past few years, I’ve been a regular user of ITO World’s very useful map analysis and comparison tools. One tool (not unique, but well done) compares OS Street View data with OSM highway names and produces both a completeness report, and also a set of map tiles which show differences.

This diff layer is very useful in JOSM to spot errors in both map databases, be it simple typos or show areas for physical survey. In my area North of Newcastle, this has typically shown up schools being closed and turned into small development plots for housing which I’ve worked into my cycle training runs to make a physical survey.

As I know my local patch well, these areas are often already tracked as landuse=brownfield, of highway=construction so it’s just a case of adding in the street names once the developer has bothered to actually put the name plates up!

One such diff intrigued me - close to where my old University digs used to be (Ethel Williams Halls were demolished years ago!), a very small stub off a residential close started changing name.

Penfold Close became Whitby Crescent which ITO dutifully reported on. I though this to be a plausible omission, so added the name change and gave the credit to OS using the source:name=OS_OpenData_Locator tag. As this is about 12 miles away, and the road section in question about 20m long, I didn’t undertake a physical survey - slapped wrists all round!

Recently, it changed again to Whitbay Crescent, which seemed strange - had a mapper in OS Towers lost a key from their keyboard? Well, as this seemed so strange and my Winter fitness improving, I cycled out from Cramlington to find out.

Off a small residential road, a t-shaped stub is surrounded by 5 blocks of semi-detached homes. Only one has a name plate showing Penfold Close, but the door numbers show odd and even consistently showing they are in the same street.

So, what is going on with OS Street View changing names of a cul-de-sac all of 20m long? Well, I suspect it could be a Copyright Easter Egg!

The suspicious mind in me wonders if OS has been making small but insignificant changes in OSSV open data to track if and how fast they appear in OSM and other databases? I don’t believe this raises any copyright issues (I added the source tag to credit their information, as requested by the licence interpretation), but the feeling of possibly being tracked is both creepy, and reassuring.

Creepy - no one likes being instrumented and put in an experiment.

Reassuring - If my paranoia is correct, could it be that OS take OSM seriously to the point that our Open Street Maps are being watched keenly and closely by intelligences greater than our own*?

With apologies to HG Wells!

After going out and about on two wheels around Blyth, I noticed that the Coast and Castles long distance cycle route, NCN1 has moved. It no longer threads North West through the South Beach estate and across the River Blyth with the A189, and follows the coast.

New shared use paths go past Blyth Harbour and through Ridley Park, then West along a new Hairpin Greenway (dedicated to the late Laura Ferguson). Sustrans have invested a lot in clear signage, down to having a ‘Turn Back’ notice if you follow the old route past Blyth Asda!

As you’d expect, I’ve surveyed the new routes and mapped the changes in OpenStreetMap with just the big relation change to move the NCN1 route over to the new ways pending. CycleStreets still beats Sustrans own slippy map, IMHO.

The new route is great for long distance cycle tourism, and has improved foot paths into shared use cycleways. This route is more focused on recreation and with the excellent signage, will be much easier to follow than threading through housing estates.

For my regular commuting to Newbiggin by the Sea, I still wish that the ‘Cramlington Connection’ project and the Cycleway to Nowhere along the A189 hadn’t ended in the wilderness.

For several years, I have been cycling past a series of open cast coal mines to the West of my home town of Cramlington. The Banks Shotton Surface Mine has created a few jobs, and now is getting national intrest as the spoil heap has been imaginatively been formed into a visitor attraction in the shape of a female figure: http://www.banksgroup.co.uk/northumberlandia-site/

Northumberlandia, (or the “Turf Tart” to her friends :-) ), is close to opening to the public next week after the Princess Royal cuts the ribbon in Monday. http://www.journallive.co.uk/north-east-news/todays-news/2012/08/30/princess-royal-to-open-northumberlandia-sculpture-on-monday-61634-31727458/

The extent of the 400m landform and the nearby live mine site has been mapped from Bing imagery for some time, however the detail of the many paths, view points, and hard landscaping forming the face itself needs boots and GPS on the ground to capture in its full glory.

Anyone fancy a mapping party in Cramlington? :-)

TTFN,

James PS A few features have been added by eye as a taster of the curves to come.

The donation of good quality aerial imagery by [[Bing]] can make a real difference to the detail it is possible to map in OSM.

After mapping roads and main points of interest using various physical GPS methods, I thought the map of [[Cramlington]] Northburn looked pretty good. All the roads were well geo-located, and some building outlines were mapped where their location could be inferred from GPS tracks.

Image:http://wiki.openstreetmap.org/wiki/File:Cramlington_northburn.gif

As you can see, good imagery opens up the possibility of including mapping right down to individual houses. Details mapped via GPS stay put, however previously inaccessible features such as complex building outlines make a dramatic appearance! After a bit of work gathering house numbers, searching for a house number and street in the nominatim gazeteer search now goes straight to the individual building.

With enough effort, it is quite possible to achieve a level of detail better than Ordnance Survey 1:10k Street View using [[Bing]] tracing. The hard part is putting in enough effort - tracing and tagging at this level takes a lot of time to get spot on.

Two estates done, another thousand to complete... :-)

Judgling from a very muddy survey expedition today, Sustrans have a lot of work on to link [[Cramlington]] to Bedlington via the old Humford Mill stepping stones over the River Blyth.

The project publicity concentrates on the exciting river crossing, but the section to the South leading to East Hartford will be the real challenge.

The first section starts out in a plantation NE of Hartford. Dropping down a flight of wooden steps, the path lowers down to the flood level of a small river and turns into a muddy swamp.

Heavy rains must swell the 1m wide stream considerably as man-made rubbish hangs in scrubby trees. A small metal bridge crosses the water and the adventure proper starts. The path hugs the side of a V-shaped dean cut by water and the sound of the flow is never far away.

The Council improved the path many years ago with wooden posts forming steps up the steep valley side and small foot bridges crossing drainage pipes. Nature has worked unseen on these foreign structures and most are rotten.

Several large trees now lie across the wet path adding to the obstacle course of briars and muddy scrub. In one spot, a topped trees have formed a considerable dam and forced the water to eat away at the bank path.

Once through the dean, the much larger River Blyth takes control of the terrain and things level out. 18 stepping stones take you across to Humford Mill and on to the first bit of tarmac. This is an ideal cycling spot, with a car park and a quiet road leading up hill to St Cuthberts Church in Bedlington main street.

A information board proclaims this area as Bedlington Country Park, and if Sustrans can get through the mud, fallen trees and scrub, this will be a great cycle route. It avoids both Plessey and Bedlington bridges which have steep hills with narrow bridges that cars have trouble staying on, making cycling rather dangerous.
http://www.sustransconnect2.org.uk/schemes/project_detail.php?id=137

To celebrate mapping Cramlington by bicycle, I decided to treat myself to a new steed. After Wiggle delivered a large box containing a new Focus Red Falls, I have been itching to get out and about on the newly mapped Cramlington Cycleway Network.

Tonight, I decided to ignore the 7C air temperature and winter night to do something different. Starting at Cramlington, I followed National Cycle Route 1 down past Horton Grange, and through new housing developments to Blyth Adsa. What is normally a horrible place to visit by car (bad traffic!), is well appointed with excellent cycleways.

Coming back though Bebside, I decided against the easy route and turned off past an old civic amenity site to follow an old road pre-dating the modern A189 trunk route.

Pedalling along on tarmac being slowly reclaimed by nature, my head torch was the only shaft of light piercing the winter night. Slowing only where a flash of hedge showed a sharp bend, the cold night was quickly forgotten as the exhilaration of exploring the unknown took hold.

The orange sodium skyglow of Cramlington confirmed the Garmin GPS was pointing true, as eerie pinpricks of light appeared. Ah- horses have white eyes!

The glow from The Three Horse Shoes pub looked inviting, but I left the empty picnic tables and headed for home.

Hathery Lane is now mapped. Unused by vehicles for many years, but a great short cut for cycles.

As a break from mapping my home town of Cramlington, I uploaded a short trace made aboard my canal boat cruising the Grand Union Canal.

Navigating a 57" (17m) boat along a narrow waterway is actually not that difficult as the typical speed is a mere 4mph (6km/h) and after 150 years of development, most hazards have been resolved. Although the UK canal network extends throughout country, a typical day's cruising may only encounter a junction every few days.

So, if navigational hazard avoidance and routing are easy peasy, what role is there for digital mapping and GPS?

Well, canals were planned in the 18th Century when roads were terrible and many towns did not exist. What may have been a busy coal mine wharf has been forgotten and is now the middle of nowhere. The feeling of traveling along a secret route is one of the main charms of canals, except when you run out of toothpaste need to find a shop on foot!

Mass market GPS navigation devices have useful Points of Interest, but usually omit waterways and public footpaths to concentrate solely on roads. Other digital maps may include canals, but exclude details such as bridges and locks which are used as reference points.

The ability to add domain specific features to the OSM map and then render the result for your needs is a killer feature, and one that got me started mapping cycle routes in Cramlington.

My short trace around Blisworth Tunnel on the Grand Union Canal was limited to half an hour by the the rainy weather rather than the capacity of my Nokia N800, but logging once a second at 4mph gives impressively dense data points.

The current canal data is missing several sections from Norton Junction to Wedon Bec, but it's good to see that one of the best pubs I visited is mapped. The Narrow Boat on the A5 (Watling Street) is included, along with amenities such as pharmacies in nearby towns.

If only there was a tag to say the pub food is excellent, but don't sink too many excellent pints of Bombadier as the steps leading back down to the canal are very steep...