pnorman's diary

Recent diary entries

Building Osm2pgsql for Testing

Posted by pnorman on 9 January 2015 in English (English)

This is a copy of a blog post on my site.

Recently I needed to run a bunch of osm2pgsql tests in a virtual machine, so optimized the process. You would not normally do development in a VM, but it's useful for testing. It's particularly useful for me because my old Ubuntu server uses a development version of PostGIS which doesn't work with the testsuite.

I was using VirtualBox to run the VMs, and started with a base Ubuntu 14.04 server install on a virtual drive of at least 30GB. The flat-nodes tests require a lot of space.

Starting with the base, I updated the OS with

sudo apt-get update && sudo apt-get dist-upgrade && sudo shutdown -r now

This gave me a VM I could clone and use for other projects. Starting from a cloned VM, I installed the dependencies. Copying and pasting into VirtualBox can be a pain, so a bit of clever shell expansion minimizes the text I need to type

sudo apt-get install libtool g++ protobuf-c-compiler postgresql-9.3-postgis-2.1 \
  lib{boost{,-system,-filesystem,-thread},xml2,geos++,pq,bz2,proj,protobuf-c0,lua5.2}-dev \

The osm2pgsql testsuite requires a custom tablespace for some of the tests, as well as normal PostgreSQL setup.

sudo -u postgres createuser -s $USER
sudo mkdir -p /tmp/psql-tablespace
sudo chown postgres.postgres /tmp/psql-tablespace
psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" -d postgres

To get osm2pgsql, I was cloning my own git repo instead of the normal one, but if you wanted to get the source, build the latest version, and run the testsuite, I did it with

git clone
cd osm2pgsql
make –j4
make check

make check tends to take awhile to run, as it has to do several imports and create a lot of databases for testing. It also has to write a 20GB flat nodes file.

Langley Imagery

Posted by pnorman on 3 November 2014 in English (English)

Note: Repost of

I just got the recent Langley 2014 imagery from their Open Data program loaded onto my server, and I’m impressed. The new imagery has at least as good spatial accuracy, while having better resolution, colours, and being more recent.

Comparison between old and new imagery

In the next few days, I want to release the new layers.

Unfortunately, while Langley is using the PDDL license, other cities in the region are using custom licenses, meaning I have to enquire with each one individually, taking significant time. If they were all using standard licenses, I would have rebuilt my BC Mosaic layer by now and this post would be about updating it to include new sources.

Location: Salmon River Uplands, Township of Langley, Greater Vancouver Regional District, British Columbia, Canada

OpenStreetMap Carto v2.22.0

Posted by pnorman on 6 October 2014 in English (English)

Labels on Shelf

OpenStreetMap Carto v2.22.0 has been released. This release focuses on labels.

The biggest change is a rewrite of landcover labelling. A landcover label is text connected to a background colour or pattern rendering, and not connected to an icon. This has been demoed extensively, and well received. It was also sent to the mailing list. The big changes are making colours better connected to the background, rendering labels on some features where they weren't before, and sizing labels based on area.

The last deserves a better explanation. Previously, the selection and choice of what labels to render didn't include area. It now does, avoiding placing labels on features that are only a few pixels in area at low and middle zooms, and selecting font size based on feature area. This results in a much more sensible label placement, more readable labels, better selection of what labels to place, and in many cases, more labels without impairing readability.

There remain some minor issues that can be followed up on (e.g. Glacier label colours)

Other changes were

  • Ordering fixes
  • Concurrent ferry rendering
  • Line wrapping improvements
  • Road label improvements in complex areas
  • Small island improvements

A full list of changes can be found on GitHub.

Labels photo Copyright CC BY 2.0 Kelvynskee @ Flickr

Central Park mapping party

Posted by pnorman on 4 June 2014 in English (English)

I am planning on hosting a mapping event in Central Park (, in Burnaby, BC.

My tentative date is Saturday, June 14th at noon, but if people want a different time I could shift it.

The park is a major park in the region, but under-mapped, with potentially not all of the trails.

Some of the things I'd like to get mapped are

  • More appropriate tags for the trails. highway=track is probably not accurate
  • The surface of trails
  • Locations of fitness equipment in the park
  • Bathrooms
  • Picnic areas
  • Other park infrastructure
  • Anything missing or interesting

At this time of year, it should be nice sunny weather, and the shade from the trees should be welcome.

I intend to bring my mapping kit (camera, GPS, etc), as well as field papers type printouts on larger paper for us to mark up. I'm not planning on bringing a laptop, or if I do, it's staying in the trunk.

After mapping, we can go somewhere nearby for food. I'd also like to discuss holding regular events.

If you're interested, please let me know so that I know other people will be coming and to attend on time myself! I also need to have printouts made in advance.

(Cross-post of

Edit: Date change to 14th

Location: Garden Village, Burnaby, Greater Vancouver Regional District, British Columbia, Canada

Organizational mapping policy

Posted by pnorman on 14 May 2014 in English (English)

This is a cross-post of

We have more and more organizations and businesses mapping in OSM. Multiple organizations have been conducting paid editing in Europe and the US. This generally comes to light after complaints are made - with the company usually not identifying who they are, what their goals are, and what they want, beforehand. There have also been difficulties determining what has been mapped on behalf of an organization.

We will likely see more of this type of editing in the future, and while not necessarily bad, there are differences between it and normal editing. Recent events in a project similar to OpenStreetMap - Wikipedia - have demonstrated that the participation of organizations in data editing can occasionally lead to misunderstandings or disharmony in the project, particularly where a lack of transparency is involved.

For this reason the DWG is considering if it is necessary to issue guidelines for organizational editing. Some previous discussion is at

There are some activities we do not want to cover in the guidelines

  • Unorganized editing by employees, e.g. a shop owner adding their shop or nearby details to the map

  • Editors mapping in response to a contest or similar where the contest organizer does not have the power to require them to edit

  • Individuals who, on their own accord, decide to participate in an organised effort or challenge, like local mapping parties, Mapathons, HOT projects, etc

Some possible guideline requirements could involve

  • Disclosing those who are directing them (e.g. employers or who they are contracting for) on the users page

  • Creating a wiki page with links to user pages of users mapping under an organization's direction

  • Requiring those working on broader projects to communicate and get feedback from the community before starting

  • Requiring disclosure of proprietary third-party sources used. Organizations may have data from third parties that they can legally use when contributing to OSM, but aren't able to directly show others the data

  • Maintaining separate accounts if doing both personal and organizational editing

The extent of editing activities covered is something else that needs to be discussed.

Some types of activities that could be covered are

  • Teachers requiring their students to edit OSM as part of a course

  • Consultants editing for multiple clients

  • Being required to edit as part of an employment relationship

SEO spammers would be covered by this policy, but are not the target. They would ignore it, so we'll just end up using the existing tools of reverting and blocking.

Paul Norman
For the Data Working Group

State of the Map kickoff/morning notes

Posted by pnorman on 19 April 2014 in English (English)

On the schedule for the Friday and on Saturday morning was the kick off party, registration, let's map!/OpenStreetMap in your organization sessions, and coffee break.

Kick off party

Thanks to a delayed flight I ended up going to the Mapbox Garage. The entrance is on the rear, so the taxi driver was confused getting there. Since I was there early, I helped with the setup, and schlepping the alcohol over from the store nearby.

The party got too loud, but I was able to catch up with a few people. We ended up running out of bottled water, and the taps were inconvenient. I walked back with a few others staying at the Washington Plaza hotel.

Saturday morning


Registration went smoothly. Conference t-shirt just said State of the Map, not State of the Map US, which seemed odd. I think most of the conference people handling registration were paid staff, which was a difference.


There was a welcome talk which I thought was rather good at the time, but seems it wasn't very memorable, because I can't actually remember much of what was said!

The conference had over 500 people check in at registration, a big growth from the 2009 SOTM-US which had about 50 people.

First session, coffee break

I ended up talking with the people from Amazon for all of this session, so missed the talks. I'm hopeful that they'll be able to commit some EC2 credits to do some dev work and performance testing. Of course, I was the same way after previous conferences and nothing came of it.

It doesn't sound like they're willing to commit to helping any infrastructure with resources in any ongoing manner.

CJK fallback fonts - testing needed

Posted by pnorman on 13 January 2014 in English (English)

Right now the main stylesheet uses Unifont as a fallback font to render Chinese, Japanese and Korean (CJK) characters, as well as any other characters not present in the DejaVu font. Unifont is mainly designed to support all characters, and is not designed to look good.

I'm looking at Droid Sans Fallback, a free font developed for Android, and evaluating if it would be a better fallback font than Unifont. Because I don't read Chinese, Japanese or Korean, I could use help.

I have prepared a demo at with three layers: conventional, tiles without any fallback font, and tiles using Droid Fallback as a fallback font.

What I would like is for people to look at the difference between the conventional and Droid Fallback tiles and see which is easier to read for the CJK glyphs. The tiles without any fallback font can be used to find areas where DejaVu doesn't have glyphs and the fallback font is being used.

Some examples

Japanese cities:

Japanese train stations:

Korean cities:

Chinese tourist attraction:

Please keep in mind that

  • My server is not nearly as powerful as, so renders slower and has less cached data
  • Only Asia is loaded on my server
  • The data is a couple of days old and isn't being updated

I would like some feedback on if Unifont or Droid Sans Fallback looks better. Please keep in mind that I don't read the languages being rendered.

Location: Downtown, New Westminster, Greater Vancouver Regional District, British Columbia, Canada

osm2pgsql command lines

Posted by pnorman on 12 November 2013 in English (English)

I had to write some osm2pgsql command lines for benchmarking recently. This is how it feels

tile rendering times

Posted by pnorman on 15 May 2013 in English (English)

Just a preview of what I'm working on right now

Chart of benchmarking results, showing most time is spent on z13 tiles

Location: St Pancras, Somers Town, London Borough of Camden, London, Greater London, England, N1C, United Kingdom

South Fraser Perimeter Road - Surveyed

Posted by pnorman on 4 December 2012 in English (English)

Today I surveyed both the new Westbound Port Mann Bridge and the north portion of the SFPR. Either of these would of been a lengthy trip in itself, combined they took two hours of driving to get traces and I still didn't get everything surveyed.

I used my camera to take photos with an interval of 8 seconds and my eTrex 20 sampling at 1 Hz, then correlated everything in JOSM.

I got off at 152 Street and went along 104 Avenue to cross over at 160th. I drove around the south side of the forest on the slope before heading over to join up with the SFPR, 104 Avenue and 176 Street Extension intersection. Little did I know how many times I would be going through here.

I then headed south to Highway 15, 96 Avenue and Golden Ears Way, heading down Golden Ears to do a U-turn in a side street. I then went straight on to the SFPR through the 104 Avenue intersection. From here it was an easy drive on new quiet pavement with no chance to turn off or turn around until King Road and 138 Street. I was struck by how few cars there were on the road. There were occasional trucks but I didn't see another car going my direction. The view was rather nice in parts

View of the Fraser from the SFPR

After a quick trip along King to verify some road closures it was back on to the SFPR for the trip back. If anything this was quieter than the westbound trip. I saw 3 cars and 2 trucks going the other way and nothing going my way. I then took my third trip through SFPR, 104 Avenue and 176 Street Extension intersection and looped around on the 96 Avenue side of the 15 and 96 Avenue.

I then headed north to get the trunk_link roads. The first one took me along the exit past the construction and a TCP guiding out trucks

TCP along the side of a road

The left-turn route at the intersection is somewhat unconventional. There is a no left turn restriction at the intersection and you are directed past the intersection and onto a loop, connecting you with the SFPR and taking you past the intersection again.

From here I took 176 Street down to 96 Avenue a third time and did the same turnaround as before, taking the loop to Highway 1.

Headed home to the Port Mann I got stuck in traffic from a three-car accident, the first major backup of the new span.

3 car accident on the Port Mann

I saw that there was some new paint for the United Boulevard exit but I didn't get a chance to map it.

The last change before getting off the highway was the merging of exits 40A and 40B into a new exit 40. It looks like this was done to make it easier to navigate by having all the exit 40 traffic exit at the same point and then split.

At this point the SFPR feels like a road without a purpose. It ends abruptly on the west end, going from 80 km/h to 50 km/h, exceeding the MOT recommended 20 km/h maximum speed limit change. When it connects through to the Alex Fraser it will make more sense as well as when it connects all the way to the ferry terminal and the existing 17 becomes the 17A.

I classified the SFPR as highway=trunk. It's very similar to the Mary Hill Bypass (7B) in design. I also used the existing convention of carrying through the higher classification until it has clearly changed. This is why the connector to Alderbridge and the north end of the Oak Street Bridge are both highway=motorway even though their speed limit is 60 km/h.

The west end is a bit weird but it's a bit weird on the ground too, so the map is simply reflecting reality.

Location: Anniedale, Surrey, Greater Vancouver Regional District, British Columbia, Canada

New Lower Mainland Imagery sources

Posted by pnorman on 27 August 2012 in English (English)

After some technical and legal work, I now have a number of imagery layers hosted on a rented server.

These layers cover from Vancouver to Hope in 20cm or better and Lions Bay to Pemberton as 40cm or better. 10cm imagery is available for Vancouver, Richmond, Ladner, West Delta, the North Shore, Whistler and Surrey.

Most of the imagery was taken in 2009, but Surrey is from 2011.

I've prepared a page with links to the imagery and Potlatch2 and JOSM URLs at

Some layers may be slow initially loading at high zooms as they may need to fetch from a remote server.

An example of two layers from there and Bing is Starting in the top left going clockwise the layers are Surrey2011, DataBC bc_gvrd_east_2009 and Bing.

Note: This post is a duplicate of a talk-ca mailing list post

Location: South Cambie, Vancouver, Greater Vancouver Regional District, British Columbia, Canada


Posted by pnorman on 2 June 2012 in English (English)

After nearly a month on a 4-drive RAID0 array...

time ~/osm/osmosis-0.40.1/bin/osmosis --rb planet-120401.osm.pbf --lp interval=60 --wd database=osm user=openstreetmap password=openstreetmap allowIncorrectSchemaVersion=yes 2>&1 | tee apidb.log
5-May-2012 4:27:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.40.1
5-May-2012 4:27:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
5-May-2012 4:27:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.


1-Jun-2012 8:36:48 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline complete.
1-Jun-2012 8:36:48 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Total execution time: 2347785558 milliseconds.

real    39129m46.249s
user    1418m52.368s
sys     415m9.705s

A graph rising to 1.1 TB over 4 weeks

Location: Victory Heights, Sapperton, New Westminster, Greater Vancouver Regional District, British Columbia, V3L2W6, Canada

Surrey Open Data Hackathon next week

Posted by pnorman on 13 November 2011 in English (English)

Surrey, BC is holding an Open Data Hackathon on Sunday November 20th. More
details are at

I will be attending and looking at writing some new conversion scripts.

Does anyone have suggestions for any printed OSM materials to bring?

Location: Sullivan, Surrey, Greater Vancouver Regional District, British Columbia, Canada
Older Entries | Newer Entries