joost schouppe's diary

Recent diary entries

First steps in historical OSM analysis

Posted by joost schouppe on 7 May 2014 in English (English)

EDIT: yeah, so my little hosting package didn't agree with your interest (you consumed 12 times my allotment). Fortunately the nice people at came to the rescue and offered me some space. Thank you Ben!

I have a big scheme in my head to do somehing fun with OSM data. Unfortunately I'm still taking babysteps. Still, here is one step that makes me pretty happy: a map of the evolution of La Paz, Bolivia. EDIT: as I'm a disaster in reading manuals, I didn't add timestamps to the first few tries. I'll re-run them with timestamps when I get the time.

La Paz, Bolivia

Gent, Belgium (with timestamp)

I can make an animation like that for any bounding box with just a couple of minutes work (and some waiting time, depending on the data-density of the area).

In fact, doing this is extremely easy. It still took me two months :) All you have to do is follow the instructions here: (this was very helpful too:

Only for me, that meant setting up a VirtualBox with Ubuntu, understanding how to install software on Ubuntu and how to fix messed up installations, getting Ubuntu to be able to read data from my host Windows 8 laptop. A big challenge was also not throwing the laptop out of the window (everyone LOVES Windows 8, right?). I could have not done this without Mazdermind Peter who didn't just make the data available in a workable format and the tools to work them, but also gave me personal support. Eternal gratitude and what not. Also free Belgian beer (or chocolate) on any future IRL meetings.

If you want something similar for a bounding box that interests you, let me know. Just send me a bounding box made with and send me the "EPSG:4326" line. That way I can stupidly copy paste the coordinates.

Next step: creating yearly statistics about the state of the map.

Here are some requests:

Krakow, Poland

Silesia, Poland

Lubumbashi, Congo: region, urban area, city center

Kathmandu, Nepal

Norco, California

EDIT: Peter suggested I make an extract of my setup for him to distribute. That would mean you can install VirtualBox (easy), load up a copy of my VM (should be easy), write three lines of codes and you have your thing (easy).

EDIT: There are some bugs visible (Krakow, Kathmandu). It might be I failed to do an update for some of the involved packages. If after a re-run they still show, I'll try and make some bugreports for Peter.

Location: Barrio Brasil, Santiago, Provincia de Santiago, XIII RegiĆ³n Metropolitana de Santiago, 8330130, Chile

using and fixing admin areas

Posted by joost schouppe on 7 April 2014 in English (English)

I woke up one morning, and realized I needed a reusable dataset of all the communities in the world. Not just X-Y, but administrative areas. Obviously, I started looking on OSM. With a bit of playing around (and a little help from my friends), I had a nice set of admin areas of various levels from OSM in a shapefile. Then I started noticing holes. If a country is mostly made of holes, you know there is no data. But if there are a few holes, well, something is fishy. What happens is, there are no extraction tool that can make an admin area if there are gaps. A line is not an area, only a closed line is. Borders in OSM are relations. These are collections of lines, joined together in virtual union. Often, someone deletes one of these lines, and replaces it with a more detailed version. That's sort of OK, but then you have to add the new line to -all- the relations the old one was part of. This is of course very exotic to new mappers, and even experienced mappers don't always seem to care.

Data use = data cleaning

Data will only get fixed, if they are used. And even on the forum, I saw people referring end data users to other sources to get their admin areas. It is complicated to extract a dataset with borders from OSM. So why care about this data? It shows up okay on OSM even if it's broken. BUT, user Wambacher made this great tool to download shapefiles by country with all available admin areas. So now that it's easy to use the data, please help maintain it.

Fixing things up

I tried several tools to help fixing things. I have a global focus, so I've mostly been doing fixups on the admin level right below the country. Often states, sometimes departments, etc. If you're going to fix a certain area, you're going to need other tools (see below) - choose a level to fix using - check which are missing - find the relation which is broken (or rarely, missing). Search doesn't always work. Zoom to a frontier which at one side is ok, at he other isn't. Click on the frontier and find the relations it is part of. Copy the ID. - Vizualize the relation with an url like this: This shows obvious defects. If defects are more subtle (little holes, almost-junctions), go to and paste the ID.

Causes of trouble

During the fixups I found many different types of errors. Borders are basically always the result of imports. Duh. Messing around with borders are a good way to understand why imports are controversial. It's easy to do it wrong, and hard to do it right. Sometimes there is data from the original shapefiles that were used, like area=123. In some cases, the original polygons are still there. And where it gets really messy, is when you add detailed borders from source X on top of general borders from source Y. It takes a lot of time and effort to clean that up, and importers don't always get around to finish it all up. Once the data are there, information may be redacted, because the original data wasn't compatible with our licence. Most often: no commercial reuse, the menace of all open data users. And redaction leaves a mess. Another source of trouble is including both seafront and a maritime border. These should be separated. Apart from that, most errors come from beginners or experienced users alike who delete a line and replace it with something else. Simple solution: use shift-click to improve geometry instead. That makes understanding the history of an area much easier too.

In depth error checking

If you want to go in depth in a certain area, these tools will come in handy: > Click "none" (bottom left), then only activate the "Boundaries" checks

This even vizualizes all the broken relations, but it's just plain depressing to use:

community building

Posted by joost schouppe on 8 March 2014 in English (English)

So, about six months after promising Ben Abelshausen, I'm finaly organizing a MeetUp in Antwerpen.

As I know organizing isn't quite my forte, I've been thinking a lot about what other thing one can do to make OSM more social today. As opposed to "how could we make it more social if we were programmers with all the time in the world". So, there was one piece of very low hanging fruit we identified on the last "monthly" "summer" meetup in Gent . We could just let all the new contributors in our area know that OSM are people. So when you join, you wouldn't think OSM is somewhere you just dump some data, but a place where people actually meet and collaborate on a common dream. (in case you wondered: yes, nerds can be romantic)

So I made a Google Spreadsheet where all new Belgian contributors are listed quite clean. You can click a link to their profile and send them a welcome message. After you sent it, you just add some info to the spreadsheet, so the same new user wouldn't be getting more than one welcome message. I made a draft standard welcome message (in Dutch) anyone can use or modify of completely throw away and make something better.

The welcome letter is available on a Google doc. The spreadsheet is available as a read only here.

I don't want to be doing this alone. I would like to share both Google Documents with anyone who wants to help out. All you need is a Google Account. We'll obviously need a French translation. And I would like there to be different letters of introduction.

If you're interested, I used (of course) one of neis-one's services. This RSS is read by an IFTTT recipe into Google Docs. It's read a bit messy, so I introduced a second worksheet that cleans it up a bit with some basic formulas.

Older Entries | Newer Entries