OpenStreetMap

Ivan Garcia's diary

Recent diary entries

New script to convert SHP into OSM relation boundaries (SHPtoOSMBoundaries.py)

Posted by Ivan Garcia on 26 September 2016 in English (English)

Hi OSM community,

my name is Ivan Garcia .

I helped many regions of the world to convert their official boundaries into OSM format and import them into OpenStreetMap following, of course, the OSM import guideline.

So I decided that since a lot of people are doing these conversions manually therefore wasting a lot of time, I created a python script that you can see and improve, it is called SHPtoOSMBoundaries.py

It already exists an script called polyshp2osm.py but it is not specific for administrative boundaries and it is not capable to extract and join together different admin_levels into a single OSM file. That script works perfectly for converting all the features from SHP to OSM format.

Normally, what OSM contributors have is a SHP file received from some official institution or downloaded through some OpenData portal, this example will show the user case of Fiji islands SHP file.

Original SHP loaded in QGIS

If we open the SHP it with QGIS, we see that it contains a lot of boundary polygons, and that each of the polygon contains a ID tag (in this case the TID) and another tag that identify the name of the deepest boundary(the tiniest), or in OSM, the one with the highest admin_level number. In our example this tag name is TIKINA(which corresponds to the village boundary in Fiji)

Also most of the times, from the same SHP we want to obtain the upper admin level boundaries, in this case we also have the PROVINCE ID (PID) and the tag name (PROVINCE).

So we know find out that:

  • For the level 8(village) we have TID and TIKINA tags
  • For the level 6(province) we have PID and PROVINCE tags

Before we can run the script, we need to create convert the polygons into lines, this is easily made with QGIS using the menu option Vector -> Geometry Tools -> Polygons to Lines

Polygons to Lines option in QGIS

So the result looks like this, a lot of lines that overlapped one to another: After converting the polygons into lines

but OSM cannot have repeated ways inside the relation boundaries, so we will break this lines and make them unique.

Luckily, the GRASS plugin in QGIS provides us of a function named v.break: v.break function in QGIS by the GRASS plugin

Once generated the result breaked lines, we export as GEOJSON format to a filename “fiji_splitted.geojson” for example, also we will export the original SHP boundaries file into GEOJSON format, example to “fiji_level8.geojson”.

Now is time to run the python(with python 2.7) script, you will need the python library “shapely”, before we run it, we edit the script to change the needed parameters, that is:

  • ALL_LEVELS_GEOJSON: filename that contains the original boundaries SHP file.
  • SPLITTED_WAYS_GEOJSON: filename with the broken, splitted ways.
  • DEEPER_LEVEL: level number and uniquetag and nametag that we discovered, in our case: DEEPER_LEVEL = {"level":"8", "uniquetag": "TID" ,"nametag":"TIKINA"}
  • OTHER_LEVELS: a python list with the other upper levels that you want to extract also and mix together, in our case: OTHER_LEVELS = [{"level":"6", "uniquetag": "PID" ,"nametag":"PROVINCE"}]

Optional: There is an extra parameter called MAINTENANCE that you can set up to True to let the script to clean up the ways in case that the GRASS v.break did split the ways a bit too much, having too many ways into OSM. When the script is runned this way, it will create a “tofix_splittedways.osm” that can be loaded into JOSM to fix and join those little ways into a bigger one. Then make sure to save the fixed result as GEOJSON to the same filename of the splitted ways we had before, and change the MAINTENANCE to FALSE before reruning the script.

Running the script will create a “final.osm” output which is the file that we need to import into JOSM in order to upload into OpenStreetMap, preferably using the JOSM multiple-parts upload option.

WARNING for uploaders: Of course, before you upload the generated result, you need to be an experienced OSM user and understand pretty well what are administrative boundaries, otherwise, learn about them see examples, ask in the forums, etc.

Before uploading, you need to see what boundaries already exists in the area you are supposed to upload(I assume you studied the area since you decided there is a lack of boundaries), for example in JOSM you can get all the boundaries of a region, to do so you can use the option “Download from Overpass API”, (you will need to have the Expert Mode activated) in JOSM. Then put the following Overpass query: ( way["boundary"="administrative"]; node(w); rel(bn)->.x; way(bn); rel(bw); ); out meta;

I would like to repeat again that you need to follow carefully the OSM Import Guideline before uploading or deleting anything.

Once you have made sure that your new boundaries are correct, accurate, exact(you can compare with satellite imagery in case some boundaries match some rivers, coastline, etc), my regular approach when I have to upload many automatically generated boundaries into Openstreetmap is the following:

  • Carefully clean up the previous boundaries useless(if any), if necessary, contacting the user or importer who did the previous one, also make sure you have permission by the data source to do so with the OpenStreetMap license, etc.
  • Upload the OSM file using JOSM multiple part (maybe with chunks of 500-1000 objects, depending on the my internet speed).
  • How to join our uploaded boundaries with the neighbor boundaries: Now this is a very careful part and you need JOSM experience with relation editing since you will have to modify already existing OSM boundaries(sometimes delicate country borders). My advice is not to delete the ways of the existent neighbors boundaries but YOUR border ways, and add the already existing ways into your new boundary relations(if necessary you can split those ways).
  • In case that you need to join your boundaries with a country border, NEVER delete the already existent border, since their OSM way ids may be being used by some 3rd party applications, if inaccurate, you can just align them or split them. There is a great online OSM tool created to verify if you messed up with some country border.

Here is how the result looks like: Final boundaries in OSM

In you have doubts or errors, you can just write a comment here or add a issue into the github of the script.

I hope it helps saving lots of time to OSM contributors.