OpenStreetMap

UN Collaborates on Zaatari Camp Data in OSM

Posted by mikelmaron on 4 September 2013 in English (English)

Zaatari, in northern Jordan, is facing huge humanitarian pressure from the Syrian conflict, considered the second largest refugee camp in the world. To aid the response and improve coordination of geographic data in the camp, a remarkable cooperation is underway there among UN agencies, the OSM community, and potentially, the refugees themselves.

Getting Here

As the Syrian crisis deepened, HOT was prompted to activate in January. It's naturally proven difficult to coordinate remote mapping inside Syria, though there has been significant and interesting activity over the past two years.

We've known Edouard Legoupil from UNHCR for a while, through even pre-Crisis Mappers days. We personally overlapped in Nairobi at the start of Map Kibera, and since, we've discussed ways of bringing open approaches to mapping and engagement in camps. Likewise, Lars Bromley from UNOSAT has been a colleague since we collaborated together in Darfur Google Earth layers project, and while he was at AAAS, supplied detailed historic satellite imagery of Kibera for our analysis of spatial extents of post-election violence. Both have innovated, very close to where things are built, and have created space within their organizations to work collaboratively. With the HOT activation, we've been able to come together and do practical work on those ideas for Zaatari.

The unhrc-jordan OSM user has imported roads, toilets, water points, health facilities, mosques, schools, offices, even restaurants. Another partner organization, UNICEF Jordan also uploaded child specific infrastructure (though this was later undone, and re-uploaded by UNHCR for some technical reason). Now, UNHCR is working with REACH to map on the ground in Zaatari, both confirming data and including "informal" infrastructure built by refugees; all the normal services of a city are growing in this place. In the future, there may be opportunity to engage refugees themselves in the processes. In traditional practice, data is shared through colleagues. Through OSM, they become openly collaboratively shared.

UNOSAT Import

UNOSAT is tasked with another piece of the geographic puzzle in Zaatari. They analyse satellite imagery to identify ALL structures in Zaatari, currently at 28174 structures. Lars confirmed that this data could be imported into OSM as well, and that I could help with that process through the UNOSAT OSM User.

Working off the July data, the first challenge of the import was transforming the ESRI Geodatabase released by UNOSAT into a format I could work with. Luckily, I was at OSFAC's office in Kinshasa where I had access to ArcGIS, and could do the conversion to Shapefile. Then, to my surprise, JOSM was able to read the Shapefile directly! In JOSM, I used the under-appreciated filters feature, to select only features with Shelter_St=1 (meaning the structure currently exists, the data contains shelters that no longer exist). Then, mapped the Structure_ column to either shelter (for 1) or administrative (for 2). Also stored the acquisition date, event code, and source, for reference (within a unosat: tag space). And also, the objectid, which could be used for applying updates later on. Removed the other tags. And because of the large size of the upload, used a script rather than JOSM. Here's an example structure from the import.

UNOSAT Conflation

So right before I was ready to do that import, Lars notified me that UNOSAT had released a new analysis. I could just redo the process above with the new data, or use the opportunity to test if it works to update based on unosat:objectid. Yea, did that!

Again, stuck with a GDB, until I was fortunately introduced to the simple and strong GDB Flee. From there, I went about writing a conflation script, which reads in the shapefile and OSM data, detects actions (create, update, or delete, either way), and then outputs OSM XML for examination in JOSM. There might be other tools to do this (please let me know in comments if so), but was useful for me to get my head around the process, a first step at abstracting conflation, and dig into some geo mangling code.

It's not fully tested against all cases, just the situations in this conflation, so putting together proper tests for this later is on the agenda. In this case, after the deletes and creates, the remaining number of structures added up precisely to what UNOSAT reported, so looks fine.

The thing to grapple with here are unexpected situations. The script deals with this by outputting OSM features with a NOTICE tag. Then in JOSM, can use filters to only show those features, and inspect and make decisions manually. For this import, there were about a dozen features that were supposedly collected in the previous analysis, but hadn't already been imported. This was strange, but didn't investigate further and since the numbers added up, decided to include them in the import.

So What?

Interesting results immediately came out from comparing multiple datasets, and putting many eyes on the data. Those dozen "missing" features were highlighting in JOSM Validation as Nodes at same position. The precisely same lat/lon, but different objectid tags. I confirmed they were definitely present in the current set of features, and that the total number of featues matched what's reported by UNOSAT. So perhaps I can conclude there may be duplicates in the source analysis, and hopefully this community view on the data can help improve the authoritative view.

Also interesting to look at how the UNHCR and UNOSAT data interact. Administrative buildings or compounds are polygons in UNHCR, points in UNOSAT. Some compounds contain multiple administrative structures; it might be interesting to associate them via a relation or tagging. Some shelter structures show up with administrative areas; do those structures need to be reinterpretted, or have the areas changed? Now that these are together in a single database, we can script or use GIS to analyse those kind of issues and prioritize further investigation.

One thing that might help is for OSM users to access the imagery directly. In fact, the most recent analysis come from State Dept HIU supplied imagery of Zaatari, so we can perhaps request access through Imagery to the Crowd. We'd want to have some good analysis first, and a clear idea of how mappers could help.

Finally, there's potential to do custom rendering which highlights all the data, and camp specific information. UNHCR is already rendering Zaatari on their data site. It would be good to also show the node buildings in an unobtrusive way, to show how the admin structures and shelter distributions interact.

Location: Share'e Alsooq Alsaudi, 05, Mafraq, JORZATD005, Jordan

Comment from mikelmaron on 8 September 2013 at 13:27

Some feedback and suggestions from taking a look at all our data-work together in the map.

Tags

The flexibility of tags are what make OSM work, when you don't find a representation existing, you're free to make your own. Then over time, others start mapping similar things, and eventually make connections, and then work towards consensus. I think we're seeing a lot of that here in Zaatari, with work from other humanitarian mapping, mapping informal settlements, and general work in OSM. So that's my first point, to look at tags, and see if we can broaden the discussion into new community consensus if needed, and do some clean up on the Zaatari data.

For example, toilets have been getting a look by the general OSM community; HOT has worked on mapping humanitarian attributes; and Map Kibera did a lot of work on different toilet types. http://wiki.openstreetmap.org/wiki/Toilets http://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Tags http://mapkibera.org/wiki/index.php?title=Tagging/Watsan

For a toilet like http://www.openstreetmap.org/browse/node/2446725252, I'd suggest adding a new value to "toilets:disposal", instead of new tag "latrine_type"; and then use "disused", rather than "status", only when there's a problem with the facility. http://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Tags/operational_status. Also, for "source", if you use a new value, good idea to document it here: http://wiki.openstreetmap.org/wiki/Key:source

For a water point like http://www.openstreetmap.org/browse/node/2446736025, "type" is typically already used as part of relations, and doesn't specifically deal with water points; might have "tank_type", split up into http://wiki.openstreetmap.org/wiki/Operator, and keys in the "watsan:" namespace or such to capture the kind of tank and volume.

Also, for the kitchens, they're tagged as "drinking_water", but since the primary function is kitchen, we might just consider using a new tag "amenity=kitchen" (you don't find commons kitchens many places. http://www.openstreetmap.org/browse/way/227361542

Anyway, could go on and on about tags. There's certainly a lot to figure out.

Linking data

With multiple sources of data, we're starting to (I think) see the same things represented twice. This structure from UNOSAT http://www.openstreetmap.org/browse/node/2440575510, and this kitchen from UNHCR http://www.openstreetmap.org/browse/way/227361542 are very near each other, and I guess the same. We could catch more of these through analysis. I wonder, how can we design a process to catch these, and merge features during imports and updates.

Similarly, http://www.openstreetmap.org/browse/node/2440522483 is tagged as a shelter, but right near a water point and toilet ... is this perhaps actually an administrative structure?

Hide this comment

Comment from mikelmaron on 11 October 2013 at 15:03

Update, on October 3, UNOSAT released a new analysis. http://www.unitar.org/unosat/node/44/1832. The officially released Shapefile version did not contain the objectid, so asked for another extract, posted at http://cern.ch/unosat-sdn/temp/Al_Zaatari_Shelters_20130930_formikel.zip. Had to adjust the conflation script slightly for changes in columns. Ran without issue, except for one warning "ERROR Deleted feature missing in past import: 63911". This feature was both detected and deleted on the same day. Otherwise, all stats match up with reported changes from UNOSAT. Import is here

There's some concern about the tags, unosat:acquisition_date, unosat:event_code and unosat:objectid, to be discussed on imports list.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment