It's important to see what exactly happened to features in a changeset. This means identifying the state of each feature, the history, including geometry and tags that changed. The OSM changeset page doesn't give you a clear idea of what happened in a changeset - you see a list of features that changed, and the bounding box of the changeset.

The changeset XML from OpenStreetMap only has current version of the features that changed in the changeset.

Overpass offers augmented diffs between two timestamps that contains current and previous versions of each feature that changed in that period. We put together an infrastructure that queries Overpass minutely, prepares changeset representation as a JSON, and stashes them on S3. The augmented diffs are also cached on S3. This means that the load to Overpass instance can reduce drastically while many of us are looking at the same changeset.

This is directly used in changeset-map - a utility to visualise OSM changesets.

JSON

The cached changeset JSONs are available here: https://s3.amazonaws.com/mapbox/real-changesets/production/changeset-id.json. The JSON looks like this for a changeset by user Rezhin Ali.

This is inspired by the work Development Seed did with Planet Stream. We use osm-adiff-parser to convert the augmented diff to changeset JSON.

// 20170411184718
// https://s3.amazonaws.com/mapbox/real-changesets/production/47656996.json

{
  "elements": [
    {
      "id": "4787752634",
      "lat": "36.1823442",
      "lon": "44.0158941",
      "version": "2",
      "timestamp": "2017-04-11T13:12:35Z",
      "changeset": "47656996",
      "uid": "5323129",
      "user": "Rezhin Ali",
      "old": {
        "id": "4787752634",
        "lat": "36.1823442",
        "lon": "44.0158941",
        "version": "1",
        "timestamp": "2017-04-11T08:02:21Z",
        "changeset": "47649032",
        "uid": "5323129",
        "user": "Rezhin Ali",
        "action": "modify",
        "type": "node",
        "tags": {
          "name": "ێەبد مەنان",
          "name:ar": "ێەبد مەنان",
          "shop": "car"
        }
      },
      "action": "modify",
      "type": "node",
      "tags": {
        "name": "Abd Manan",
        "name:ar": "Abd Manan",
        "shop": "car"
      }
    }
  ],
  "metadata": {
    "id": "47656996",
    "created_at": "2017-04-11T13:12:34Z",
    "open": "true",
    "user": "Rezhin Ali",
    "uid": "5323129",
    "min_lat": "36.1823442",
    "min_lon": "44.0158941",
    "max_lat": "36.1823442",
    "max_lon": "44.0158941",
    "comments_count": "0",
    "tag": [
      {
        "k": "created_by",
        "v": "MAPS.ME ios 7.2.3"
      },
      {
        "k": "comment",
        "v": "Updated a car shop"
      },
      {
        "k": "bundle_id",
        "v": "com.mapswithme.full"
      }
    ]
  }
}

Empty changesets

It's possible that certain changesets are empty. They could have been opened, but failed to upload changes due to unreliable network, and eventually gets closed in 60 minutes. Empty changesets are not cached.

Long changesets

Changesets can also remain open for a long time. For example this one from user Manuchehr was opened 36 mins. Experienced users like to survey outdoors, and upload data in bulk. Some editors also don't close changesets automatically. Idle changesets get closed eventually after 60 mins.

When features of changeset comes through in a later minutely diff, we update the cache on S3. This will ensure, changeset remain complete.

Database transactions and augmented diffs

A changeset being closed doesn't mean that all features that changed have been committed to the OSM database, and appear in the minutely diff right after. Some features may take longer to commit to the database, we handle these by updating the augmented diff from S3, and then recreating the changeset JSON. You can read more about this case here.

Missing changesets

Changesets that are after March 1, 2017 are cached. We are considering doing a slow backfill, but this is entirely dependent on Overpass. If you see something missing, or unclear, please open a ticket and let us know!

I'll be at State of the Map in Brussels this week presenting all of our work on validating OpenStreetMap data. At Mapbox, we spent the last few months looking at changes that happen in OpenStreetMap closely - geometry, tags, users and the community. I'm really excited to share what we learned, and to open the conversation on how the community can focus on keeping the map from breaking while growing. I'll be presenting the tools we have been building, insights about problematic changes, response, mechanics of communication and more during my talk.

Our team just published an approach to validating OpenStreetMap data - talking about identifying problematic changes, inspecting them, communicating and eventually fixing. Let me know what you think!

What makes OpenStreetMap special is the community. The community is what makes OpenStreetMap a truly self-healing map. The community is the map.

Map of recently reverted changesets

If you haven't already, watch Sanjay Bhangar's presentation at State of the Map US.

Looking forward!

I'm a software developer at Mapbox based in Bangalore, India, co-leading our data team. Prior to Mapbox, I've been part of projects at Karnataka Learning Partnership, Moabi, and organizations like the Center for Internet and Society and Tactical Technology Collective. What makes writing this note special is that I'm in Doha this week, talking about OpenStreetMap data, tools and running a Digital Humanitarian workshop along with Heather Leson and the Qatar Red Crescent Society. In doing so, I've been able to engage entrepreneurs, engineers and scientist - introducing them to the largest living map, and OpenStreetMap as a project that is truly "moving the map." What makes me believe in what we do at HOT and OpenStreetMap is the power of bringing people together.

I have been part of OpenStreetMap since 2008. Since the Haiti activation, I regularly lurked on the HOT mailing list and occasionally mapped. More than mapping, I was drawn to how the OpenStreetMap software infrastructure worked and the idea of mapping anything and everything. I started off my career building data infrastructures and advocated the use of OpenStreetMap software for other use cases. The recent Nepal activation is when I truly got involved in HOT. Right after the earthquake, our team in Bangalore got together to start mapping priority areas. I was communicating with folks at the Kathmandu Living Labs and relaying requests to map, make map data available for download, and working with actors on the ground to design maps for print. I also got a chance to help and learn about imagery acquisition through Mapbox and the Indian Space Research Organisation.

India doesn't have a strong OpenStreetMap community. We managed to build momentum and groups across different cities - New Delhi, Pondicherry, Ahmedabad, Hyderabad and Mumbai to contribute to the Nepal mapping tasks. Arun and I did training over Skype, phone and hangouts. It was a surreal experience to have been in the middle of a crisis at the same time I realise from the conversations with everyone at KLL that it was very real. I've been part of the most recent activations in Mexico and Afghanistan, rallying communities to map and getting in touch with people on the ground who could tell us more about what's important. I have run events, trained individuals to map and spoken to journalists and the media, in India, about what we do at HOT.

I think HOT is in an excellent position to do something that will change how crisis response works, from data acquisition and communication to controlling and supporting distribution of aid. Our community is strong, technology is robust and we are continuously improving, as we learn from each activation. In my mind, HOT's challenges are more than drumming up people to map. In times of crisis, people want to help. What's missing is the link between traditional disaster response agencies and HOT. During the Nepal and Afghanistan activations, I have reached out to local aid agencies in India like the Red Cross chapters, Goonj, Indian Airforce - they were all responding to Nepal - but I failed to make any connection and failed to see our maps being used by them. All the journalists I spoke to were interested in the story how a group of digital humanitarians are helping map Nepal and were not going to help speak about how these maps need to be used by local agencies. In Afghanistan, I heard firsthand that the agents on mission have practically no communication channels. HOT has work to do to break these silos, work with partners to and make our maps known and used. I'd like to start small, understand how local aid agencies work and make our ways flexible to collaborate closely.