(Hallo! I’m Martin Dittus, a PhD student at UCL. You can read more about my research in an earlier post.)

The volunteers of the Humanitarian OpenStreetMap Team (HOT) and its affiliated projects have spent many thousands of labour hours on the creation of new maps for humanitarian purposes. Yet mapping all the undocumented and crisis-stricken regions of the world is a formidable task. The 2014 response to the Ebola epidemic illustrated this well: even after months of work by thousands of volunteers, the new maps of Central and West Africa are still nowhere near complete.

Many people within HOT now believe that this can best be addressed by growing the community by a few orders of magnitude. An MSF article about Missing Maps articulates this ambition:

To reach our goal, we need the Missing Maps Project to be the biggest instance of digital volunteerism the world has ever seen.

So let’s say we’d want to grow HOT to a million volunteer contributors. How can we train new contributors at that scale? What are our barriers to entry? How can we retain contributors once they’ve had first experiences? Etc… many open questions.

As a first step let’s learn from existing experience. How does engagement compare across the different mapping initiatives right now? Let’s start with a simple comparative study.

Comparing three large HOT initiatives

I’m particularly interested in the engagement profile of first-time contributors: people who may have OSM experience, but who have never before contributed to HOT. How much work do they provide in the first couple of days? How long do they stick around?

In this post I’ll compare the first-time contributor engagement profiles of three initiatives. Each has a different purpose, and a different mode of organisation:

• Typhoon Haiyan (TH) in Nov 2013: A high-profile and urgent initiative. A first “CNN moment” which brought many newcomers to HOT. Accompanied by a larger number of one-off mapathons around the world.
• Ebola Response (ER) throughout 2014: A high-profile, multi-month sustained effort. A large amount of media coverage. Coincided with an initial wave of monthly mapathons in several cities.
• Missing Maps (MM) from Nov 2014 onwards: A larger initiative across a range of humanitarian causes. Proactive, low in urgency, with less media attention: the focus is on community-building. Monthly mapathons, heavy use of social media for promotion.

I’m using the OSM edit history as the basis for my analysis, focusing on an 18-month period from from 16th of June 2013 to 15th of December 2014. During this time, 1,582 first-time contributors joined HOT to participate in one of these three initiatives, joining one of about 100 projects. (There were many thousands more contributing, but for now we’re just interested in first-timers.)

Here’s a timeline of when these contributors first joined, with a bubble for each new contributor: Each new contributor is visualised with a bubble. Bubble sizes represent the amount of labour hours the person contributed in the first 2 days. Contributors are ordered vertically by their OSM ID: older user accounts at the top, new accounts at the bottom.

For each of these contributors we’ll build an engagement profile. For the purpose of this analysis I’m using quantitative measures of engagement, these are easy for me to produce across a wide range of projects:

• Short-term activity: labour hours, contribution rate in the first two days.
• Short-term retention: the share of contributors who remain active in HOT on day 2.
• Long-term retention: the share of contributors who remain active in HOT in month 2 and 3.

Findings: initial activity and retention of first-time contributors

When we model first-time contributor engagement in this way we can see some similarities across the three initiatives, but also some striking differences. I’ll discuss five key observations.

1. Baseline activity in the first 48h is surprisingly high! Many first-time contributors participate for multiple days in a row. The median contribution activity is ~70 mins in the first 48h. This may sound small for a typical volunteer organisation, but for an online project it’s massive! We further find that between the three initiatives, MM contributors map at the slowest pace. We’ll come back to that in a second. Median contribution activity in the first 48 hours: labour hours (left) and contribution rate (right, in edits per hour).

2. Prior experience affects performance. More experienced users tended to contribute faster and work for more hours, and come back the next day. This effect can be observed globally, and for each of the project groups we observed. This either suggests that there is a training effect for OSM users which is transferrable to HOT, or a self-selection bias: contributors who enjoy mapping may simply be more engaged in general, be it in HOT or other OSM activities. Distribution of initial activity by prior OSM experience: the amount of labour hours l48h (left) and the rate of contributions c48h (right). In each plot, contributors are segmented by their degree of prior OSM experience. Median values are marked with a red line.

3. MM contributors tend to be OSM newcomers. How much experience does a typical first-time HOT contributor have? It turns out that this can vary wildly based on the initiative. The TH and ER groups have a mix of both OSM experts and OSM newcomers, whereas by far the most first-time MM contributors have virtually no prior OSM experience. Share of participants with a given amount of prior OSM experience, measured in the number of days on which they contributed to OSM.

4. These newbies are catching up quickly. Contributors to MM start slowly, however they catch up with others: many increase their pace of contributions in the first 48h. Compared to that, TH and ER contributors tend to maintain their initial pace. Share of participants based on their change in contribution pace between the first and second day.

5. Project purposes or modes of organisation likely have an impact on contributor retention. How many contributors to each of the initiatives are retained as HOT contributors? This is maybe the most important aspect if we care about growing an active volunteer community. For each first-time contributor we determine if they return on the second day, and whether they remain active contributors to any HOT project during the second and third month after their initial contribution. Comparing HOT initiatives in this manner uncovers some remarkable differences in retention.

Contributors to TH engaged in much short-term activity in the first few days, however in the longer term none of the contributors remained active! In comparison, about 8% ER contributors are retained as HOT contributors in the second month, and 1% in the third: they slowly fade away. In contrast to this MM has the lowest short-term retention, yet the highest long-term retention: contributors do not tend to come back on the second day, however they are more likely to remain active a month or two later. A remarkable accomplishment. Median retention for day 2, and months 2 and 3.

Implications

I would argue that the HOT community is highly engaged already. Most volunteers contribute for more than an hour within the first two days of their initial contribution, and a significant percentage of contributors is retained for longer periods.

The data suggests that the capacity-building strategies of ER and MM initiatives work particularly well: in these two initiatives, a good share of contributors kept coming back. No doubt this is because both were longer-term initiatives, so first-time contributors may have felt a responsibility to keep contributing. However I suspect there may be additional reasons. Maybe most importantly, monthly mapathons in a growing number of cities provide welcoming social spaces with expert guidance, peer learning, and all kinds of enjoyable experiences. In addition to that MM appears to foster a more well-connected community, with the means of notifying interested contributors of new causes via Facebook, Twitter, email alerts, …

I believe that given a choice, newcomers are best placed in projects where they have a higher likelihood of being retained. In our case this would be the ER and particularly MM initiatives: projects that are specifically set up as long-term initiatives. Additionally there are indications that particularly MM was successful at retaining and training absolute newcomers with no prior OSM experience.

Another key observation is that as HOT grows and starts new initiatives we’re gradually reaching outside the existing OSM community. Most first-time contributors now have no prior OSM experience, this was quite different in the beginning. This certainly affects how we should approach and support HOT newcomers.

---

By Martin Dittus (@dekstop) in 2015. This was produced as part of my academic research together with Licia Capra and Giovanni Quattrone. A paper which includes this work and more is now under review.