Recent diary entries
After my talk at State of the Map in Brussels, Nick Allen asked: are newcomers to HOT more likely to be retained if we give them positive validation feedback? And conversely, do we discourage them if we invalidate their work? I had no answer at the time, in part because many validation interactions are not public. However, I agreed with his observation that these are likely important early encounters, and that we should make an effort to understand them better. In particular, we should be able to provide basic guidance to validators, based on empirical observations of past outcomes. What are the elements of impactful feedback?
I spoke to Tyler Radford about these concerns that same day, and within a few days we signed an agreement which gives me permission to look at the data, provided I do not share any personal information. The full write-up of the resulting research is now going through peer review, and I will share it when that's done. In the meantime, I thought I should publish the most important findings.
Manually labelling 1,300 messages...
I spent the next months diving into the data, reviewing 1,300 validation messages that have been sent to first-time mappers. I labelled the content of each message using models from motivational psychology, and feedback in education settings. For now I'll skip a detailed discussion of the details, but feel free to ask questions in the comments.
I assessed the impact of different kinds of newcomer feedback:
- Positive performance feedback: messages including comments like "good job", "great work", "looks good", ...
- Negative performance feedback: "doesn’t look complete", "missing tags", "needs improvement", ...
- Corrective feedback: guidance about specific improvements to improve future work, including links to documentation.
- Verbal rewards: messages containing positive performance feedback, gratitude ("thanks!"), or encouragement ("keep mapping").
Here's a chart of the frequency of each type of feedback across the messages I labelled:
To measure the effect of these feedback types, I collected the contributions for each newcomer over a 45-day period after their initial edit, and labelled the content of the first feedback message they received during this time. I then observed for how many days they remained active, or whether they dropped out (as measured with an additional 45-day period of inactivity). I then used a Cox proportional hazards model to explain the retention rates we observed, based on a set of features and control variables. This is comparable to a regression analysis, but specifically intended to model participant "survival". In the context of this study, the term `hazard' is a synonym for the risk of abandoning HOT participation. A hazards model yields a hazard rate (or rate of risk) for each contributing factor, denoting the relative increase in hazard when a particular feature is present. For example, a hazard rate of 2.0 means that the person is twice as likely to stop contributing within the observation period, compared to the average. Conversely, a low hazard rate of 0.5 means they are twice as likely to still remain active at the end of the observation period.
Social affirmation matters: someone else cares
Maybe most importantly, I found that the feedback can be an important source of social affirmation, which in turn can improve newcomer retention. This effect is most clear among newcomers who contributed comparatively less on their first day (mapping less that the median of 75 minutes), possibly because they have low intrinsic motivation or self-efficacy. Among these, people who received verbal rewards in their first feedback message were significantly more likely to keep mapping, at a reduction of the hazard rate to 80%. In comparison, newcomers who already start with a high degree of engagement may not require such affective-supportive feedback to remain engaged.
This makes sense when you consider the wider context. The process of contributing to HOT online can be considered a depersonalised form of interaction: it is often focused on the task, rather than the learner. In the absence of other prominent social cues, small phrases of support may have a large effect. In the case of validation feedback, it's likely also important that this is not simply an automated message. Instead, someone else looked over your work and then took the effort to write some kind words.
To my surprise, negative performance feedback in itself is not necessarily discouraging to newcomers: while it may demotivate some individuals, in aggregate across all newcomers there was no significant effect on retention. This includes instances of invalidated tasks, and negative performance feedback such as "your buildings are all untagged". This may be because the feedback is private: people don't have to be concerned about the impact on their reputation, and can focus on improving their skills. In communities like Wikipedia where feedback tends to be public (in the form of comments or reversions), it was found that negative feedback can harm newcomer retention. It's also worth mentioning that even "negative" feedback in HOT still tends to be polite and constructive: HOT validators are generally a very polite bunch, based on the messages I've seen. They might simply point out that you forgot to square your buildings.
The timing of feedback matters: feedback that is sent a week after a contribution is significantly less likely to still have a motivational impact. In comparison, feedback that is sent within 28 hours or less (the median delay) yielded a reduction of the hazard rate to 80%. Any additional day of delay increased the hazard rate.
I now believe that this places validators at the core of the HOT community: for many contributors who can't attend a mapathon, and who haven't subscribed to the mailing list or joined IRC, validation feedback is their first experience of a social encounter. For a number of reasons, the current iteration of the tasking manager doesn't easily support such interactions (maybe a topic for a future post); but I'm looking forward to the next iteration, which is already in planning. As I've learned through discussions, the validator community already has some great ideas about improving it even further.
The fine print
First off, this is an observational study, which comes with some constraints: we can identify links between validation styles and outcomes, and control for confounding factors through careful model design, which gives us some confidence in the findings. However, we would have to run actual experiments to confirm each link.
The models behind these findings account for a number of confounding factors. For example, I consider each newcomer's initial contribution activity: were they already enthusiastic contributors to begin with? I also look at the particular project they start with: did they join during a disaster campaign, possibly in a wave of public interest? Such newcomers tend to not stick around for long.
And my usual caveat applies: I assessed the impact on contributor activity and retention, but not on contribution quality. In part because I still haven't found a good approach to assessing contribution quality at this scale: there is no ground truth available for comparisons, and contribution practices are diverse and often specific to the geographic/thematic context. Developing methods to assess data quality at this scale is a research project in its own right.
This is certainly not the final word on validation feedback, and I expect many others will add to this (maybe in the comments)? But it can hopefully serve as one contribution to our growing body of knowledge about how best to support our maturing community.
Thank you Ben Abelshausen for nominating me as a HOT voting member, and to Jorieke Vyncke and Harry Wood for additional support!
How did you become involved in HOT?
I have been aware of humanitarian mapping activities on OSM early on, but first really got to know HOT as an organisation through Kate Chapman's recorded talks. In 2013 I attended State of the Map in Birmingham where I met Ben and Jorieke, and learned about the growing range of development and aid activities that had grown out of the wider OSM network. In Summer 2014, a group of people started the first regular HOT mapathons in London (they would later co-found Missing Maps). I became an early participant, and my involvement grew from there.
Could you tell us about your involvement in HOT, mapping and/or humanitarian response?
I initially became active in HOT as a PhD student researching community organisations, and after some months of exploration decided to make HOT the centre of my work. Over the last 2-3 years I've gradually expanded my involvement. At some point during this time I also joined a growing volunteer team around Ivan Gayton, Pete Masters and Andrew Braye to help run their mapathons and other HOT-related activities.
My first tangible contribution is maybe the talk I gave at the first HOT Summit in 2015 (slides). I showed empirical evidence of some HOT community activities and outcomes, and discussed the implications. The talk resonated well, and sparked great debate during the session. Based on the feedback I got I think this helped people gain a different understanding of their work, and their priorities. (Unfortunately the video was never published, maybe we can get it online sometime.)
After the talk, Alyssa Wright approached me and suggested I should make my findings accessible to the wider community. This motivated me to start a research diary, where I now share findings from my various explorations of HOT activities. The first post discusses my motivation: to help develop a broader understanding of HOT through analytics and visualisations, contextualising the data, providing evidence to substantiate design choices, and offering conceptual models which help reason about HOT as a social phenomenon.
My research has progressed a lot since these early days, but most of the time it is still driven by a desire to use my research skills to support HOT as an organisation, and to inform and strengthen HOT practice. In addition, I've also been regularly approached by other community members with ideas about aspects to look at; have a look at some of my past diary posts for examples of this.
What does HOT mean to you?
My guest blog post for State of the Map 2013 ends with an observation that still motivates me today: HOT to me reflects a turning point in community technologies. It takes OSM as a starting point, but expands on it by connecting to a large universe of social concerns. In my opinion, a key contribution that HOT is making to the world is that it places community at the centre of its activities, and that it embraces and balances a multiplicity of perspectives. But also that it finds a delicate balance between a kind of volunteerism that is driven by enjoyment and personal enthusiasm, and an honest professionalism that connects to funding sources and places where "serious people" live. In that, HOT represents a rare synthesis of the lessons of open source culture and the aid and volunteering sector, hopefully managing to keep the best parts of each.
Why do you want to be a voting member?
I have experienced HOT from the "outside" for a few years now, and have become more and more personally invested in its future. I would like to formalise this relationship, and help take on the burden of making sure that it remains a healthy organisation for a long time to come.
As a voting member of HOT what do you see as your most important responsibility?
I think one of the most important contributions any member can make is their approach to internal discourse. I see it as my responsibility to promote things that I think are important, to alert the community of risks, but most importantly to do so in a manner that is constructive, never divisive, and to help moderate internal debates when emotion takes over.
How do you plan to be involved in HOT as a voting member?
I will keep up my enthusiasm for finding new HOT corners to explore, helping foster community engagement, seeking to help tackle community coordination challenges, and supporting daily practice in a range of ways. In addition, I look forward to participate in the governance of HOT. I have spent the last decade with a wide range of community organisations, and have had much exposure to the governance challenges they may bring, and some potential means of addressing them. I plan to bring this experience into my involvement with HOT, but also to come with an open mind, and to take time to listen.
What do you see as HOT's greatest challenge and how do you plan to help HOT meet that challenge?
HOT is attempting to foster a new kind of practice while the word is shifting around us. As a consequence, there is a long list of challenges. On top of that there are the challenges of a maturing organisation: managing funds, emergent factions, maintaining the tech. Others will have thought about these aspects quite deeply already. A personal concern for me is HOT's relationship to community growth, and community cohesion. Internally, and in its relationship to other organisations, and the wider OSM ecosystem. How large do we want to grow this? Do we have the means to deal with the consequences? I think there are many open questions related to this; but also a growing body of knowledge that we can draw from.
My talk for State of the Map 2016: Building large-scale crowdsourcing communities with the Humanitarian OpenStreetMap TeamPosted by dekstop on 21 November 2016 in English (English)
I just saw that the video for my SotM16 talk has already been online for a month... many thanks to the organisers and video team in Brussels for making this happen so quickly, and in such a high quality! You can find some summary notes further below, along with recommendations to HOT organisers.
(This was recorded at the tail end of an unusually busy summer, after a couple of weeks of deadlines on little sleep, in a morning slot, with little time for rehearsal... throughout these short 30 mins I really, really wanted to go back to bed. If you know me a little you might notice it in the recording, everyone else may simply think I'm a little slow :)
Among the key observations to date
The talk summarises much of my research to date. It includes updated statistics and visualisations, and the results of three studies of HOT community engagement.
Over the course of this work, I've stopped thinking about community engagement as a process of "converting" people. Instead I now also think of it as a process of discovering and activating the right people: many of our most prolific contributors were already prepared to be engaged. Maybe they were looking for community, for a spare-time activity that has a bigger impact than just watching TV; maybe they already had some GIS experience and didn't know they can use it for a social purpose. In this sense, fostering community engagement is as much about the initial recruiting process as it is about the actual contribution process.
Among the key observations to date:
- HOT is now a key source of community growth for OSM: among the 32,000 HOT contributors to date, 80% are newcomers to OpenStreetMap! (I have not yet investigated whether they then also contribute to other parts of the map.)
- Over their contributor lifetime, 50% of HOT mappers dedicate at least 65 minutes to their contributions. This may sound like a small average for a volunteering organisation, but for an online platform it's a massive achievement.
- Emergency response events can also be key recruiting moments: during HOT activations for Typhoon Haiyan, the Nepal earthquake in 2015, the earthquake in Ecuador in 2016, and others, many new volunteers joined HOT.
- ... and much, much more.
Recommendations to organisers
In the talk I also make some recommendations to HOT organisers, based on study findings, and informed by my interactions with the wider community:
- During large disaster events, carefully manage the tasking manager task listing. People who join during these events don't tend to stay active for long, and their contributions tend to have a lower quality. Point them towards newcomer-friendly projects where they can make some early experiences.
- At the same time, HOT can likely benefit greatly from a notification mechanism for contributors who are interested in future campaigns. Currently there is no good means of reactivating mappers who have already made some early experiences. Instead we rely on our volunteers to discover new campaigns on the mailing list or on social media. While this may work for the core community, there is likely a larger number of mappers who may be willing to help out again. How can we best inform them when they're needed?
- Generally, try to connect newcomers to the existing community as soon as possible, and do so in a setting that is appropriate for absolute beginners. The mailing list works well for a few hundred core contributors. Yet as we grow, is it still the best default location for a newcomer who has a question for an expert?
The visualisation below shows the regions of the world where the HOT community has contributed edits to OSM, which is one way in which we can show the impact of our community. The chart visualises contributions before 23rd Sept 2016. By this date, 32,000 people had contributed at least one edit, accounting for a total of 182,000,000 edits. This took an estimated 240,000 labour hours.
As mentioned before, I've been showing the visualisation in talks for a while now, and I regularly receive messages by people who would like to use it for their own slides, for mapathons and training sessions, and other uses.
There is also a PDF version (11MB), a high-resolution PNG (1.3MB), and a folder with older versions if you want to do a visual comparison of map growth. Send me an email if you would prefer a version without annotations -- I simply ask that you provide credit when you're using it.
(Despite my best efforts I've not yet managed to make to switch to the Robinson projection, as recommended by BushmanK... the QGIS renderer acts up every time I try changing the projection string. I'm probably simply doing something wrong.)
A few months ago I posted a draft specification for an OSM quality assurance tool. The first beta for the project was launched last week, it is now called OSM Analytics. Cristiano Giovando posted an announcement on the HOT blog.
I also gave a brief introduction to the tool and its uses at the most recent Missing Maps mapathon in London, there's a recording by the BRC maps team on YouTube. Unfortunately we had wifi problems at the venue, so it's not a very fluid presentation, but Chris Glithero took care to edit out the gaps so it's still a decent flow.
Today I took some time to update my list of HOT mapping initiatives -- a bit of a messy process because there's no official listing. These days I simply review new projects in the OSM edit history that have a minimum number of contributors, and label them with a simple term. The intention is to identify groups of projects that have a common theme. Typically these are disaster events, larger mapping campaigns like Missing Maps, or organisations that organise projects for their members. Of course the boundaries between them are blurry, e.g. Missing Maps is really a meta-initiative across many discrete projects.
Here's a timeline of the initiatives I've identified so far -- let me know if I missed any! There's also a PDF version, in case you want to include this in presentation slides.
Andrew Braye, Jo Wilkin and I spoke at the Oxford Internet Institute earlier this month as part of their ICT4D seminar series. Andrew gave a high-level overview of HOT and Missing Maps, Jo spoke about data collection in the field, and I spoke about my HOT community research. We had a great time! The video is now on YouTube and is about 1h long.
I particularly enjoyed Jo's part which starts 7:30 minutes into the video, she gives some background on what happens after HOT remote mappers have produced a basemap. She shows specific examples in Katanga (DRC), Lubumbashi (Congo), Dhaka (Bangladesh), and other places where HOT coordinated field mapping activities with local communities, either using field papers or OpenMapKit on smartphones, covering a wide range of purposes. In Sierra Leone, local motorcyclists collected names and population counts for several hundred villages, which became an important information resource to help curb the Ebola epidemic. According to Jo, since Missing Maps launched in 2014 they have coordinated one field trip a month, if not more... pretty impressive.
I spoke just after. Some of the things I covered have already been posted here, and other aspects will become part of future posts. For now I just want to highlight two charts:
HOT contributor activity spikes in relation to large humanitarian events.
Cumulative number of HOT user accounts. Large events are often also recruiting opportunities, they draw their own crowds. We just have to make sure that we're prepared and can give people something to do.
In total this covers around 120 million changes to the map, by almost 20,000 contributors across 1,000 projects. This required an estimated 165,000 hours of volunteer work! There's a monthly breakdown of this activity in this Google spreadsheet: "2016-01 HOT activity timeline".
I'm keen to do an animated version at some point! Also, could a cartography geek please recommend a suitable projection for this map? Atm it's just the default WGS84, with apologies :)
Blake sent an email to the HOT Community WG asking for ideas on how to increase the number of regular HOT mappers. This is squarely in my research domain, so it was a fun question to respond to... I suggested things that now to me are pretty obvious, but weren't just a year ago.
My suggestions follow, in no particular order.
Identify existing communities with a propensity for this kind of work: GIS experts, aid org volunteers, and others who are similarly embedded in existing contributor communities.
Partner with more large corporates, but choose the right ones: where there are already some HOT mappers on staff, and people who can coordinate company mapathons. Don't go through exec, instead identify existing mappers who care. (Cf Arup, others)
Set up regular online events where people can come together in a more social fashion. Online chats, twitch streams, etc; play with the format.
More regular mapathons around the world, organised by new groups; learn from Missing Maps in London, they're now world experts in how to do it well.
Better communication of ongoing needs: e.g. a weekly (or monthly) email which provides background info on current projects, incl mapping tips about specific pitfalls.
A well-managed validator process, similar to Missing Maps in London: try to ensure that new contributors receive good and constructive feedback early.
Better guidance on the TM homepage: instead of "pick from infinite list of words", try to emphasise different aspects that may resonate with particular types of mappers. The easy ones: degree of urgency, type of purpose, participating organisations, "almost done" projects, projects in specific countries, ... I'm sure there are loads more aspects. (Then measure which of these things people actually respond to.)
Find means of identifying people who are actually interested (or likely to be interested), and then give them more specific support. For example, make sure they're connected to a mentor or a peer group.
Based on existing experiences, figure out what kinds of social mapping settings are quick to set up and easy to replicate in different places, and by different people. Then write up some simple design patterns for how to set up your own mapping group. How to pick a good organiser, who should you invite, what's a good venue, what tech is needed, what support will first-time mappers require, where do you go with more specific questions, etc.
Increase social presence: give people a social identity beyond their username, then get them to chat, share experiences, etc.
Joost asks in a direct message:
I'm organizing a Missing Maps event in Antwerp. One of the co-organizers wants to try giving a tweaked JOSM version on a USB stick to all the participants (preloaded settings etc) and use JOSM as a default editor. [...] Did anyone try this at an event? Did you have a look at first timers using JOSM having a higher or lower OSM/MM retention? (It might be too much self-selection to really prove anything...)
I thought this was an interesting angle, and it connects with some of the work I’m currently doing, so I had a look at the data and am posting the results here. The short answer, based on a small sample: we've actually seen a difference in retention! However not in the way you might expect. I was surprised.
Before I begin I should say that I'm very interested in other perspectives on this question, particularly actual teaching experiences. This is a good scenario where statistics might be misleading, and where it helps to have actually talked to the mappers and observed what happened. Looking forward to people's comments!
It’s actually really hard to measure this well and generalise from past experiences, because every mapathon has its own story; different people attending, different things going right or wrong, etc. Different editors are also often used for different kinds of work: JOSM often gets used for field paper tracing and validation as well as satellite tracing. Unfortunately I haven't been to most of the JOSM training sessions I'll quantify below, so I don't know what people actually did!
Furthermore, editor choice affects all kinds of follow-up considerations that may affect the outcomes of such a study; e.g. I’ve seen people forget how to launch JOSM a month after they first installed it, or OS updates causing java versioning issues, all of which is not something that can happen with iD.
And so on. You get the idea: many factors to keep in mind when we look at these numbers.
We can still look at general trends across the JOSM newcomers so far. Unfortunately there’s not a lot of observational data to make any strong statements, however I do think we can see some trends. And I’d certainly say that there is plenty of scope for further experiments!
Our observations so far...
The following statistics compare two groups of attendees at our monthly Missing Maps event in London: people who started with iD at their first mapathon, and people who started with JOSM. To make the comparison somewhat fair I'm only looking at attendees who have little prior OSM experience, with no more than 5 days of prior OSM contributions before their first mapathon attendance. I've also excluded the small number of people who used both editors at their first mapathon.
At our monthly mapathons, 37 people started with JOSM right away, spread across 12 events. On the other hand 298 first-time mappers started with iD (13 events).
Activity at the first event
16% of the JOSM mappers contributed for more than 2h in the initial mapathon edit session; this is about half as much as the people starting with iD, where 33% contribute for more than 2 hours. A histogram of their session durations illustrates the difference:
You may notice that the two distributions are quite different. JOSM contributors tend to have shorter contribution sessions. I verified that this is a general pattern across multiple events, and not biased by a single mapathon. Note however that this does not necessarily mean that JOSM trainees tend to lose patience more quickly -- they may simply be doing different kinds of work.
Update: As Joost suggests in the comments, it might also simply mean that JOSM collects edit timestamps differently. In past explorations I've seen JOSM preserve timestamps for individual edits within a changeset, but I don't know enough about the editor to understand what exactly is going on.
Joost however was asking about the impact on retention, so let's see what happens in the days and weeks after the first attendance. For that we will observe everyone's subsequent contributions to HOT, at home or at a mapathon, up to a period of 90 days after their first mapathon attendance.
A month later the picture flips. 32% of JOSM newcomers were still active 30 days after they first came to a mapathon. On the other hand, only 20% of iD users were still mapping.
To assess these numbers further we can look at survival plots, these show how likely it is that a certain group is still active after some time has passed. Most importantly they tell us whether these trends are statistically significant.
The wide confidence interval for the JOSM group (the shaded region around the curves) illustrates how little data there is. The JOSM group has larger confidence intervals, which means there is a variety of retention profiles in this group, and not enough samples to determine a clear trend. As a result the confidence intervals of the two curves overlap, which means there’s likely not enough data to say for certain that the groups differ significantly.
However the curves do suggest an apparent trend: at Missing Maps monthly events, people who start with JOSM tend to remain actively engaged for longer.
Unexpectedly for me we do get some clear differences in outcome when looking at Missing Maps monthly events in London! Namely:
- I looks like newcomers learning JOSM were more likely to stop early in their first session, compared to iD trainees. (Alternatively, JOSM and iD differ in how they collect edit timestamps.)
- On the other hand, a larger share of JOSM trainees were retained as mappers over the following weeks.
Although I was surprised by this, this is not actually entirely unexpected. JOSM use tends to be associated with higher engagement: the most active mappers are often JOSM users.
However this does not necessarily mean that JOSM is the key trigger. It might simply reflect that the JOSM mappers at our events are a great bunch of people, fun to hang out with, and many of them know each other quite well; whereas the people at our iD tables are typically newcomers who are not yet as well-connected to the community. So maybe the difference is in the people, not the editor.
In closing I would say that we need many more observations across different kinds of settings to make these statistics meaningful. At the moment this is little more than anecdotal evidence. There's definitely space for further experiments!
I had a recent shift in perspective in my research of HOT contributor engagement. I will try to articulate a growing intuition: a sense that current-generation HOT tools and processes would do well to also recognise the secondary benefits HOT volunteers get from their participation, for example their social experiences. I think we currently don't necessarily create social online spaces for new contributors, and that is an omission of some consequence. In contrast to Wikipedia and comparable platforms, HOT contributors are not also typically the primary beneficiaries of the collective output. Secondary benefits can make up for this lack in direct utility: they have important motivational power.
As usual, please let me know your thoughts on this. It's informed by my own experiences of the HOT and Missing Maps community, and I am very curious to learn what I might have overlooked, how else to express it, or find other ways to look at things.
What factors influence sustained engagement?
I'm researching contributor engagement in humanitarian mapping, trying to understand the factors that affect sustained engagement. Over the course of the past year I'd been looking at contribution mechanics and project designs (the microfoundations), mapathons as social contribution settings (group experiences), and am starting to look at the contributor flows between larger initiatives over time (collective experiences.)
I've looked at it from many different perspectives. Do certain task designs put people off? Does it make a difference when there's a food break where people can socialise, when the wifi dies as you try to save which might cause frustration, or when a charismatic field worker speaks who can instil the practice with meaning and purpose? Does it make a difference that you're sitting next to experts who can help you get started and build confidence quickly?
In every one of those instances I found (maybe unsurprisingly) that these factors may have some effects on short- and long-term engagement, however they are never consistently a trigger that converts people; alone or in combination. They likely contribute, but they don't create engaged mappers in themselves. And, crucially, many of these things aren't strong barriers to community growth: many people have already figured out how to map, with or without help.
In preparation for my annual research report I went back to some of the fundamental literature in my field, papers outlining the state of crowdsourcing knowledge. Contributor motivations in crowdsourcing are fairly well-understood, there are enough empirical studies which find recurring categories of motivation; in literature on volunteering and charitable giving, citizen science, Wikipedia, and even OpenStreetMap.
Secondary benefits of HOT participation
As you may suspect, people have a wealth of reasons to participate in volunteering projects like HOT. Some classic motivational categories relate to shared values, the social experience, gaining understanding, career development, self-improvement, and enjoyment of the process. One aspect in particular seemed appealing to me to ponder: the concept of social identity. The notion that when contributors weigh cost and benefits of their participation, an important consideration is what the practice means for them as an individual. Does it relate to their personal or professional interests? To an aspect of their biography, a past experience? To their relationships with the world? Their image of themselves? Does the practice allow them to form, articulate, and perform an identity? You might call this the secondary benefits of participation.
This may be an obvious realisation, but having it framed for me in this manner did rearrange my brain a little bit, and it changed my thinking. I remembered many conversations I'd had with mappers and organisers, and under this lense a theme emerged from all these chats; I can now see that many contributors have quite a clear understanding of the secondary benefits they derive from participation.
This is why there are so many geographers and GIS people among our volunteers. Why it's not surprising to meet mappers who have been to Nepal or the Congo. Why people love socialising at mapathons, hearing the stories, forming relationships with organisers; why it's so important to encourage beginners with constructive feedback, and to give experienced mappers opportunities to dive deeper, or to teach others, or to take on responsibilities.
What do people get out of the act of mapping itself, the individual clicks? Some people may be able to ascribe it with a concrete purpose: "a year ago I walked past this very house, hopefully my map can help make sure that people are cared for". Others might say they find the activity meditative, soothing. However I would now posit that for many, the act itself is only attached to fairly abstract motivations. In contrast to Wikipedia, HOT maps don't actually have utility to its contributors; they benefit aid workers, and people on the ground. The more concrete fulfilment for contributors comes out of all the things around the activity.
This is particularly clear at a mapathon, where there's always so much happening; in London we're now world experts at how to run a great HOT mapathon. Many blog posts and tweets can illustrate this, as do the photo albums of the Missing Maps Facebook account.
However after people go home, the community is on hold until next time. Many of our mapathon attendees don't tend to map at home.
The social identities of HOT online contributors
What is the equivalent of these social experiences and other secondary benefits when you're mapping at home? For example, how can the act help you form, articulate, experience, perform or promote social identities? I think for that we still have few answers; I think we still understand very little about what makes remote participation work. And crucially I think we don't quite offer the means for social identity experiences online: our platforms are focused on the work itself. I would argue that the contributor collective is not actually well-connected at all, except for a few highly-engaged people who are subscribed to the mailing lists or chatting on IRC. However many of the thousands who participated over the last year actually have no place to go to socialise, or to discuss their experiences.
From that perspective I'm now not surprised that contributors don't stick around after a high-profile disaster response (where there's urgency and a direct purpose), and that many repeat attendees of mapathons don't tend to map at home.
However I'm now also buzzing with ideas for things we can offer to fill these gaps; countless opportunities to improve our newcomer support, to introduce social online spaces, to form and perform social identities, to give people easy means to tell their own stories about what they just accomplished. New ways of telling people where help is needed, how they can improve their skills, and ways of making it a shared experience. Because at core this is what a community is: not a bunch of people who do a bunch of work, but a collective with shared as well as divergent identities, with values and reasons, with stories. And every new contributor who starts mapping because they saw us in the news should be able to participate in that.
Inspired by recent Transifex discussions I though it’d be interesting to see what languages our contributors actually speak — to the extent that we can easily find out. It turns out that as of May 2015, iD now submits a “locale" changeset tag — JOSM has been sending that information for a while already.
The top entries across both editors are shown below, for May-October 2015 (inclusive). Note that a locale with a small number of contributors is not a locale that matters less -- as we've established before, a small number of contributors can make a significant impact on the maps of a region.
There's also a Google spreadsheet with separate tabs for iD and JOSM contributors, if you want to dive further into the data: "HOT contributor locales, May-Oct 2015". Or as CSV files: combined, iD, JOSM. It's interesting to compare their distributions. E.g. iD has a much longer tail, which I guess is not a surprise -- browser locale vs limited JOSM translations?
Harold D. Craft's classic visualisation technique applied to a timeline of HOT project activity. As previewed before and used in the Missing Maps review, but updated for early November 2015. Click through for the full version.
A line per tasking manager project, its height along the implied z-axis is proportional to the number of project contributors on the respective date. Projects tend to be most active in the beginning, and then activity tails off. However some large projects are eternally active... MapLesotho (#597/599) is among these, partially covering the equally long-running South Sudan (#591). Remarkable how massive the Nepal contributor community actually was, in the scheme of things -- the big spike in the centre would be even taller if the work hadn't been spread across multiple projects (between #994 and #1090).
There's also a PDF version if you want to print it out.
If it looks fuzzy on your screen then make sure you're looking at the image in its native resolution: open it in a new tab and zoom to 100%. At an information density of several hundred high-contrast lines within just a few inches of digital display space it's hard to avoid moiré effects. The preview you're seeing above is optimised for smaller display sizes... It's still not great.
On a skype today, Kate Chapman said that analysis after the earthquake in Haiti, she found that '40 people did 90% of the work' within the community.
Is the workload more evenly spread throughout the community when it comes to Missing Maps tasks as opposed to HOT tasks? Is it more evenly spread during non-emergencies?
I thought I can look at this quickly because I'd done similar work around participation inequality in the context of OSM; in the end took much longer than expected and I can't see that I found a simple answer. If anything it serves as a good reminder why it's challenging to produce meaningful statistics for social spaces: the devil is in the many nuances. This writeup here can probably give you some impression of that.
Unfortunately I don't have contributor statistics for Haiti since it predates the tasking manager, instead I will compare Missing Maps with other large HOT initiatives, most importantly Typhon Haiyan in the Philippines in 2013, but also the Ebola activation in 2014, and Nepal in 2015.
The impatient can skip the more in-depth discussion and jump to the conclusion section at the bottom. Note that this is just a quick exploration, not a thorough statistical analysis. I'm sure I've overlooked things, so please give feedback.
As usual I'm looking at labour hours as a measure of work. The results are probably not that different than if I'd used map edits, however I find they're a better reflection of the effort spent on contributing. Time moves at the same pace for everyone, while the same number of clicks could yield a different number of edits depending on what you're doing. Edit counts are also a potentially confusing measure because there's no standard way of counting them: as the number of version increases of geometries, or the number of changesets? Etc. So here's a key limitation of these stats: I'm not actually looking at map impact, instead I'm looking at a measure of individual effort.
You're of course welcome to do your own analyses and compare, the raw data is linked below. Including edit counts!
Group sizes and average labour hours
Just to get a first impression: how much work do people do in each group, on average?
I apologise for the messy table, this is quite unreadable, but useful for reference later. According to my contributor database (which atm has data up to early August 2015) there were about 6,400 contributors to Nepal, 2,800 to Missing Maps, 650 to Haiyan, etc. In other words, Missing Maps has more than 4 times the number of contributors than Haiyan.
Here the median labour hours per group as a plot:
The median contributor effort looks comparable between Missing Maps and Haiyan. However bear in mind that we're looking at a data set that is long-tail distributed, as this histogram suggests:
When looking at long-tail distributions we have to pick our aggregate measures carefully: the mean and even median are likely heavily skewed by outliers. There is no general measure of central tendency for long-tail distributions, nor can there be one. Repeat after me: "There is no average user".
Instead we should compute measures of distribution: how is work distributed among the group?
The Gini index as a basic inequality measure
The Gini index is a classic measure in economics used to describe inequality in groups, usually income inequality in societies. It's typically a number between 0 and 100 (sometimes 0 and 1), and a higher number means "more unequal". According to the CIA fact book, the US has a Gini index of around 45 while the UK's is around 33, and Germany is at 27.
The Gini index is also sometimes used to describe participation inequalities in online communities such as HOT. Online communities tend to be highly unequal, with a small share of highly active users; we will come back to that in a bit. It's important to know that we can't compare Gini scores across different kinds of social systems, e.g. we couldn't fairly compare Wikipedia scores with HOT scores unless we're sure they've both been measured in the same way. We can however simply use it to compare different subgroups within a community. Here: different HOT initiatives.
Observations: Missing Maps & Haiyan seem fairly similar. Haiyan might even be a little bit more fairly distributed, however these kinds of "social" statistics tend to be messy in all kinds of ways, with a high degree of measurement error, so for practical purposes I would consider them equal. Work for Nepal and particularly Ebola on the other hand is more unequally distributed -- with either a smaller number of hardcore contributors, or a larger number of people who do very little.
Distribution of work: the bottom end
Ok so let's look at the actual distribution of contributions -- as a first step, let's see how many people do a minimum amount of hours in each group.
This plot shows the distribution of work in absolute terms: how many contributors work for x hours? For example we can see that in the Nepal and Ebola groups, a large number of people contribute very little: many already stop within the first 30 minutes. Missing Maps on the other hand has a nice bump: many people contribute for up to 2h. Is this the mapathon bump?
The Haiyan group is too small to be easily discernible in this plot, so let's look at relative numbers..
... it's somewhere in between. Not an extreme spike of early leavers, but also no mapathon bump. Otoh it likely has a longer tail: a larger number of highly prolific contributors who each do loads of work.
Based on these charts you could say Missing Maps manages to raise the lower threshold of participation above the bare minimum, which is an achievement in itself. However this doesn't yet answer Kate's question: how much work do the top 40 contributors do?
Impact of highly prolific contributors
As we've seen in the summary table above, the initiatives have very different sizes. Based on that alone we can expect that the top 40 contributors in Missing Maps are likely to have had a smaller impact on the overall output, because it's a larger overall group. Let's check:
Hah, unexpected: The top 40 contributors in Missing Maps and Haiyan had about the same impact on their groups, both carry around 50% of the total effort! Intriguing. If I may speculate about a cause: Missing Maps is a larger project, but also has been running for much longer, so while the overall output is larger, the top contributors also have more time to do their share. (There's only so much time in the day a person has available to do mapping.)
[NOTE: Kate said 90% for Haiti, I got 50% for Haiyan/MM for the same number of people. Why? Don't know -- don't have data on Haiti, and would also need to compare how each statistic was computed. Stats are hard.]
Let's look at it in relative numbers instead -- the impact of top 6.25% contributors (the equivalent of 40 in 639 contributors for Haiyan, according to my records).
As expected: we're now covering a larger absolute number of Missing Maps contributors, and of course they collectively account for a larger share of the work at almost 70%. I.e., the "core" contributor group in Missing Maps is larger and does more work than in Haiyan, but only because Missing Maps involves many more people.
Let's also look at the impact of the top 20% contributors, just because that's a classic number people tend to use.
Here we see a classic 80-20 distribution: 20% of users are responsible for 80% of the work. Aka the Pareto principle. Widely observed among online communities. Interesting that it seems to approximately apply for every single one of the HOT initiatives shown here.
As in all things relating to people it's complex, there are different ways of looking at the question, and likely many contributing effects: how were the initiatives promoted, were they executed by a core community or lots of one-off contributors, to what extent did they attract hardcore OSM experts, how long did activity last, etc. I think we barely scratched the surface here. It also serves as a good reminder of why we should be sceptical of simple analytics when looking at online communities.
Based on the charts here we could say that:
- Work is always "unfairly" distributed in HOT -- that's also a well-known empirical finding in many other social settings.
- Missing Maps and Haiyan (and other initiatives) are comparable in terms of participation inequality in some respects, however there are also differences.
- E.g. most have a similar 80-20 split, where 20% of highly active contributors do about 80% of the work -- typical for online communities. This appears to be independent of group size, length of activity period, and other factors.
- It gets even more extreme at the top. A very small number of the most active contributors might be responsible for a surprisingly large share of the work -- e.g. we found for both Haiyan and Missing Maps that 40 people are responsible for about half the work.
- Ebola and Nepal have a higher Gini index compared to Missing Maps or Haiyan, which means work is more unequally distributed in these groups. We find that in these two groups, a larger share of contributors drop out within the first 30 minutes -- more people do less.
- Missing Maps on the other hand appears to have raised the bar in terms of minimum participation. Compared to the other groups we looked at, people don't tend to drop out right away, and instead many stay active for 2h or more. This might be a result of the regular mapathons organised by Missing Maps teams around the world, or of the fact that it's a long-running effort so people contribute more over time.
My inner academic would further argue that in order to gain confidence in these claims we'd have to do actual statistical analyses, and not just look at charts and summary statistics. For long-tailed distributions we might use statistical tests of independence, such as the Mann–Whitney U test or the Wilcoxon signed-rank test, to determine whether these distributions of labour are actually statistically different across the different groups. That's for another time -- or maybe someone else wants to take it on? The data is linked below.
Do any differences relate to a sense of urgency? Not sure we have looked at enough evidence to answer this; of the four groups we could say Haiyan & Nepal are "urgent", Missing Maps is "not urgent", and Ebola may be somewhere in between. I believe the data we've looked at so far won't easily accommodate simple interpretations. A study for another day, or another person :)
Other suggestions for analyses not shown here?
The data used for these analyses:
- hot_contributors_20150810.csv (682kB), a list of all contributors per initiative, along with their total number of labour hours and edits.
I had no time to prepare for this, so on the plane from London I simply went through my work of the last few months and collected things that seemed appropriate for the occasion. I intended this as a quick 30-minute review, but it ended up stimulating lots of debate throughout... so the session took 2h instead. For a conference setting this would have been disastrous, but since this was a team gathering it was actually quite useful. Data visualisation as conversational catalyst!
See also Dale's recap of the powwow.
Thank you both! It's good to know that the work resonates. It's surprisingly challenging to design research projects that are academically strong and also of relevance to the outside world... As my supervisor Licia Capra likes to joke, academics like new methods while practitioners like old methods applied to new systems.
A few weeks ago pedrito1414 asked me to determine the share of HOT contributions that are attributable to Missing Maps. It took me a while to get around to it... but I finally did. If you follow me on Twitter you may already have seen a couple of these, but here's the full set.
(Interesting to see the post-Nepal uptake in MM activity. I didn't actually check where this activity is going, but I expect the main driver are the mapping efforts for South Kivu, a new Missing Maps initiative launched in June with a very ambitious geographic scale.)
(Note that averages are misleading, it's unlikely that many MM volunteers actually contribute that much. These contributor stats are typically long-tail distributed, with a small subset of highly prolific users that raise the overall average, and a large number of people who contribute little. In fact a good mantra for any community research is "there is no average user", partially because of the prevalence of long-tail distributions. Investigating the actual distribution of MM contributions is a task for another day...)
I only recently realised that HOT contributors need to mark at least one task as "done" to be listed as project contributor in the tasking manager. This made me wonder: how many people start contributing to a HOT project but never finish their first task? What proportion of all HOT edits are contributed in this manner?
Summary: about half of all HOT contributors never complete their first task on a project, although they do contribute to the map. These "partial" contributions account for 10-20% of all HOT edits.
Here's a timeline of the number of monthly HOT contributors, compared with the number of those who completed at least one task:
And here the corresponding timeline of the number of edits contributed by both groups of people:
Expressed as percentages:
We don’t know why these contributors never completed the task, we can speculate but really we would need to ask them. Some may have forgotten to close it after they were done, some may not have had the confidence to mark it as "complete" and wanted someone else to have a second look, some may have gotten distracted, or lost motivation, etc.
It's also worth bearing in mind that we can always expect some proportion of tasks to be abandoned early: not everyone is interested in contributing to HOT in the long term. Many people are likely simply curious and try it out for a bit. Many may have come across HOT because a friend sent them a link, or because it was in the news, and we can't expect all of them to stick around.
However we should also be mindful of these early experiences. On one hand we can improve our understanding of what makes people stop early. On the other hand we should also consider the impact these contributions have on our map, and on validation and QA efforts. Where should we send absolute newcomers the next time we're in the news?
Some background info on the analysis...
I’m identifying HOT contributions in the OSM edit history as follows:
- The contribution needs to fall within the geographic boundaries of a HOT project
- The contribution needs to happen within the activity period of the HOT project
- And then...
- EITHER the user is a listed project contributor (they marked at least one task as done),
- OR the changeset is tagged with a valid HOT project ID (the contributor never marked a task as done, but likely did start a task in the tasking manager before contributing edits.)
There are some caveats with this data:
- In this analysis, one completed task by a contributor is enough to regard all their contributions to the same project to be marked as "done". The simple heuristics above do not allow me to distinguish task completion states for all individual changesets of a contributor to a project.
- We can't distinguish contributors who never mark a task as "done" from validators, or expert contributors who manually tag changesets with a project ID. We don’t have the data to distinguish these cases, e.g. there is no published list of validators to compare against.
- We can only reliably track this from Aug 2014 when iD started carrying over project-specific changeset tags from the tasking manager. We won't be able to identify "unsubmitted" contributions before then.
If you're subscribed to the HOT mailing list you've seen a recent invitation to help develop a funding application for the Knight Prototype Fund, coordinated by Russ and Blake. The intention was to discuss project proposals that may be suitable for this grant. The initial IRC meeting then developed into a larger conversation around current HOT needs for better tools: the resulting Google Doc with meeting notes lists six project ideas.
The strongest candidate was a proposal to develop a HOT/OSM tool to support Quality Assurance (QA). You can read some details in the grant proposal writeup, however it's a fairly high-level text. Informed by our discussion I also developed a draft specification, with a more detailed list of considerations and potential features.
I'm posting this draft specification here to get your feedback, and to hopefully stimulate some debate about what a good QA support tool might look like. The proposal is a result of conversations with HOT practitioners, and based on my own use of HOT and OSM data. However there are likely many community members with further ideas, and some may even have worked on HOT QA initiatives. We would love to hear from you! In particular we would love to hear from validators, and from existing users of HOT data. What specific data quality concerns arise in practice?
(I should also state that I don't have a deep understanding of the Humanitarian Data Model -- there are likely some useful concepts in there that could be more emphasised in the spec.)
Our general ambition is to make HOT progress more visible. More specifically, the proposal aims to support our existing QA processes around HOT validation. Crucially it further aspires to provide a means of demonstrating HOT data quality to prospective users of the maps.
Aims of the proposed QA support tool:
- Impact analysis of HOT coordination efforts: to describe our outputs in ways that are meaningful to the HOT community, to prospective data users, and to a wider public.
- Evaluating fitness for specific purposes: to assess the quality of the data in relation to the specific concerns of data users.
- Integration support: to assess the structure of the data in relation to the Humanitarian Data Model (HDM).
The design of the QA support tool should be informed by the needs of existing users of HOT data: most importantly HOT activation partners, and requesting organisations with specific information needs. This also includes prospective users in aid organisations who still need to be convinced that the data can be useful.
It should also be informed by the needs and experiences of HOT validators: they are most well-informed about HOT data quality concerns, and they are likely going to be the most active users. The QA support tool should integrate well with HOT validator workflows, however it is not meant as a replacement for existing tools. I imagine its most useful function will be as a final check: a summary report of the outcomes of a particular mapping initiative.
The design could further consider the needs of other potential users of HOT data: people who want to report on current issues, or who as part of their work can make use of geospatial data. This includes local communities, local and international journalists, engaged citizens, and supporters of aid organisations.
What are their needs?
(This is a bit speculative. Please share your thoughts on this.)
"Which data sets are available?" Which regions are covered? What kind of information is captured?
"What is the quality of the data?" An assessment of map completeness (coverage), consistency (e.g. of annotations), and various measures of accuracy. An assessment of the age of the data, and of its provenance: which imagery sources were used to produce these maps?
"How can we access the data?"
"How can we integrate it with our information systems?" For example, how well does it map to the Humanitarian Data Model, or other standard data models?
The QA process: tests and reports
I. Basic report (derived from OSM edit history):
- How much data is there?
- How many people contributed?
- How old is the data?
II. Coordination report (derived from edit history and TM2 data):
- HOT project identifiers: links to the projects that produced this data
- Have contributions been reviewed (validated)? where? what changes were made?
III. Automated QA (basic validation):
- Untagged objects
- Overlapping objects
IV. Annotations report: which annotations are available?
- Geospatial information: road names, place names, ...
- Data provenance: description of imagery source
- Data management: review-related annotations (e.g. 'typhoon:reviewed')
V. Humanitarian data report (derived from OSM edit history, HDM):
- What map object types have been mapped? how many objects are there?
- E.g. "150 buildings, 15 hospitals, 3 helipads"
VI. "Fitness for purpose" reports: assessing the availability and completeness of data in relation to specific needs:
- Availability of building data needed for population density models
- Availability of road data for transport planning
- Availability of infrastructure data (hospitals, schools, helipads, ...) for aid coordination and logistics
Should a QA support tool also include its own workflows to address specific issues, or focus on descriptive reports as outlined here? Will our existing validator workflows remain sufficient as we grow?
Who should be doing QA work? How much of QA requires "expert" knowledge? Can we consider QA a general community activity that's open to all? E.g. by using guided workflows with good documentation. (This is also a discussion about HOT validation practices.)
In a response to an acute shortage of validators, Missing Maps in London are now training people up at their monthly events: first to learn JOSM, then validation. I think that's great! It's particularly fitting that validators are trained from the same volunteer pool as new HOT contributors. That way, at least in principle, their numbers can grow together. While currently validation is often on the shoulders of a few expert insiders, in this new model it instead can become an important training aspect for larger numbers of highly engaged HOT contributors. Becoming a validator could be an important rite of passage for certain new contributors.
It's good that validation in particular is now being taken so seriously. It enacts an important process by which the OSM community can manage the growing flood of incoming contributions. Unchecked floods of contributions are likely harmful in the long run: to the quality of the map, and to OSM maintainer morale. As Elinor Ostrom demonstrated: such shared limited resources need to be managed and defended by its beneficiaries.
I think validation also fulfils an important social role for newcomers: it can provide encouraging feedback and useful training experiences, and in the beginning these exchanges may be quite impactful. Maybe most importantly, validation provides a rare opportunity for a contextual social encounter. It's a perfect opportunity to catch first-time contributors for a chat, without requiring them to subscribe to a mailing list or join IRC first. It's a form of socialisation: the teaching of techniques and community norms.
Any such validation process however must acknowledge the scale of the challenge: that most HOT contributors likely ever only contribute little, and may never return. The first encounter should be brief yet impactful, and most importantly it must be repeatable at large scale. Thousands, and maybe soon millions of times.
It must also be noted that validation is not (yet) a well-defined practice, and instead often depends on the interests and skills of the individual validator, and on how they were trained -- just like any other OSM contribution. Everyone does it a little differently. I liked Lisa Marie Owen's recent diary entry on her global validation procedure, with good discussions in the comments; there are likely many other examples.
I wonder if there's also an opportunity to create validator networks: online or offline places where validating users can meet with other domain experts, where norms can be clarified and negotiated, and where contributors can hang out and bond. IRC channels, Facebook groups, mailing lists, meetups, ... whatever may be practical. Maybe there's already an existing space in the OSM or HOT universe where we could send new validators?