3 years of welcome messages, more than 3400 of them

Posted by SimonPoole on 13 December 2016 in English (English). Last updated on 14 December 2016.

Three years back I started sending welcome messages to new mappers on behalf of SOSM, using a simple tool chain based on the RSS feed of new mappers from Pascal Neis.

The message is not customised as from the beginning it was seen as a replacement for the welcome message that had gone away with the web site redesign. The text (in 4 languages, no Romansh version though) is designed to welcome the new contributor and give pointers to the resources available nationally.

Has it worked? Difficult to say, as the whole point was a replacement of something that existed before, we didn’t expect drastic change and we didn’t get it.

And, no, I’m not stopping :-)

Comment from Andy Allan on 13 December 2016 at 11:16

You could try an A/B experiment for a while - take the next 1000 users, send welcome messages to 50% of them, and then see if there’s any difference between the two groups of users.

Comment from Vincent de Phily on 13 December 2016 at 16:30

Or even compare against all the users (in other locations probably) which you didn’t welcome to see if, for example, the retention ratio is better ?

Comment from Andy Allan on 13 December 2016 at 16:32

Vincent de Phily - you would need to choose them randomly for it to be effective. If your choice of send/not-send is based on a particular factor (e.g. location) then it skews the results of the experiment.

Comment from SomeoneElse on 14 December 2016 at 01:35

A while ago I did have a look at the retention rate in the Polish community, where 75% were welcomed and 25% not (see and the links back from there). There wasn’t an appreciable difference when I looked between 2 and 3 months after the message was sent (or not).

What I didn’t look at that would have been interesting was contribution quality, and I’ve not gone back to look at longer term retention (anyone can do this - just look at new Polish mappers from that period whose UID was divisble by 4 or not).

Comment from SimonPoole on 14 December 2016 at 11:16

@Vincent de Phily the difficulty is naturally that, on a larger scale, we don’t actually know who got a welcome message or not (only who we sent one).

The more general problem with measuring the success (or absence of such) is on the one hand that the numbers are very small: we would expect perhaps 10% of contributors to edit more than once and an even smaller number to become regulars. Halving the numbers to do A/B testing (I wouldn’t want to stop for a while because sign ups very often depend on external factors which we can’t control) is not going to help with that. And on the other hand that effects may be hard to measure, contribution quality particularly. We do know, as already said that there hasn’t been any drastic changes, wit: which would appear to be stable (contrary for example to Germany where the number of daily editors has been going down for a while).

Interesting enough one thing has subjectively changed over the last 2-3 years: we have a lot less broken municipality boundaries than we used to have (it used to be a at least once per week, and now that is extremely rare), but that could just as well be due to better tools.

Comment from mmd on 16 December 2016 at 18:43

Has it worked?

Well, it would be interesting to hear if you’ve got any feedback from new mappers? Did they find the welcoming message useful/interesting/helpful?

The message is not customised as from the beginning

I used to monitor my area for new mappers for some time, try to guess what they’re most interested in and sent them some customized hints. Certainly, that approach doesn’t scale at all, but it would be interesting to if such targeted hints would be more effective, or if it just doesn’t matter.

Comment from SimonPoole on 16 December 2016 at 19:00

@mmd rarely, however I do superficially check contributions and if necessary follow up with a 2nd message or changeset comment, that will typically get some more feedback.

Login to leave a comment