OpenStreetMap

The first 30-day challenge: retrospective

Posted by bdiscoe on 12 March 2014 in English (English)

The first Scout-Telenav 30-day OSM Mapping Challenge just ended. Let me share some of the story.

When it was announce February 11, I was excited. At that time I was already an "addicted mapper", and fairly sure of my fast-accurate JOSM editing skills, so I figured I could win it. The challenge was for the USA. I usually trace Bing in remote parts of the world, but I did know of a lot of roads in Hawaii that could be quickly cleaned up, so I figured that would give me a quick start.

Week 1

My Hawaii edits did produce a good number of points, but experienced Canadian mapper ingalls was in the lead! He was cleaning Tiger in Texas at an impressive rate. I was slowly catching up, but he remained ahead.

Week 2

Suddenly, when ingalls and I were both at ~30k points, he stopped mapping. I breathed a sigh of relief and took the lead. I found myself doing too many steps in JOSM while editing, and started wondering if I could set up keyboard shortcuts that would let me go faster...

Week 3

Just when I seemed safely in the lead, a user ada_s appeared in the rankings and rapidly went up to second place. All their edits had the same comment, "Add address information + split way when exiting the city border" That seemed like an odd thing to do, but it sure racked a lot of points. I struggled to find enough time to stay ahead (I do have a full-time job and girlfriend) and ada_s continued to gain. At this point, my exploration of the JOSM scripting engine produced some results - I was able to create a lot of single-key shortcuts (like Shift+S, set highway=service) that let me go faster (more about those scripts in my next diary entry). I was working faster now, but ada_s was still gaining on me.

Week 4

I pulled a couple late nights editing, which put me at 57k points but ada_s was at 50k and picking up speed. After another day where our scores both leapt up, I finally took a look at exactly what ada_s was doing. They were putting "addr" and "in_in" tags ... on highways. Like, every single road and driveway in Lincoln Nebraska was tagged with "addr:city=Lincoln" and "addr:state=NE". This seemed very odd to me (not to mention useless), so I took a look at the page for addr and sure enough, it doesn't say anything about using it on highways (because, why would you?) I sent ada_s a note asking politely why they were adding those addr tags. I also put in a few changesets removing those same tags from a few cities where ada_s had added them (along with other improvements). I then found a particularly messy Tiger region in South Carolina, and dug into it for another late night, my JOSM edits now at great speed. ada_s never responded but they did, suddenly, stop editing. (Maybe they just didn't know that those tags were useless and nonstandard? It could have been innocent.) They were up to 72k by then, but partly due to undoing their odd tags, I was at 108k. I pulled one more late night then stopped myself. My final score was 145k, ada_s at 72k, followed by quality good editors like "rickmastfan67" and "jonesydesign" at 40-50k.

Conclusion: Having a contest to make the most "edits" does risk people going for questionable things that touch a lot of ways. Perhaps 55k of ada_s's points were in that category (and hence 55k of my own score undoing them, so my real score should be around 90k; still in first place but not crazy). However, I'm certain that the contest did inspire a big increase in overall quality editing. I certainly got a lot faster, learned JOSM better (and spent time improving the USA, where I usually wouldn't bother)

They're doing the contest again ("with simplified rules-and more prizes to win") and that seems like a good thing to me. I won't be entering next time (to give you all a chance :-) and I'll be sharing my JOSM extensions in my next post. My main interest is in getting everyone more productive at editing, for the greater good of OSM.

Comment from AndiG88 on 12 March 2014 at 07:13

Conclusion: Having a contest to make the most "edits" does risk people going for questionable things that touch a lot of ways.

First thing I expected when reading about the contest :/

It also means that unless you are 100% committed you stand no chance.

I think contests around OpenStreetMap should rather be based on making a few edits over a certain time frame (make 5 edits every day for a week) and then you pick a winner out of all who managed to do that. This would also get a lot more people started with OSM.

Maybe you could also base it around a Quality Assurance like a bet in Germany Or you limit it to a certain tag for example opening hours added to shops.

Hide this comment

Comment from Vincent de Phily on 12 March 2014 at 10:01

Thanks for the retrospective :)

There's always that problem with OSM contests: people will start gaming the scoring algorythm at the detriment of contribution quality. This is especially true if you target the contest at relative newbies (as I assume most scouts are).

It's very hard to avoid that (it's human nature), and I imagine it's one of the reason why we don't see more contests (which are otherwise fun and positive). Maybe we should ditch automatic scoring altogether, and use a jury appreciation instead. But of course that a lot more work to organize.

Hide this comment

Comment from russdeffner on 12 March 2014 at 10:48

Thanks and Congratulations! Double thanks for sharing your thoughts on how to make a better contest all around, and...

Happy Mapping, =Russ

Hide this comment

Comment from Jedrzej Pelka on 12 March 2014 at 11:25

We need to find a good way to judge one's contribution. Number of edited objects is not a good measure, there is little correlation between number of edits and their value. It's easy to make a massive edit with little added value, but an edit adding very important information or solving a mistake might be of a very little weight.

Hide this comment

Comment from bbmiller on 12 March 2014 at 14:22

Yes, thanks for the recap. If nothing else, this contest provided an interesting opportunity to see the various techniques people used to make edits, and score points. I guess I'm kind of proud to be at #9, considering I spend most of my editing time in iD.

Hide this comment

Comment from nfgusedautoparts on 12 March 2014 at 22:41

i think a better approach to the behavior of ada_s would have been to notify both the DWG and the contest organizers, and let them deal with it. deleting all of his work is a little too close to engaging in an edit war for me to be comfortable with it.

Hide this comment

Comment from Your Village Maps on 12 March 2014 at 23:18

I want to "second" what Jedrzej Pelka said. I couldn't say it any better. It looks like this new contest might eliminate some of the potential "cheating" problems that were in the first contest. I also know I would have gotten a lot more points if I had more quickly gotten my "contest focus" instead of tending to keep editing like I usually do (adding things in the areas I'm working on streets that count for fewer points or even no points at all).

It also isn't 100% clear to me how you got so many points so quickly, especially when your time is so limited by a full-time job and girlfriend. I don't have a gf or even a part-time job right now, so I had plenty of time to do edits in the contest. I have to ask about that, bdiscoe. Other than using scripts, which was forbidden according to the contest rules, how could you produce so much in so little time? Sure, keyboard shortcuts would help, but still...that doesn't seem like it would be enough.

Hide this comment

Comment from rickmastfan67 on 13 March 2014 at 05:28

@nfgusedautoparts I had reported ada_s to the DWG & the contest back on Feb 21st. I did get a response back from the DWG that they were going to contact him, but it seems like it got lost in the shuffle and nothing was ever done. As for the contest people, no idea if they got the e-mail or not...

=====

@bdiscoe I'm with 'Your Village Maps' on wondering how you could get so many points so quickly when you claim to be limited by a full-time job + gf. I'm in the same boat was YVM right now, and I couldn't even score the same amount of points in one day as you (were you using your 'Auto roads' to do a lot of your edits as well?). I mean, I took ~7 hours on one changeset and I was lucky that day to get, I think it was, just over 3000 points for that work cleaning up all of I-376's interchanges (lane counts and alignment fixes) and throwing in the parking lots @ Pittsburgh International Airport into the same changeset.

I also did notice earlier in the contest that you did several 'name expansion' edits using your script that you posted recently about in a previous diary entry (one of them with over 700 edits and you did it somehow in 7 seconds from a previous changeset and you gained ~2100 points). You do know that using scripts like that was against the contest rules, and should have lead to a DQ, right?

  1. Only edits for the United States map will be awarded points. Edits for other countries will not qualify for points. Edits must be made by editors themselves. The use of any automated launching, entry software or any other mechanical or electronic means that permits the participant to automatically register and/or edit repeatedly is prohibited and all such entries will be disqualified. Editors registering multiple user IDs will be disqualified.

http://webcache.googleusercontent.com/search?q=cache:EX5wKJoSoaUJ:www.scout.me/osm

Don't know how long that link will be active with old contest page, but that's a direct copy and paste from the rules of the contest.

Oh, and I thought you weren't going to enter the new contest... I do see you're doing edits for it already it seems.

Hide this comment

Comment from bdiscoe on 13 March 2014 at 17:40

@YourVillageMaps I wish I could record and share a video of me doing Tiger cleanup in JOSM with my shortcuts. The pixels fly! :) I also wish i could teach others how to do it. Telenav should hire me to train a building full of interns to do it, then they USA would be clean in no time :) (It also helps to have fast internet, a large monitor and a good mouse.) I am very proud of the work i got done in this contest, for example. the entire city of Lumberton, NC (http://www.openstreetmap.org/#map=15/34.6269/-78.9999) (And Dillon, SC, and several other towns in that region.) I also did tons of work like adding buildings, fixing streams and wetlands etc. even though they didn't count for the contest, just because i love editing. I also deleted lots of entirely non-existent Tiger roads, which also didn't count. I also fixed/added hundreds of bridges (over those corrected streams) which presumably counted as "adding a road", since bridges do involve inserting a way.

Hide this comment

Comment from bdiscoe on 13 March 2014 at 17:44

@rickmastfan67 I am very impressed by your work on interchanges, which sounds much more laborious than general road cleanup, which would explain why it takes more time. It's valuable and appreciated! I wish i had an "auto roads" that worked, but that was just a programming experiment some months back. As for the term about "automatically [..] edit repeatedly", it doesn't seem useful to interpret that in a way that would prevent, for example, the use of JOSM's wonderful, powerful auto-validation commands. I certainly review every individual change before commit (as should everyone, contest or not) so that the changes are not "automatic", as a bot would do them. In any case, I had initially thought that fixing abbreviated street names would be a big help, but in the end it turned out to be a very small part of the changes I made. The great majority were: 1. Cleaning the noise out of really bad Tiger roads in South Carolina (JOSM Ctrl+F Find "way highway tiger:reviewed=no", then Shift+Y Simplify, then a quick manual touch-up pass over the affected roads.)
2. In endless suburban sprawl where the geometry isn't so bad, using Shift+T to add the missing turning_circles, scooting a few nodes to align with Bing, confirm the name and then T to clear Tiger.

Hide this comment

Comment from bdiscoe on 13 March 2014 at 18:38

@rickmastfan67 I'm not entered in the new contest. I was just testing out the freshly released, newly improved maproulette for curiosity and to support Martijn's great efforts. There was an initial glitch with Chrome not popping up a confirm on fix, which got fixed. I do wonder, though, how this new contest will work, since the maproulette issue aren't many, it seems they'll run out very quickly. Maybe Martijn has many more kinds of detectable issue coming soon.

Hide this comment

Comment from mvexel on 13 March 2014 at 20:22

Ben - Thanks again for taking part and for the great summary, and congratulations :) !! We (speaking as a Telenav employee) knew this was going to be a challenge not only for the participants but also for us. We learned some things along the way, and the new iteration of the contest (started yesterday) channels participants to MapRoulette so we can be more certain that we get quality edits. Let me know if we can do more to improve on the contest idea. I do believe a contest can be a good tool to help improve OSM faster, and I am eager to get it right.

And as for MapRoulette - it's an open source project and while I initially came up with the idea, the current version is as much Serge's work as it is mine. So credit where credit is due. Also, anyone with a good idea for a challenge and some server resources and scripting skillz can submit challenges to MapRoulette. I will be writing up a how-to some time soon. Let me know if you are interested!

Hide this comment

Comment from baditaflorin on 13 March 2014 at 21:29

Me, as a engineer and a hacker, i would do the same, but adding relevant tags.

I would see what i can automate, like selecting all the buildings inside of a city and adding the city address, or something like this.

But i think that now is irrelevant, because this can be calculated automatic.

then i would add all the street lamps that i can find, and power tower`s, i would be able to add around 500 power towers per hour, at least, so in 6 hours i would have 3000 power pole,

Then would be to slow, so i would copy and paste the tag:pole=power, or what tag i would use, being able to add around 1500 per hours, or 9000 power pole`s in 6 hours.

So, for me, with this contest, would be how i can automatize and be as efficient as possible, while adding a lot of relevand data

Hide this comment

Comment from aseerel4c26 on 22 March 2014 at 21:11

On 12 February 2014 I exchanged six mails with Alex-7 (who had added a note of the contest in the Wiki calendar). Let me quote an excerpt of what I wrote to him since it is (if I read correctly) matching the reality.

I see problems in giving people a monetary reward for "points". They may not act in the best interests for the projects but in their own interest. And if the point system is not designed carefully this may lead to careless, inaccurate and maybe even wrong armchair edits which are not useful.

[…]

if the point system is designed to be matching our project goals. No Sweatshop workers in India (badly) "mapping" in the US to win tablet computers, please. By the way: I did not have a look at the actual point system, but I just guess it is not designed good since this is very hard.

Hide this comment

Comment from aseerel4c26 on 22 March 2014 at 22:28

… I would like to add: this quote sounds only negative. However, the challenge certainly had its good parts. Probably the overall effect of it is positive.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment