OpenStreetMap

Are maproulette challenges undiscussed mechanical edits?

Posted by imagico on 28 March 2017 in English (English)

Maproulette challenges have become fairly popular recently, especially due to Jochen's Area fixing project. But it seems this has gotten out of hand now and creates serious damage to the OpenStreetMap project.

In general this kind of tool is prone to inviting mechanical work. But with the recent Island and Shoreline Alignment challenge this really gets over the top. I first saw this when various edits turned up in remote areas of the world by various mappers in very high frequency editing islands in changing locations far apart within minutes, often without factual basis and often factually incorrect.

This challenge does everything wrong that can be done wrong with a fixing effort:

  • there are no useful instructions to the mapper what to do and what problems to consider. It only says: ''Align the highlighted island to match imagery''.
  • there is no documentation who created this task and how the allegedly misaligned islands are detected.
  • and most importantly: the task covers areas where the global images routinely available offer no basis for improving the existing data.

Task: Align the highlighted island to match imagery Task: Align the highlighted island to match imagery (link)

Especially the last point is a big practical issue now since the edits made through this challenge misalign and worsen a lot of data in OSM. I commented on two most obvious cases where Bing offers no image at all but mappers none the less blindly followed the task to Align the highlighted island to match none-existing imagery with obvious results. But even in areas where Bing offers low resolution images these hardly ever allow improving existing data. These images in Bing are mostly from >15 years old L1G Landsat 7 images which have positional errors of sometimes more than 100m and rarely allow substantially improving existing mapping. In most cases attempts to do so made within the challenge worsen data which has often been mapped from either better data or with better alignment of the images.

With edits like this the island in question is not necessarily less accurate than before - the mapping before was done based on - if at all - only slightly better aligned images. But it is no improvement since it is likely not more accurate than before in absolute terms and is definitely less accurate relative to the surrounding features.

Yes, these images show a lot of potential to improve mapping here Yes, these images show a lot of potential to improve mapping here

The least that needs to be done here is

  • stop the challenge as it is now
  • limit it to areas where high resolution images are available in common sources
  • create clear instructions for the mapper advising them to properly check image alignment and find the best quality sources in the area and check if existing mapping might already be of better quality based on other sources. This is not possible to do within a few minutes if you pick random locations all around the world of course.

Of course not all edits made within these challenges even in areas where Bing is poor are bad, there are also some experienced mappers participating here who know how to properly assess and select images.

If whoever created this challenge seriously wants to improve mapping in remote areas the most basic, most productive and most obvious way would be to provide better quality images.

And in general i think we need to put a review regime on organized mapping efforts like maproulette challenges requiring at least basic documentation of the process used to generate the task and ensuring there are proper instructions for the mappers and no nonsense tasks on a larger scale. A lot of thoughtful tasks have been offered in maproulette in the past but apparently this is not something that can be relied upon to be ensured without a QA process for the QA process...

Comment from Zverik on 28 March 2017 at 11:27

So that's where these people come from! I've seen one shifting islands in my watched area. Had to teach him an imagery offset plugin.

Hide this comment

Comment from imagico on 28 March 2017 at 12:04

It seems by the way this challenge has been flying under the radar in Maproulette - you don't see it on the metrics or using the search function, you need the direct link. I wonder where all the people working on this got the info from...

Hide this comment

Comment from Jochen Topf on 28 March 2017 at 12:06

I understand the problem, but on the other hand: Where do we stop? Do authors of QA tools like OSM Inspector and Osmose don't have similar "power"? What about authors of a wiki page? They also can mislead lots of people into mapping something in the wrong way.

In contrast to totally automated edits, we have a human in the loop with Maproulette & Co. So the problem is maybe not on the part of the person creating the challenge, but on the part of the mappers who work on it. Maybe they should have to prove that they know what they are doing?

I think it all leads back to: We need better documentation and especially better best-practice type documentation to teach everybody, regardless which mapping technique they use.

Hide this comment

Comment from Zverik on 28 March 2017 at 12:24

I wonder where all the people working on this got the info from... I assume they are working at the same company, not sure which one. Each of these has got thousands of similar changesets:

Googling these names shows that they also participated in #MapLesotho, nothing else.

Hide this comment

Comment from imagico on 28 March 2017 at 12:31

@Jochen - we don't know yet how this challenge was created so it is somewhat premature to draw far conclusions but obviously you cannot identify potentially misaligned islands without external data unless you simply put all islands in the challenge which does not seem to happen here. This makes it very different from tools like OSMI or Osmose which focus on internal inconsistencies in the data.

The primary problem is here definitely the person who created the challenge, instructing mappers to align the highlighted island to match imagery which is just a plain wrong instruction here misleading mappers into doing what they should not do (namely aligning the island to what they see in the images without knowing the area, the imagery or the previous mapping work done).

The secondary problem is that those running Maproulette have apparently no system in place to ensure even the most basic level of quality on the challenges posted. At the same time and in contrast to the wiki (which you mentioned for comparison) others have no way to fix problems with the challenges and remove misleading stuff.

And after that there is of course also the problem that some mappers let themselves be lured into following the faulty instructions given and damage the map in a misguided attempt to improve the map. But the responsibility lies with others, you cannot just put a mapper into the loop and then put all the blame on the mapper while you hide in anonymity.

Hide this comment

Comment from imagico on 28 March 2017 at 12:34

And yes, of course these are different from a fully automated edit, my title is a bit of a clickbait obviously.

Hide this comment

Comment from SimonPoole on 28 March 2017 at 12:53

Have any comments been left on the relevant changesets?

Hide this comment

Comment from imagico on 28 March 2017 at 13:03

I have already linked to two commented changesets (where i did not yet know this is a Maproulette challenge):

https://www.openstreetmap.org/changeset/47030425

https://www.openstreetmap.org/changeset/47032866

Today i made another comment:

http://www.openstreetmap.org/changeset/47195925

Hide this comment

Comment from mvexel on 28 March 2017 at 15:34

I don't think this is necessarily a MapRoulette problem, but I do agree that it would be beneficial to have challenges checked for some basic quality standards, so that people who fix things through MapRoulette are properly prepared. That is part of the reason why I asked for someone or a few people to step up and become Challenge admins in my latest MapRoulette newsletter. I don't yet know precisely how that would work, but I envisage for now that I would provide a weekly list of new public challenges and the users who created them for the admins to review.

In this particular case, I would get in touch with the Challenge creator (there is a link in MapRoulette to their user profile) and ask them to comment here. Any problems with individual mappers making bad edits should be addressed with them. Again, I don't think that users making questionable edits is specifically a MapRoulette problem. If I were to shut down this Challenge, What is to stop them from just scrolling along the coastline looking for islands to fix in the same way they are doing it now?

Hide this comment

Comment from TomH on 28 March 2017 at 15:40

Obviously I'm missing something then because I've searched everywhere that I can see in the MapRoulette information and I can't even see a name for the creator, never mind a link.

Hide this comment

Comment from imagico on 28 March 2017 at 15:51

In this particular case, I would get in touch with the Challenge creator (there is a link in MapRoulette to their user profile) and ask them to comment here.

I see no way to determine who created this challenge within Maproulette.

If I were to shut down this Challenge, What is to stop them from just scrolling along the coastline looking for islands to fix in the same way they are doing it now?

Actually that would be a great improvement over what happens with the challenge. Right now mappers tend to edit only the island in question before moving to the next one in a completely different area and not look around and edit more stuff in the same area. Look for example here or here where there is lots of stuff to fix and improve around (the second one is near to what is probably the largest group of unmapped islands in Europe). Without the challenge mappers inclined to fix inaccurate islands would select an area they are interested in from the start and would also learn further about the area while working on it. This would be much better.

Hide this comment

Comment from maxerickson on 28 March 2017 at 15:55

mvexel, could you explain how to find the challenge creator on MapRoulette?

I can't work out how to do it.

Hide this comment

Comment from mvexel on 28 March 2017 at 16:12

The challenge creator should be visible when you click the challenge's marker on the map on the main page. You can use the search box in the top left.

However if you try to apply that to this challenge you will come up blank. The reason why is that the user's project is not set to 'visible' which makes all the challenges that belong to their project also won't show up. The creator apparently did not intend to make this challenge public yet, and as such I made a mistake to highlight it.

The 'visibility' of a challenge only applies to whether it is visible on the map on the main page, and whether it shows up in the search results. It will still show up in the API. The reason why users can't make challenges completely 'private' is exactly what we are talking about here. I don't want there to be some 'shadow' activity that nobody can inspect.

As you can see from the API output the user who created this challenge has OSM ID 5441336.

I realize that it's not so easy right now to get a complete overview of the Challenges available in MapRoulette. It is one of the priorities to fix this. One thing someone could do in the mean time is to use the MapRoulette API to create an external list of Challenges.

Hide this comment

Comment from mvexel on 28 March 2017 at 16:20

To clarify how to find the challenge creator for a visible challenge: use the search box (or the map) to identify the challenge you are looking for:

search

Then click on the marker, which will have a link to the OSM user who created it:

info

You will see 'SuperUser' for older challenges because we did not store creator information in earlier versions.

Hide this comment

Comment from imagico on 28 March 2017 at 16:39

@mvexel - Since massive work on this challenge by the mappers mentioned above started before you posted the link in public the challenge obviously has been used as exactly the kind of shadow activity you did not want.

I am not really inclined here to try finding out details about this via PM from a user who obviously wants to hide his identity with a user account specifically created for this purpose.

To me this all just emphasizes my initial impression that we need enforceable regulations for this kind of large scale organized mapping activity.

Hide this comment

Comment from woodpeck on 29 March 2017 at 10:16

I agree that we do have to think about how to deal with large-scale directed mapping activity, and what requirements we have there. Like with an import or a mechanical edit, a mistake that someone makes in designing a task for 100 employees or 100 students or 100 contractors to follow, can have dire consequences and they can unfold quickly. I think that Martijn's plan to have volunteer challenge administrators is a good first step provided we can find the volunteers ;) community guidelines about directed mapping can follow later.

Hide this comment

Comment from imagico on 29 March 2017 at 10:55

I have my doubts if without a clear mandate in form of binding rules volunteer administrators can do much good here but of course Maproulette can introduce such rules independent of the OSM community as a whole and if they work well this might serve as a basis for developing a broader general policy.

In all fairness one other thing needs to be said - that a contributing factor for the problems of this challenge is that we still use totally unsuitable images for mapping from Bing and Mapbox in many parts of the world and editors do not even give a warning not to. This would be very easy with Bing (Capture date in the metadata is 1/1/1999-12/31/2003 or 1/1/1999-12/31/2014), somewhat more complicated with Mapbox. We cannot and should not forbid using these images but a clear warning to any mappers doing this from the editors would be a big advantage.

Hide this comment

Comment from Zverik on 29 March 2017 at 11:41

I think we should agree that the issue discussed here is not related to MapRoulette. The website is used only as a coordination tool. When you create a challenge and advertize it, you get a few interested people who make a dozen changes. Not twenty mappers who work all day producing thousands of changesets to complete your challenge. MapRoulette is good, making undiscussed mass edits, regardless of automatization, is not.

Hide this comment

Comment from imagico on 29 March 2017 at 12:05

Yes, agreed. And @mvexel already indicated he does not want Maproulette to be used for shadow mapping activity and i assume there will be changes to prevent such abuse in the future. And if there is further need for more thorough QA for the challenges posted there would be options for that too (like for example a grace period between publishing a challenge and it becoming available for mappers to work on which can be used for evaluation).

Hide this comment

Comment from imagico on 30 March 2017 at 07:34

Editing has resumed, some of the changeset comments have been replied to, including the admission that this is an organized effort but still pretending this to be a normal group of regular OSM mappers and not telling who instructed and is paying them.

Hide this comment

Comment from imagico on 31 March 2017 at 09:34

A quick update: the challenge now has a more elaborate description:

This challenge consists of islands which have overly sharp angles or are over-simplified. Check all available imagery before editing the islands. Bing may be offset, so only fix a task if you have calibrated imagery of the area. Please skip tasks where imagery is not available. Please familiarize yourself with the proper alignment of coastlines, this information can be found on the OpenStreetMap Wiki.

This is not quite correct, tasks also exist for islands with fairly smooth outlines so their thresholds are fairly off but it explains how they produce their tasks.

And those working on the tasks now seem to be more selective in what they change, there are now nearly as many new tasks marked as false positive as there are as fixed.

Overall this is now mostly sad in terms of waste of ressources. All of these problems could have been avoided if they'd have discussed their plans beforehand. If half the tasks you look at turn out to be bogus that is just not very efficient.

So a policy for organized editing would not only be in the interest of OSM, it would also be in the interest of those organizing such edits. ;-)

Hide this comment

Comment from imagico on 19 April 2017 at 12:45

Another short update: There has now been a user block put on the involved accounts by the DWG:

https://www.openstreetmap.org/user_blocks/1319

Editing activities in the meantime had continued mostly unchanged after the previous update (which is more selective than earlier but still often questionable regarding the whole approach). The Maproulette challenge is about half complete now with about half of the tasks done being marked as false positive.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment