OpenStreetMap

Mapillary seems to be Losing It’s Marbles

Posted by alexkemp on 9 February 2019 in English (English)
  1. Mapillary seems to be Losing It’s Marbles
  2. Git Along, Little Dogies
    (speeding up renaming files)
  3. I’m losing my Marbles because of Mapillary
  4. A new record for Mapillary - 62 blurs on a street sign

Mapillary is a location-photo-sharing site dedicated to inter-working with OSM & I like it a lot.

Mapillary is the street-level imagery platform that uses computer vision to fix the world’s maps.
450.7 million images; 6.5 million kilometers covered

I am registered on Mapillary since Apr 10, 2016 and it tells me that in that time I have uploaded 6,200 photographs shot whilst walking across 244.7 km of Nottingham’s suburbs. Most of those photos are registered as links with various PoI on the map, which means that someone browsing OSM can find a link attached to a Nottingham point-of-interest that will instantly let them see a blurred photo of what it looks like 32% of the time.

Blurred Photos

At the moment that I am writing these words the following photo has 33 auto-blurs attached to it, 32 of which affect the street-sign on the right (the 33rd is on the brickwork to the right):–

blurry street-sign

At this moment I am going through each & every one of those 6200 photos removing the stupid-blurs. Nothing will happen until at least next Monday (11 Feb), as all the blur-removals are actually blur-removal requests (a Mapillary admin has to agree/refuse the request).

33 blurs is bad, but not the worst - one photo had more than 40 blurs (a patch of shrubbery) (whilst another just now had 45 all on top of another street sign) (and the record worst has 62), all of which were stupid-blurs.

So what is going on?

Mapillary has always edited uploaded images before showing them:–

What Mapillary blurs:

  • Faces
  • Vehicle license plates
  • Inappropriate or private parts of the image

What Mapillary does not blur:

  • Whole people
  • People from behind when the face is not seen
  • Billboards, advertisements, brand labels
  • Street, traffic and information signs

So what is the problem?

I have zero problem with Mapillary enforcing privacy blurs. I’ve dealt with too many MPGs spontaneously combusting just because I’m stood outside their house asking if I may take a photo of their house-name not to know that the English have a particular issue with Google StreetView (if you get my drift through all those disparate topics).

In the early days Mapillary kept blurring out street-signs, and it has taken quite a few years for them to stop doing that. Most of the time now it seems to work OK. However, the Blur Algorithm appears to have recently been put onto steroids [update: actually Apr 2018; see bottom], in that it is blurring areas so small that they can barely be seen. Worse, the programming has schoolboy-level errors; as just one example, some of the street-sign blurs are quite large yet inside that area may be yet another 5 smaller blurs - what is the point of that?

Look at the photo below; it is the example used in the help-page to illustrate what the privacy-blur edit-page looks like. Those orange/green/white numbered circles locate the presence of a blur enabled by the Mapillary privacy-bot.

Mapillary privacy-blur edit-page

Each blur has to be removed one-by-one using a mouse, no keyboard allowed. This is tedium ratched-up to mind-numbing proportions.

  1. Hover over a blur
  2. Move the mouse up to a combo of green tick✔ + orange cross✘ that then appears
  3. Hover over & click on the cross
    (maybe you are hovering over the cross, maybe not)
  4. Next blur…

My constant experience across the last two days has included a bunch of 5-or-more of those circles located on top of each other in such a fashion that I cannot click on the ones underneath. Or of circles located under image navigation features (arrows & chevrons, forward/back arrows or attribution-feature); those are all located in top-of-z-order, which means that nothing underneath them can be clicked on. These are what I am calling “stupid-blurs”.

The Modified Date for the Blur Help Page is January 09, 2019 13:01 (there is a set of blurs on all earlier photos in addition to these recent blurs; those earlier blurs are no longer editable). I therefore assume that the blur-bot was very recently unleashed on all 451m Mapillary photos to enforce further blurs. Or, more accurately, further stupid-blurs. Get more intelligent more quickly, please, Mapillary-bot!

Stats

Being a scientist I thought some measure of the problems was required.

I’m working my way from latest to earliest and have already gone through 26 sets of photos. Here are the accumulated stats from the 27ᵗʰ set of 111 photos, shot on 31 January 2017 in the streets near Phoenix Avenue close to Gedling:–

  • Total photos: 111
  • Contains blurs: 35 (32%)
    (following applies to blurred photos only)
  • smallest # blurs: 1
  • largest # blurs: 25
  • Total blurs: 233 (avg: 7/photo)
  • Valid blurs: 19 (8%)
  • Stupid blurs: 214 (92%)

Addendum

Tues 12 Feb: Mapillary have sent me an email to say that the team is looking at it.
Tues 19 Feb: Map email: “The dev that would be the best to investigate and answer these questions is currently on vacation and will be back after this week.”
Fri 22 Feb: Shortly after 11pm I corrected the last blur on the last set of photographs; earlier in the evening I had saved the last ‘lost’ photo to my home hdd. Mapillary have still to apologise for the many bugs & errors, or acknowledge any of these blur corrections, let alone act on them. They say in a recent email “our algorithms work on street level imagery at best and tend to fail with close-up imagery, such as close-up images of signs.” Ah, spit.
Sun 24 Feb: The stupid-blurs appear to have started on 19 Apr 2018 and took 3 weeks to vandalise all images (500 images/second).

The goal is to blur all identifiable faces and license plates. At the same time, we need to preserve important information in the images for mapping. This makes it a challenging problem.

The resulting privacy blurring model is able to detect 99.9% of all identifiable faces and license plates while wrongly blurring only 0.078% of the pixels. This means that on average, we miss just one out of 1,000 identifiable faces and license plates.

Sun Apr 21: 6,600 photos processed for Stupid Blurs™ and zero response, so yet another email:

I’ve uploaded another set of photos and have removed hundreds of Stupid Blurs™ from them, just like all the previous photos. None of those edits have been acknowledged. Nor has any of the previous blur edits been acknowledged nor acted on. That now makes it:–

  • 6,600 photos uploaded since 10 April 2016
    (3 years; 1,102 days)
  • 13,854 blurs examined
    (avg == 7/photo)
  • 12,724 Stupid Blurs™ marked for removal
    (92% failure rate)
  • 12,724 Stupid Blurs™ ignored by Mapillary
    (100% failure rate)

So far you have lied in your every response to me.

12 Feb: we are looking at it.
19 Feb: the dev that is best to investigate is back next week
6 Apr: we have had processing delays recently.

“processing delays”? No kidding.

I checked carefully, and the words “Collateral Damage” and/or “Bugs” are not yet used at any point.

Location: Arnold and Carlton, Gedling, Nottinghamshire, East Midlands, England, United Kingdom

Comment from westnordost on 17 February 2019 at 15:41

I hope the blur-algorithm is actually a machine-learning one, so your corrections actually make the algorithm better.

Comment from alexkemp on 17 February 2019 at 15:54

Hi westnordost

My recall from early days is that yes, it is a so-called intelligent algorithm. I certainly hope so, since that is the reason that I keep plugging away making the edits.

My main worry is that there has been zero feedback (support says “I have fed your comments to the developers”, but nothing other than that) & zero changes to edited photos. This means that I am unsure whether my edits are even connected to Mapillary & their routines at all.

Comment from majkaz on 22 February 2019 at 09:47

That is a longstanding problem, I had put several issues into the old bug tracker for Mapillary.

What I suggested: Upload the sequence without their processing, change locations of some run-away photos or change the view direction in case it gets interpreted wrong. For fully manual processing, just after this manually mark the blur areas with their app and then put it in the processing queue. This would be the most logical one if you are going manually process your own photos. For automatic processing, just confirm that the sequence has the orientation/position OK and should now go into the “area recognition” first - where cars/street/etc. are recognised. And leave the blurring process after this - and run detection for the faces just where there were people detected, license plates just where there were cars detected, other privacy probably just windows in houses. Mapillary detects all these parts, and almost flawlessly as well.

What you got: Upload, wait for processing. Change view for the sequence, wait for processing. Move some photos, wait again for it to register. Go to the blurring/deblurring process. At every photo, delete all the blurs, and do your own, because it is faster this way. You had to go one photo after another, you could not skip to the next blurred one to get it all in one go. Take a “quick” view on the finished sequence, blur what was missed.

Mapillary was generating immense load on their own servers for no reason, just because they have the processing sequence the wrong way around.

Not sure if the current process is still the same - I stopped to put my photos there and just use the sequences I take privately for my local edits. I have simply lost my patience with the Mapillary app on web.

Comment from alexkemp on 22 February 2019 at 15:15

Hi @majkaz

Well, that was excellent feedback, and they ignored it. In fact it is worse now, since you can only edit your own blurs; any stupid blurs from any other source must remain.

Comment from skorbut on 24 February 2019 at 16:44

It’s my impression that Mapillary looses the contact with the user base generating business value for them more and more. They did a few things to annoy contributors in the past:

  • Not releasing the app under an open licence (so we – the community – cannot improve it)
  • Changing the app so it suddenly requires Google services (or whatever it is called exactly) and therefore denying users with alternative Android operating systems such as Fairphone Open OS the ability to upload from the app
  • Closing the Github Issue tracker and therefore denying insights into what problem already have been reported

Things might have changed recently, I didn’t bother anymore to inform myself closely.

To be fair, Mapillary had good moments too, such as: https://forum.mapillary.com/t/why-is-mapillary-breaking-everything/2142/2#

skorbut (who has uploaded more than 250k images to Mapillary)

Login to leave a comment