Apps endanger OSM hosting by hammering the tiles servers
Posted by !i! on 6 October 2011 in English (English).Hi there,
as the most of the community know, that tools as Mobile Atlas Creator or JTileDownloader stressing our servers, since long time ago. But now the count of connections get's critical so the English Universities that currently honestly host our project, say we have to find a solution. To much connections stressing even their hardware (servers, firewalls, routers,...) that is system critical for them. So if nothing happened, OSM has to pay for commercial hosting...
So we started labeling the Wiki pages of the Apps, that violate the Tiles Usage Policy and the admins try to limit the tile download by finding patterns and redirecting them to blocked applications landing page.
The german channel already talk about the problem (Forums,, mailing lists) and there are a few ideas:
1. Apps should use alternative services (as Mapquest, Cloudmade, ...)
2. Apps should setup their own rendering servers
3. Apps should buy/donate for using a third party service
4. Apps should use offline-rendering
Every idea has several pros/cons but we need to point out, that the usage of OSM in that way will interrupt the work of us mappers as well, and so bring the whole project down (worst case).
So what can you personally do?
- Search the web for people asking for old versions of MOBAC etc. to surround the limitations and show them that this non social behavior endangers OSM and so the product they like to use, even in years
- If you have knowledge about setting up an OSM render stack, help updating/simplifying the documentation in the wiki
- If you are a developer, you might help delploying easy to use OSM software packages or a cross platform offline-rendering library
Comment from chillly on 6 October 2011 at 17:05
Every OSMer should contact the authors of the free-loading apps and tell them to stop causing a problem.
Any OSMer who has one these apps should give it a huge negative feedback on the App store or Market until they stop free-loading from OSM tiles.
Comment from !i! on 6 October 2011 at 18:26
Thats a good idea, but on the other hand, we want to spread out the word and let users use our map ;) So on the other side we should take the chance to make it easier for using OSM offline and with vectors.
Comment from Rovastar on 6 October 2011 at 23:29
From what I understand this is to do with the mass downloads from these apps.
They download thousands of times a normal user from the servers, all in one go.
Why not just have flood protection on the servers? Many sites have similar problems and I have had architect solutions to this before.
Also maybe we should actually look at some other hosting environments and getting donations for this (hosting companies, cloud providers to individual monetary donations to pay for more bandwidth/capacity,etc).
Should we also consider more that for many OSM is the rendered map rather than just the db and work out how to do better provision.
Comment from !i! on 7 October 2011 at 00:31
Well AFAIK Flood protection isn't that easy because esp. the older versions doesn't have a valid and unique http refer or user agent string, so it's pretty hard to identify them. I guess if the amount of connections even kill university routers, there is no easy way to use more intelligent techniques like Intrusion-Detection-Systems use. But I'm definitly not very familar with system administration, so that questions might get answered by Firefishy or other 'seniors' with more experience.
Currently we seem not to have the money to get a commercial hosting. Even there is the scalation problem and the question if the project itself should (and of course could) serve tiles in a similar amount as Google Maps does.
But again, this couldn't be the next but maybe the third or fourth step to solve the problem ;)
Comment from Chaos99 on 7 October 2011 at 05:51
Well, from a conceptual viewpoint I would say, telling those developers to stop using the map is a very wrong thing. And not only because we can't enforce this properly.
Providing a usable map is our main goal. Providing the map data in ways not possible with i.e. Google maps is on of our key features. It's what draws people to OSM, it's what fuels the steady stream of developers using OSM. It's just great. If I weren't able to actually use the map I help to create, I would surely stop contributing to it too. And telling OSM user to go and harass app developers will surely not do us a favor. It's the apps the non-mappers see the map through. They are kind of our window to the world. (Nobody with a mobile phone will ever use the websites slippy map.)
Of course we have to find technical solutions to the problem at hand. But saying 'No' isn't a solution. Making it easier to do live rendering would be a big step. (I personally tried to do this for an own application, but didn't succeeded with the wiki alone.)
Sadly it's out of the question for mobile applications. Spreading those requests on multiple services (Cloudmade) could also be done on server-side.
(Or at least providing a API call with a list of alternatives and the server with the least load.)
Also, as was stated before, most of the pressure comes from mass downloads, like creating offline tile maps of entire regions. Again a feature I won't want to miss. What about creating a service to deliver those big chunks on request with a scheduled renderer/packager? So tools don't have to request tile after tile from the live map, but could request a bounding box and get the tile data as a big package after some minutes?
Comment from wieland on 7 October 2011 at 06:02
This is normal because OSM is growing and not only for editing but also for viewing. Blocking mobile apps just because they can be used for mass downloading is stupit. I used openmaps somedays ago and tried to load around 10 tiles and was blocked. So I had to switch to the mobile brower and loaded much more.
Comment from kerosin on 7 October 2011 at 07:41
Could p2p be a solution for map tile sharing and spreading, so that the traffic is borne by the community?
Comment from Richard on 7 October 2011 at 07:42
If you want to provide unlimited downloading for mobile apps, go for it. Find some sponsorship from somewhere, get some servers, and start running a tile service. Call it osmtileserver.org. I'm sure it'd be hugely popular.
The current (unpaid, volunteer) sysadmins, however, are kept fully stretched by maintaining OSM for editors like you or I, and don't have the resources to run free servers for the benefit of people charging £4 per app in the Android app store. Nor do the places from whom we rent the bandwidth permit it.
Comment from Chaos99 on 7 October 2011 at 09:44
@Richard I never said continuing as it is now is an option. I just said that banning external apps is none either.
That's more in the spirit: "First version of #Ubuntu #OpenStreetMap tile-server packages available! (German-HowTo with 5 steps) http://bit.ly/nD9xae - thx Kai!" just tweeted by Pascal Neis
And even if you make it sound as it is an impossible option: Yes, setting up my own tile server is what I intend to do, once there are the tools for it and I'm no longer too stupid to use them. You are right that I won't be able to support a lot of other users, but every one counts, doesn't it?
'[M]aintaining OSM for editors like you or I' is a concept that baffles me. OSM is for the editors? I always thought it's for the users?
Has anyone tried to contact the App developers yet? I know one 4Euro Android app. I'll ask what the developer has to say about it. I'll report back.
Comment from wieland on 7 October 2011 at 10:17
It will be difficult to find a solution, if everything is mixed up.
I have never paid 4EUR for an OSM-app. But in case I meet somebody who sells it, I will ask him to give his app out for free. The problem is, that in this case more people will use the mobile app and you will have more traffic.
Do you think OSM should get part of the money?
Why don't you ask for donations for the traffic problem?
Why is a free app better than a paid app?
For me it's better, because I don't have to pay, but for you?
For most people it is not obvious, that editing OSM is allowed but viewing is not allowed?
Is is ok, if I send a friend a permalink, so I can see where I live or where I have been or where some interesting location is? My friend is no editor.
But he will use his browser and not his mobile, so it's ok.
I am quite sure Google has more than one tileserver in more than one location.
Maybe OSM should look for tilemirrors. Which means also more traffic at first.
Comment from Rovastar on 7 October 2011 at 10:28
Most people (normal users, devs) are going to use the osm site for the tiles even if another is available. This will be more so if there are no restrictions on use for this. People will use it unless we actively try and stop them then abuse will continue.
Sadly with no sysadmin group I miss out on the discussions for this and the chance to input and I would volunteer to help out with this team if needed.
I am unsure where the bottleneck is (previously I thought it was the servers themselves now it implies it is the network infrastructure).
For an abuse situation if it is load like I understood before an offline map program would try and download enitre large cities or countries on all zoom levels. This is understandable a lot of content all in one go and what I understood was/is the problem. Maybe this is a different issue now.
But is the problem serving content or just the requests even if they fail?
Is it these hogging apps getting through sometimes and then causing issues with excessive processing and bandwidth or if they fail at the server level (so the same amount of the request are routed through the servers but less processing and bandwidth is used to send a rejected response)? Maybe it is just the sheer volume of traffic irrespective of the server response.
But to stop requested being processed by user agent, referrers is not the most intelligent way as these can easily be spoofed to use generic ones. I would look at stopping them based on the pattern of the downloads per session (or IP is that is not possible). This a distinct pattern; all tiles next to each other, massive amounts of consecutive tiles.
Re-architecture might be required too.
There are a few options.
For example,
Changing the tile location and only having access to it via a logged on source.
OSM site will have a logon to all traffic routed to the tiles servers as an app tier.
All other apps that want to use it have to do so via logging in by a unique to that app username (silently in the background). If there app/user is not on the list they cannot get in. Or too much abuse then the user (therefore app) access rights is removed.
True, it sounds more like a commercial (and controlling) model for this but something you need to consider when a web project goes from a hobby site to a bigger used setup....and once you have that then you can consider different charging options for different apps..but for a open source project that far too capitalist for many here. :)
Comment from Chaos99 on 7 October 2011 at 11:35
Can't find anything wrong about the '4Euro App'. It clearly identifies itself with a valid User-agent string, it has a download limit in place, it offers alternate servers and the developer even donates large amounts to OSM. Also, as this doesn't get mentioned here, there is a free version of the app. You don't pay for the access to OSM data, that's all the same in the free version.
Seems like nobody talks to the author, and when, not in a very nice tone.
He actually is very cooperative. Why not find a solution together, instead of arguing and blocking without a notice to the dev?
Comment from Firefishy on 7 October 2011 at 12:09
Hi,
I'm one of the volunteer sysadmin...
We have 4 servers running tiles. (3 live and 1 fallover "static" backup)
"Flood Protection": We already have use a QOS based rate limiting system. Clients that do mass downloading of tiles are slowed down after downloading a few megabytes.
Examples: MOBAC alone is responsible for around 73 tile request per second and 42GB of traffic per day.
Comment from wieland on 7 October 2011 at 13:19
@Firefishy: is 73 tile request per second the peak or average?
How much in percent is 73 out of all tile requests?
Do you mean 73 tile requests per second from one user (one IP)?
Is it possible to lower the limits?
Comment from wieland on 7 October 2011 at 13:20
@Firefishy: And thanks for your work :-)
Comment from chriscf on 8 October 2011 at 00:13
Maybe we can serve rogue downloaders the equivalent of OpenWhateverMap?
Comment from Chaos99 on 8 October 2011 at 08:41
What about making the current tile policy available via an API call? So any app can ask how much tiles and at what rate it is allowed to load. This gives OSM the possibility for fine grain control, even in real time, should the servers face a hard limit.
Any good behaving app will listen to those constraints, any apps who don't are really worth blocking.
Just remember: its not 'MOBAC' creating this load, and it's not 'MOBAC users', it's 'OSM users, who come via MOBAC'. We can't give more than we have, but they are just the same as the 'website users'.
Comment from Firefishy on 8 October 2011 at 12:53
@wieland 73 tiles per second is the average over 7 days. All IPs combined.
Avg overall rate is somewhere between 800 and 1200 tiles per second.
Comment from Rovastar on 8 October 2011 at 13:45
Ok so less then 10% so it is really not that much.
I thought a few years it was a much bigger percentage and therefore a bigger problem. Reducing this doesn't sound like a massive improvement to your web infrastructure - if it is an capacity issue now then you/we have bigger problem than a offline map downloader.
Comment from !i! on 8 October 2011 at 14:13
The question is if all apps use a appropiate user-agent. I don't think so if I view the logs:
http://grant.dev.openstreetmap.org/tmp/logs/03102011/yevaud-UA-hits-1week-20k.txt
BTW there is an update on some tools fixing fixing some aspects:
Locus: http://forum.asamm.cz/viewtopic.php?f=10&t=1166&p=6874#p6870
OpenMaps: http://lists.openstreetmap.org/pipermail/talk/2011-October/060336.html
NaviComputer wrote me a mail pointing that he did some modifications already
@Firefishy please tell us if we can remove the labels on the wiki pages :)
Comment from wieland on 8 October 2011 at 15:32
Less than 10% is not really much, but still too much.
Even if MOBAC had it's own tileserver, the network problem is not solved.
Just wait a month or two and 1000 new users and it will be the same level.
I think the idea to send diffent UA for viewing and for bulk download is a good idea from openmaps (see http://lists.openstreetmap.org/pipermail/talk/2011-October/060336.html )
Comment from dcp on 8 October 2011 at 16:07
Eliminating redundant nodes with a global application tool would help to alleviate the problem. I have just done a trial on five ways and with the help of the JOSM "Simplify Way" tool did the following analysis:
way 1 1977 reduced by 488 nodes
way 2 1920 reduced by 723 nodes
way 3 1210 reduced by 363 nodes
way 4 1155 reduced by 494 nodes
way 5 846 reduced by 407 nodes
Total 7108 reduced by 3632 nodes i.e. a reduction of approx. 51% of the nodes.
I am not saying that this will decrease traffic by 51% but it would do its little bit.
Comment from chriscf on 8 October 2011 at 18:33
I would think we could regard the "Mozilla" UA as fake, since any such UA always includes a version, typically "Mozilla/5.0" if the browser is up to much.
Comment from amm on 8 October 2011 at 18:49
10% for MOBAC might not sound that much, but then take 5 of those apps (e.g. NaviComputer, OpenMaps,...) and you are already at 50% of total traffic. Also don't forget, it is only that low because of all the throttling mechanisms that are already in place.
Of cause, it would be nice to offer more tiles to various mobil apps, as it is one of the main ways for people to use and benefit from OSM, however, it is just not feasible with the current hosting and donations.
The admins have already been fairly lenient with what they allow and have pushed the possibilities to a maximum by only starting to block scrapers when it became necessary due to resource constraints.
As we are more and more hitting these limits, we need to find more sustainable ways to use to use the data without inconveniencing the end users. But hopefully new apps will be developed to meet those needs.
Comment from wieland on 8 October 2011 at 21:22
@dcp: I will decrease the tile traffic by 0% maybe 0.1%. The discussion is about the tiles and not the size of the database or api-traffic.
@amm: and without throttling MOBAC would be 50% and together with the other evil apps it will be 250% and that much will kill any hardware. :-)
Comment from Rovastar on 9 October 2011 at 11:06
Thanks a lot for the stats you provided.
The list for those interested was the first 20,000 user agents (so not all the traffic) and a total of 173,184,462 requests.
A brief breakdown (I would expect these percentages to reduce a little if we had the over total of requests)
MOBAC/* 15.8x% (20+ UA)
JTileDownloader 0.50% (4 UA)
NaviComputer 6.4x% (1 UA)
OpenMaps 5.6x% (20+ UA)
Locus 4.7x% (20+ UA)
omaps 2.6x% (20+ UA)
OSMand 1.1x% (10+ UA)
People complain about JDownloadtiler with 0.5% when something called JOSM has 400+ UAs and more than double the downloads at 1.1x%
And from the corps
Google
GoogleEarth/* 0.5% (nearly 100 UA)
ERSI mapping company
ArcGIS 0.9x% (30+ UA)
I hope that adds some more facts to this debate. I haven't covered everything just ones that were mentioned and which I thought were of interest.
amn,
So all the different mass downloaders don't take up all 50% of the traffic. True it may/will be more but flood protection is normal and many large sites we are not unique here. Lets get more aggressive with it then. I think the real problem is not people that want to say download a small area offline it is those that try and download whole countries, cities, etc.
What are end users here? Users of mobile apps? They are the most useful for me as a user for OSM data having a map on my mobile with GPS to locate/track where I am. This is the modern way people us the map. We need to think of ways of accommodating these people. Also I actually do like to be able to download maps on my phone for areas where sometimes there is not 3g signal available but GPS works. e.g. going walking no 3G signal, GPS works would like to download maps beforehand.
The questions of strategy and architecture of tile usage image will not be even tackled here as they are difficult questions. If OSM doesn't provide tile map data the project will die.
Comment from Rovastar on 9 October 2011 at 12:15
I thought I would add if anyone is even listening that OpenMaps (5.6x%) claim never to have been aware that they might be abusing the tile servers (did anyone even tell them *shrug*) but are looking to fix this in the next version.
http://blog.izeize.com/2011/10/openmaps-app-blocked-by-openstreetmap.html
Comment from chriscf on 9 October 2011 at 18:52
"People complain about JDownloadtiler with 0.5% when something called JOSM has 400+ UAs and more than double the downloads at 1.1x%"
The important difference being that JOSM is used by active contributors. OSM's primary product is data - the rendered tiles are just a convenient demonstration.
Comment from !i! on 9 October 2011 at 19:38
Well let me just say a few words, as the starter of this thread.
I understand both sides but all in all we doesn't have the resources (servers, traffic, admins, ...) within this project, to deal with the tile problem alone. Besides there is a 'political' problem if the OSMF would offer a commercial premium service.
So what might come up is a project (or company) just dealing with tiles deployment, as pointed already. But this is in future far far away....
What counts is what we all can do NOW.
- improving the DOCs about the rendering stack -> more people will use their own servers
- make it easier to setup the stack (as Kai already does by offering Debian packages)
- assist on developing a Vector based offline rendering lib for Apps
Comment from menion asamm on 10 October 2011 at 09:11
Hi Guys,
I wanna put also some words to this discussion. I'm author of Locus app. As you can see on stats (4.7% of server request), even when app have all requirements from Tile usage policy page, number of users (which is really high) cause such high usage.
I in Saturday released new version 1.13.4, that have completely BLOCKED ability to download data from OSM, allow only viewing and caching tiles. So please, anyone who have access to some user agents stats, check what do Locus after this update
about Vector maps rendering. Exist very nice project MapsForge, that have public library for vector map rendering that is very easy to implement, so many apps may use this. I have own server that generate and distribute vector maps for locus so I hope that usage on OSM server will be low and low.
Also I have today meeting with one friend and we'll probably start creating own tile server for Locus, so at the end, no traffic on OSM servers should be. Anyway I suggest you to do some protection steps to prevent such stupid naive developers like me was, to use your servers without warning and some limiting (locus caused troubles like mobac few months ago)
Comment from !i! on 10 October 2011 at 12:18
Hi menion, thanks for your reply :) Oh and of course for the fast steps to limit our problems.
Convercing the prevention of new 'naive' Devs, what does you suggest?
To me it's always a question of logic, that if I use stuff that is for free, that I look for any policy/readme/license/... and try to get in contact with the project to get hints for important stuff. But this process can't be enforced without using limited access as developer keys (that I personaly dislike) or?
Comment from menion asamm on 11 October 2011 at 06:04
you're welcome!
"To me it's always a question of logic, that if I use stuff that is for free, that I look for any policy/readme/license/" - yes for me too, now ..., but look at me before an year, I was ignoring this (true is that before year, Locus used 1000 users so it wasn't such problem and the grow was in Q1 2011 really rapid).
Anyway I think that access thanks to some unique key shouldn't a problem to devs. It should be simple process that allow you to control number of apps that use OSM data and also have contact to them. Anyway I have no experience with this, but for me personally ... when I now add any new map, I always firstly contact owner and discuss with him, so I really don't see a problem in need to firstly contact anyone from OSM community to get access to maps directly from openstreetmap.org server
Comment from !i! on 11 October 2011 at 06:16
Well managing Dev keys is another task that needs people to add this feature and to keep it running. Have a look at our website, there is so much stuff to do but we doesn't have that much developers to do the job. By the way this would break all current third party usage :(
Mhh...even for open project it's clear that they have limited ressources... Would you start scrapening Wikipedia without any talks to the Admins (who would point you to some dumps) ? ;)
Comment from menion asamm on 11 October 2011 at 09:06
perfectly understand Matthias, I also do not see simple working solution that will not break current working solution. Maybe just put big orange or red rectangle to Tile usage policy page, that describe problematic little bit ... better.
Don't know .. anyway wish you good luck and hope that Locus will not cause any new troubles. I have just ordered new server (unfortunately it will be available after around 10 days), so I hope that till one month, I'll have ready working tile server and will be able to cut off Locus completely from main osm server!
Comment from menion asamm on 13 October 2011 at 14:40
Matthias,
don't you have some new stats about using of Locus on OSM servers? I'm just asking because for layers 17 and 18 is Locus still blocked, so if there will be any possibility to unblock it for new version (1.13.4 and higher) which have disable ability to download
Comment from Firefishy on 24 October 2011 at 15:12
Update stats: http://grant.dev.openstreetmap.org/tmp/logs/24102011/
Comment from !i! on 24 October 2011 at 21:08
Anybody willing to create some nice charts, so we all can see the trends?
Comment from Firefishy on 27 October 2011 at 11:15
Please do not contact the websites from there referer list. They are not a problem.
Comment from !i! on 27 October 2011 at 13:28
Sorry was my fault :(
Comment from !i! on 29 October 2011 at 11:58
Ok I did some visualisations, maybe that this will help for discussions http://www.openstreetmap.org/user/!i!/diary/15190