At 09:45 UTC the OSM sites are back up again after ‘too many people accessing’ notices for ~45 minutes. Do we have some info on what was happening, please?

The Existential Question

08:45 5 November
Having slept on this I’ve come to realise that it is far more important than at first I realised, since it concerns the very existence of OSM: in what form is it going to continue? (all life on this planet grows, withers or transmutes; what is OSM going to do?).

In short (there is fuller info with links at bottom):

1. 28 Oct: zerebubuth (Matt Amos) opened a discussion on Github : the tile-servers are hitting capacity; what policy is OSM going to follow in the future? Restrict to open-source apps or restrict to publicly-accessible sites (notice that both options involve denial of access)?
    
   There are 59 machines internationally — including 20 globally distributed tile-caches — serving up OSM tiles to anybody that asks for them. Funded entirely by donations. Outbound peak traffic on tile.openstreetmap.org (served from cache) peaks at 1 Gbit/sec (6,800 GB/day; 528 million requests at 13.6 kB/request), but a recent measurement indicates that just 11% of this is supplied to OSM websites. The rest is 3rd party sites and apps.
    
2. 4 Nov: the system seized up tight.
    
   ironbelly had to be restarted (look at ‘uptime’) and re-init (at a guess) on the ramoth network (look at ‘memory’ and also ‘network’ - the server system became saturated with FIN_WAIT & TIME_WAIT connections for all of 2 Nov, possibly indicating TIME-WAIT loading problems)
    
3. As soon as you provide something good at zero cost there is large demand. How is OSM going to respond? 

Added at 24:00 4 November:

Zero assistance from OSM Folk, so had to find an answer myself. Took a long while.

Found it on the Munin OSM site. In particular: a critical error where the Chef node status on ironbelly went AWOL between 18:00 Thu 3 Oct & 09:45 Fri 4 Oct (whoops). In addition, ironbelly + ramoth disks show a gap at ~9am; presumably the server was stopped, fixed & restarted with a 45 minute gap as I experienced this morning. As further proof of an issue, the Squid Client Requests began to show gaps in reporting on 31 October & following days then reporting died completely on Thursday & has not resumed.

Of course, the important point is why & not what (that has also been my point to Harry in the comments). The answer is probably here, as discussed on github.

PS
The tile-server peaks at >1 Gbits/sec outbound serving from cache. Phew!

No direct Connection between this recent Fail & the Tileserver Issue

Andy Allan has left a comment pointing out that there is zero connection between the recent issues & Matt’s discussion.

Andy actually works on the physical servers & therefore knows a touch more about them than me. He points out that what I detailed above were two separate issues, and that each is unconnected with the tileserver discussion:–

1. Extra Disk Drives for ironbelly
   Fri 4 Nov: The extra load from the subsequent RAID rebuild caused ironbelly to fail. ironbelly provides services that other servers depend upon, including the servers that power the website and the editing API (which each also failed). The server guys had to make some config changes so that ironbelly continued to provide essential services whilst the RAID build continued.
2. DB Server Index Rebuild
   Wed 2 Nov: The databases are arranged in primary (katla) & slave fashion; the index was rebuilt on katla (smaller, faster) and the vast network traffic are those indexes being transmitted out to the slaves (such as ramoth).