OpenStreetMap

Bounty for rendering queue fix

Posted by Komяpa on 1 December 2017 in English (English)

OpenStreetMap.org tile rendering operates on edge of its capacity. It's possible to get more from already existing cluster if rendering queue is not continuously dropped, but instead is preserved and processed during low load hours.

Ttile.OpenStreetMap.org uses mod_tile and renderd to manage rendering queue. mod_tile is currently a core ecosystem project lacking an active maintainer for years. There are currently 19 Pull Requests that nobody merges.

I tweaked values in source code to allow for a longer queue. Pull request is available at https://github.com/openstreetmap/mod_tile/pull/152. In comments there was a proposal to make it a configurable parameter.

I offer 50 USD to person who manages to:

  • create pull request to mod_tile/renderd that makes queue depth a configurable parameter, and set defaults to preserve the queue to shift day's load to night;

  • get it depolyed to tile.openstreetmap.org adjusted accordingly.

Shift should be visible on https://munin.openstreetmap.org/openstreetmap/render.openstreetmap/renderd_queue.html graphs, most likely by Dirty queue growing beyond 1000 tiles per server, and being non-empty for the whole day, shortening during the nighttime.

This should make the map browsing experience better for everyone.

stat

Comment from mmd on 4 December 2017 at 18:22

Is this bounty is still relevant in its current form?

As we have found out in the meantime, the proper solution would be much more complex than just exposing the queue depth as a configurable parameter.

Anyone considering this topic should take a very close look at the detailed analysis posted by @apmon first: https://github.com/openstreetmap/mod_tile/pull/152#issuecomment-348805403

Comment from Komяpa on 5 December 2017 at 10:10

Hey mmd,

from comment of @apmon it seems that to get it working with bigger queue length it is essential to handle the case of re-requesting a tile already in queue. At very least, it can be done via checking only for presence of tile in non-dirty queues on request, and not spilling it twice to dirty queue. Of course there's also a way to do it better.

As I understand that this is harder task, I increase size of bounty to 100 USD.

Login to leave a comment