OpenMapTiles planet-scale vector tile debugging at low zoomPosted by ZeLonewolf on 10 January 2023 in English (English). Last updated on 11 January 2023.
The Americana vector style uses OpenMapTiles as its backing data schema. When the project desires to add a new feature that isn’t available in OpenMapTiles, someone from the team typically submits a PR to add it. Eventually, OpenMapTiles will create a release, which gets picked up by the Planetiler OpenMapTiles profile, after which I would re-render the planet on an AWS instance. This process from end-to-end often takes months before we see the results at planet scale.
Because planetiler’s update cycle follows OpenMapTiles, contributors need to use the older openmaptiles-tools, which can take days, weeks, or even months to render a planet, depending on how powerful the developer’s computer is.
Therefore, when testing a change to OpenMapTiles, a contributor would typically test their changes on a small area, with a command like:
This command would download a PBF extract from Geofabrik, and run a series of scripts that ultimately produce an .mbtiles file of Rhode Island. If you’re testing a feature that appears at high zoom, you can edit
.env and change the setting to render down to the maximum zoom of 14. Because Rhode Island is so small, a full-depth render only takes a few minutes.
However, what if you are testing a low zoom feature like an ocean or sea label? If you need to test whether the Atlantic Ocean label is rendering properly, there is no extract short of the planet that will contain an ocean.
The solution for developers working with these features is to download the planet file, and then pre-filter it using the tags-filter feature in osmium tool for just the features that you care about testing at low zoom, and then render that into tiles.
First, you download the planet pbf file:
AREA=planet make download
This will download a file
planet.osm.pbf into the
Next, run osmium to filter the planet file. In my case, I wanted boundaries, protected areas, places, water features, rivers/canals, top-level highways, and US/Canada route information. The command below filters produces a new file called
slim-planet.osm.pbf with just those features:
osmium tags-filter -v -O -o slim-planet.osm.pbf planet.osm.pbf r/boundary=administrative,protected_area
Next, you can replace your original
planet.osm.pbf with the slim one that you just created, and run
./quickstart.sh planet. The script will detect that you already have a planet file and proceed on. The size difference is dramatic! The full planet file is 67GB, while the filtered extract is just 6.6GB. While this is still “big”, it’s now down to the size where openmaptiles-tools is able to process it on a typical developer’s high-end laptop. And of course, if you care about fewer features, you can create an even slimmer extract.
I then started up my vector tile server with
make start-tileserver, pointed my Americana style to it, and started browsing the map at planet-level zoom. It was then that I saw there was a bug that I needed to fix:
The difference in style between “Gulf of Bothnia” and “Baltic Sea” was due to an SQL error that I had introduced in my PR. I quickly fixed the bug, the PR got merged a few days later, and now I sit confidently knowing that low-zoom seas will render properly after the next release of OpenMapTiles.
Comment from zstadler on 11 January 2023 at 09:54
How do you create the definition for
osmium tags-filter? Is there a utility that takes a
mapping.yamlfile of an OpenMapTiles layer and extracts the OSM tags it needs?
Comment from stephan75 on 11 January 2023 at 12:57
I would ask myself: is the gol-tool of https://www.geodesk.com/ able to do the filtering you have mentioned that has been done so far by osmium?
Are there any differences in memory or time consumptions?
Comment from ZeLonewolf on 11 January 2023 at 14:22
@zstadler, I don’t believe there’s a utility to pre-filter a PBF based on an OpenMapTiles yaml mapping, but that’s certainly a cool idea. The definitions I listed in this diary entry represented just the features that I wished to render for testing.
@stephan75 I’m not familiar with gol-tool but I would welcome such a comparison!
Comment from zstadler on 12 January 2023 at 08:40
How long did it take
AREA=planet make downloadto complete?
Comment from ZeLonewolf on 12 January 2023 at 14:04
Download speeds of course will vary depending on the quality of your Internet connection. For me, it was a 43 minute download on my home wifi.