
mmd's Diary
Recent diary entries
For the first time in many years, I decided to repeat a series of planet imports using the Overpass API. As I didn’t want to spend too much money on hardware, I rented a commodity Intel machine for about 40€/month, with 2 Data center SSDs and 64GB ECC RAM.
Much to my surprise, the import on release 0.7.58.5 took about 33.5 hours, which is at least 10 hours longer than I expected. I was able to improve runtime a bit by tweaking a few configuration settings, such as enabling lz4 compression everywhere and increasing a chunk size parameter, but still ended up with 26.5 hours.
I continued testing with my own experimental Overpass fork, that includes support for PBF, multithreading, and many other changes under the hood. Initial measurements looked quite promising with 10.5 hours total runtime. After some further analysis and improving some data structures, the import took 7 hours and 23 minutes. Peak memory consumption was still quite ok at 22G. I tried different settings to achieve lower memory consumption, at the cost of longer processing time (e.g. 8 hours and 13G peak memory).
Depending on compression settings, the final planet database was in a range of 230-265GB.
Detailed results are available on this wiki page: https://wiki.openstreetmap.org/wiki/User:Mmd/Planet_import_challenge_22
That’s all for today.
In this blog post we want to count addr:housenumber nodes across all EU member states, broken down by country. That’s a bit above 34 million nodes in total.
Queries are tested on different Overpass instances: two rather busy public ones, one idle private instance and a dev instance. With the exception of the latter, queries run on offical Overpass releases. Database contains a full planet (including metadata and history).
Query: https://overpass-turbo.eu/s/1adO
[out:csv(name, count)];
rel(2668952);rel(r);
map_to_area -> .areas;
foreach.areas->.area (
node["addr:housenumber"](area.area);
make data count = count(nodes),
name = area.set(t["name"]);
out;
);
Result: https://gist.github.com/mmd-osm/5219028c9fa93db17ac7c271a99183a1
Runtimes
kumi.systems: gateway timeout after 1h
overpass-api.de: gateway timeout after 2h
private server: timeout after >2h
dev branch: 30s
NB: dev branch is exploring alternative implementation options and is currently not released.
CGImap 0.8.0 released
Posted by mmd on 16 February 2020 in English (English). Last updated on 14 April 2023.A new version of CGImap, the rails port acceleration layer written in C++, has been released!
Full release notes: https://github.com/zerebubuth/openstreetmap-cgimap/releases/tag/v0.8.0
A little experiment...
Posted by mmd on 15 December 2019 in English (English). Last updated on 14 April 2023.🧙✨
Changeset upload is in production since May 2019, as part of CGImap 0.7.5
Posted by mmd on 4 July 2018 in English (English). Last updated on 14 April 2023.Download along my own virtual way in Overpass API
Posted by mmd on 12 August 2017 in English (English). Last updated on 2 May 2018.THIS FEATURE IS NOW OFFICIALLY AVAILABLE IN OVERPASS API 0.7.55
Again a bit of prototyping today.
I always wondered what the best way to download all objects along some GPX track (more generally a list of lat/lon pairs) in Overpass API might be.
Usually, you don’t want to create some complex poly-filter for this, or even worse upload your gpx track as OSM way just for the sake of finding out what’s close by.
You might have come across the around
filter, which comes in handy to find objects around a center point. I have extended this idea a bit to also handle linestrings.
Here’s a first result: a turquoise circle in the middle is our great made up GPX track, consisting of about 20 points. All the yellow stuff in the background is what Overpass API returned as highways up to 500m away from our circle. Obviously, that circle doesn’t exist in OSM, so we’re really looking at what’s close to our virtual way.
Circle
Live demo: http://overpass-turbo.eu/s/yt2
Road trip
Imagine you want to create a map with only minimum features (motorways, some important nodes), and some place names along your own lat/lon pairs. Here’s an example for such a trip from Frankfurt to Basel, downloaded as 10MB PBF file, processed via osm2pgsql and rendered by Kosmtik:
Live demo: http://overpass-turbo.eu/s/qXI
Github issue: https://github.com/drolbr/Overpass-API/issues/418
(Clickable roads demo is no longer available.)
Posted by mmd on 7 April 2017 in English (English). Last updated on 14 April 2023.(Clickable roads demo is no longer available.)
[demo] Emojis in overpass turbo!! 😍😈😃
Posted by mmd on 7 January 2017 in English (English). Last updated on 10 January 2017.Emojis are only a fingertip away on your iPhone… it was only a matter of time until they show up on OSM as well. The following query finds all those emojis - worldwide: http://overpass-turbo.eu/s/l6r
Thanks to tyr_asd for the MapCSS enhanced version below
A bit more serious: how about searching for mixed language names like name=Vinh-Loi Asien-Supermarkt 榮利市場
: check out the following query! http://overpass-turbo.eu/s/l6s
Do you find regular expressions with full Unicode support useful and want to see them in one of the next official Overpass API releases? It’s your turn now, vote for it on Github or leave a comment here!
(demo no longer available)
Posted by mmd on 30 September 2016 in English (English). Last updated on 14 April 2023.(demo no longer available)