For the first time in many years, I decided to repeat a series of planet imports using the Overpass API. As I didn’t want to spend too much money on hardware, I rented a commodity Intel machine for about 40€/month, with 2 Data center SSDs and 64GB ECC RAM.
Much to my surprise, the import on release 0.7.58.5 took about 33.5 hours, which is at least 10 hours longer than I expected. I was able to improve runtime a bit by tweaking a few configuration settings, such as enabling lz4 compression everywhere and increasing a chunk size parameter, but still ended up with 26.5 hours.
I continued testing with my own experimental Overpass fork, that includes support for PBF, multithreading, and many other changes under the hood. Initial measurements looked quite promising with 10.5 hours total runtime. After some further analysis and improving some data structures, the import took 7 hours and 23 minutes. Peak memory consumption was still quite ok at 22G. I tried different settings to achieve lower memory consumption, at the cost of longer processing time (e.g. 8 hours and 13G peak memory).
Depending on compression settings, the final planet database was in a range of 230-265GB.
Detailed results are available on this wiki page: https://wiki.openstreetmap.org/wiki/User:Mmd/Planet_import_challenge_22
That’s all for today.