OpenStreetMap
How many days or weeks does it take to upload the whole planet via Osmosis ?
Posted by baditaflorin on 15 September 2015 in English.I am now in day 6 of waiting
There is a 16 GB RAM server, octacore environment, running ubuntu 14.10 ( don`t ask why )
I have a 317 GB file generated called copy********n
And there is a file called copy******w that is still being generated, growing by 10-20 GB per day , now is at 76 GB
And a file called copy*******wn at 30 GB
i am guessing that the n means nodes and w means ways, but how much should it grow, procentualy compared with the 317 GB n file ?
The command that i had used to give the command looks like this
sudo osmosis –rbf latest-planet.osm.pbf –wp host=localhost database=mydatabase user=*** password=****
Discussion
Comment from Warin61 on 15 September 2015 at 06:57
Your probably on the lowest priority. Why? Probably the reason for your load is unknown. Try contacting the administrator of the site, have a justification for your load on the site ready for them .. not here.
Do you really really need to whole planet? Already tested what you want to do with a small sample? These are the kind of questions I’d have answers ready.
Comment from baditaflorin on 15 September 2015 at 08:03
I have access to the server, what should i change / could i change
I want the whole planet, so i can do some statistics for the whole planet, not just a country or a continent.
top - 11:00:23 up 62 days, 15:27, 7 users, load average: 2.03, 2.07, 2.10 Tasks: 305 total, 2 running, 280 sleeping, 23 stopped, 0 zombie %Cpu(s): 3.5 us, 9.6 sy, 0.0 ni, 73.7 id, 13.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 16341788 total, 16174748 used, 167040 free, 25816 buffers KiB Swap: 16684028 total, 16521344 used, 162684 free. 9697612 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21798 postgres 20 0 369660 190580 144392 R 0.0 1.2 16935:45 postgres 22127 postgres 20 0 295484 144512 144468 S 0.0 0.9 600:18.35 postgres
Comment from Jean-Marc Liotier on 15 September 2015 at 08:13
sudo ? Why ?
Comment from baditaflorin on 15 September 2015 at 08:16
@jean osmosis have a bug and it does not work to process a file and upload it to postgres without the sudo command.
Comment from Vincent de Phily on 15 September 2015 at 08:41
I can’t imagine an osmosis bug that requires root to solve. Can you psql to the db ? Fix you pg_hba.conf if not. Can you write to temp files that osmisis creates ? Use chmod/chown if not.
16GB of ram is at the lower end of usable nowadays. And if you aren’t running this on an SSD, you should probably forget about importing the whole world, it’ll take months.
Always start with a smaller extract, not the whole world. Try Africa first, for example. You say you want to run stats, could you run stats on individual continent databases ?
The biggest speed improvement you can get is by osmfilter-ing out data you don’t need before the import. Do you really need building outlines for example ?
Comment from baditaflorin on 15 September 2015 at 08:52
i want to do statistics about everything that is on the planet file, from every piece of the world.
I know that i could just import each continent and then do some join to stitch all together, but then whats the point of a planet file.
I already Have Africa loaded, some more countries loaded, but the aim of my statistics will be to show everything that is on the map, how it relates, what particularities exists in different parts of the world
Comment from baditaflorin on 15 September 2015 at 15:23
I will kill the task and try 2 different version :
1 imposm3
the only information that found says that i still have to wait at least 7 more days http://ksmapper.blogspot.ro/2011/04/parse-planet-into-pgsnapshot-database.html
I think i got an answer, based on
If the copy*****n it`s 317 GB
What is the approximate size of copy****w 80 % ( 240~ GB ) copy*****wn 10-20 % (50-60 GB)
Comment from pnorman on 16 September 2015 at 01:18
What area did you use for testing your command, and how long did it take?
Did you load in either the linestring or bbox columns?
What speed drives do you have?
Comment from species on 24 September 2015 at 06:56
I see the load is only 2, which should be at least 8 on an octa-core. And your Mem and Swap are both 100% full… More interesting than “top” would be the output of “iotop” - I guess its your I/O that’s limiting you.
Do you have an SSD? Without that, forget importing the whole planet^^. Africa may has worked because the DB fitted (at least important parts) inside your 16GB RAM. You need either enough RAM (512G?) or an SSD for OSM planet imports :-)