Recent diary entries
When you want to do a import, and you start with a SHP file, there are 3 challenges that i had identified :
How do you convert and split the file, in such a way that a way will not be longer then 2000 points ? How do you remove the line duplicates after you explode the file ( at first all of the municipality is a polygon, that means that at common borders, there will actually be 2 ways.
How do you elevate the properties of the shp file, now the information is on the way, and you want to move this information into the relation.
I only get the data per continents, for some types of roads. for europe i only got them for 6 types, because osmosis is not able to load the whole Europe file, at least with a server that have 16 GB RAM and octacore processor. I had a deadline, and from my estimation would have took more then 2 days to load the whole europe, so i had used osmfilter and osmconvert to extract different types of highway types
That is, the average number of tags that each highway type have, per continents.
There are 12 ways that have over 200 tags attached to them in asia
you can see them in overpass-turbo using this link http://overpass-turbo.eu/s/bAD
Thanks sandert17 for the suggestion to use overpass-turbo links
Ever wandered how many POI over 100 tags ? Now you can know.
There are 33 nodes that have over 100 tags, mostly if not all being lighthouse
I am not able to save the list with the tags also, at least not in OpenJump ( SHP will truncate at 255, other extension does not work ) , QGIS ( Error after the forth value ) ( filled bug request ) Pgadmin (don`t know how to save directly so that i could open it
[Update] A list with all the nodes tags that have over 50 tags
My hypothesis is that we could reduce the planet file with 10-100 MB by only removing the unnecessary points that exist on the map.
I am trying to figure it out the correct postgis query to find out exactly this.
I am trying to compare the points of a linesting, and if the degrees between 3 adjacent points is 0 degrees, that means that the point in the middle can be deleted.
The only check in place that i see is to check that the point in the middle is not connected to a point and check that the hstore tag of that node is empty, meaning that there is no value added to the node ( pedestrian crossing, motorway_jucntion, exit_to, etc )
This is just a dump of some ideas and stats that you should not take for granted, the data its most of the time inaccurate , because i run 2 commands at the same time.
But anyhow, its a point of conversation First, i had filtered with osmconvert and osmfilter so i remained with a planet dump, composed of only the tag highway I am trying to import all the roads in the world, after i tried with a direct attempt and did not scale up, i had now spitted the world into each continent
Size = only the highway , saved as a osm.pbf
africa_highway ( 356 MB ) = 46 Minute to import into postgis Asia_highway ( 1318 MB ) = 213 Minute to import into postgis Australia_highway ( 259 MB ) = 14 Minute to import into postgis Central America ( 63 MB ) = 10 Minute to import into postgis
I am now in day 6 of waiting
There is a 16 GB RAM server, octacore environment, running ubuntu 14.10 ( don`t ask why )
I have a 317 GB file generated called copy********************n
And there is a file called copy**************w that is still being generated, growing by 10-20 GB per day , now is at 76 GB
And a file called copy*******************wn at 30 GB
i am guessing that the n means nodes and w means ways, but how much should it grow, procentualy compared with the 317 GB n file ?
The command that i had used to give the command looks like this
sudo osmosis --rbf latest-planet.osm.pbf --wp host=localhost database=mydatabase user=******* password=**********
Have you ever asked yourself what are the TOP 25 cities in the world, based on the number of POI in OpenStreetMap ?
To my surprise, the number 1 city is not a European one, but a city in Japan, that have 121.154 POI
I have calculated all the poi from the following category's : amenity, leisure, shop, sport, tourism, man_made, office
I did not include the historic POI, because when i filtered the world planet, i had extracted the history tag, instead of historic
I had converter the ways into polygons, and then combined the 2 data-sets into one.
Then i had counted, based on the bbox of the city, the number of POI in each of the cities. I will give here the TOP 25 cities by the number of POI
Waiting for your opinions, comments, etc
Yokohama 121154 Paris 113165 Tokio 97078 Kawasaki 90576 Rio Grande 85238 London 84681 Saitama 75783 Berlin 74002 Birmingham 65093 Moskau 61579 MĂĽnchen 54533 Essen 53935 Dusseldorf 52342 Stuttgart 50712 Madrid 48335 Vienna 47396 Toronto 47324 KĂ¶ln 45018 Dortmund 44534 Lyon 44441 Hamburg 44334 Sete 41723 Milan 41317 Osaka 40756 Frankfurt 38128
Short Legend : Bigger green means bigger version, bigger red ways means that the way contains more tags inside. The same with the points
You can download the bucharest city view from here 4A0 - 250 DPI - 180 MB file 23405px * 16555 px
In Romania, we are planning to add the relations for all the national Roads
I made this youtube video-clip to show how easy is to add a relation
Playing with Mapbox Studio and the ability to use the osm_id of a POI, i managed to make a historic map of the POI of 24 selected cities around the world
In a totally random way i will show some of the different type of cities that we have in OSM, in the terms of relevance to people that are interested in going in that city and using osm for POI
Most of the POI in Ankara were added in 2006 - 2008
Budapest - An example of a systematic, mapping party approach ( somebody with local knowledge can correct me if i am wrong )
Buenos Aires - It would be useful to see how many of the new edits are from local people, to see how sustainable is that and how much the local community will continue to add after the SOTM
Cluj Napoca Another example of the power of mapping party, for this ever-growing city, that will be in 2015 the European Capital of Youth
Frankfurt - Central, POI only around important streets, approach
L.A. - From the types of POI that where added in 2009, Los Angeles had an import of date in 2009, but except from that the city is almost dead in terms of POI growth
London - is more complete then it looks on this map. In Mapbox i took only the POI that are a node . In London there are a lot of POI that they are tagged as a building. For the sake of simplicity those tag`s are only rendered as text, with a small white dot.
It was a technical stuff, i would have to calculate the age for the POI of the ways.
Melbourne - a example of a city center POI useful city
Moskow - A beautiful example of a complete city, where you can find POI in all the city, not just in the city center
New York - except a import in 2009, in the last 2 years the city center started getting some kind of attention
Tokio - Is growing fast, in a decentralized, all over the map type
You can download all the Full Resolution maps here https://www.dropbox.com/sh/whixmqcr14n873y/AAA5cRJev87o7dcdjAnKsylda?lst
After doing the Romanian OSM node density map, i wanted to do something more globally
The map was created by exporting using Qgis Print Composer in 4 different 24806x17716px , because of the limitations of QGIS that does not allow the export of bigger files yet.
The date was obtained using overpass-api, but for simplicity, if you want to do something similar, use the europe planet extract.
There where over 7 million shop,amenity, tourism and leisure POI in Europe that where combined in a single file using the Qgis Plugin > Vector > Data Management Tools > Merge ShapeFiles to one
Using Qgis plugin mmqgis i created a network of around 800.000 hexagons, and using the Qgis plugin "count in polygon" i was able to figure it how many POI are in each hexagon. To get to a more correct answer, i calculated the area for each of the 800.000 hexagons, that is ranging from about 18 square kilometers in south of Sweden to 28 sq km in Madrid, Spain and did a POI/area to get a more exact answer, POI per sq km
TOP 200 POI by number or POI
OpenStreetMap POI density per km2
1.61 GigaPixel OpenStreetMap POI Map of Europe
Just how much of Europe have at least 1 POI per km2 ?
- 361701 km2, similar with the area or Germany, have a density of just 1 POI per km2
- 169428 km2, similar with the area or Greece, have a density 2 POI per km2
- 97592 km2, similar with the area or Hungary, have a density 3 POI per km2
- 68037 km2, similar with the area or Lithuania, have a density 4 POI per km2
- 48142 km2, similar with the area or Slovakia, have a density 5 POI per km2
- 35867 km2, similar with the area or Moldova, have a density 6 POI per km2
The total sum is 983412 km2, to put this in perspective, this means the area of Ukraine and Germany together. The rest of Europe has a density that is less then 1 POI per km2
What i learned.
- Trying to so something as big as this is impossible in QGIS < 2.4
- Except UK, Germany and neighboring county's, the rest of europe has good POI coverage only in big and medium cities
- There is still no simple way to extract all the POI of a country or a continent ( amenity, shop, leisure, tourism )
- There is no possible way to export this map in a interactive way, at least from the Qgis Plugins
It would be possbile only country by country, if somebody with coding skills could make qgis2leaf work with rule-based colors
i was using this into the overpass api
[timeout:86400]; (way(34.759984,-12.216797,71.587596,36.298828)[shop];); out meta;
[timeout:86400]; (way(34.759984,-12.216797,71.587596,36.298828)[shop];); /added by auto repair/ (._;>;); /end of auto repair/ out meta;
I made some visualizations of the nodes density of roads in OpenStreetMap
For Romania, there where 379.445 ways that generated over 4.7 milions points that where counted in a grid of over 31000 polygons , with area between 7.43 square kilometers to 8.08 square kilometers
Zoom Out - 14000x9900 pixels Middle zoom Close zoom
2048 OpenStreetMap style density map Glitchy OpenStreetMap density style map
All the images can be downloaded from my Dropbox public folder https://www.dropbox.com/sh/qpgi10q3b62s5c0/AADN0qUyY9x3bNsHQlD0NRB5a?lst
This would not be possible to made in old QGIS 2.2, 2,0 it would have taken 10 times more time. Big thanks for all the dev that are mentaining and improving QGIS
I would really love to just import and play with the data in the planet extract, or the data for Romania, and try to do some statistics, but it`s to impossible just to install and make the software work.
I i know a thing or 2 about linux, but it still to complicated.
I really believe we should try and have a more integrated way of exploring the data, with GUI Tools, not complicated as is now.
For the sake of simplicity, this for now can be accomplished by ready to go, VirtualBox or VMware image disk, where i should only put the extract that i get from geofabrik
I was today on the field, to map this area http://mapcraft.nanodesu.ru/pie/407
The focus was on house-numbers. In total i added over 300 housenumbers, 18 pedestrian crossing, and 2 POI`s
Time spend :
2 Hours on the field, and 1 hour adding the data into JOSM
One of the roads was a oneway, that was not added on the map.
The final result can be seen here http://www.openstreetmap.org/#map=18/44.44068/26.14206
The mapping party will take place on the 17 and 18th of may
Just finished to import a city - before and after pictures
More interactive images can be found on the OSM Romania Page
In total we imported over 5000 buildings, 3000 postal adresses and 2500 properties
Just publish the first edition of state of the map Romania.
The infographic is in romanian languages, so i will highlight some of the key stats - State of the map Romania 2013
194,859 kilometers of road in total
73,207 kilometers added in 2013 12,986 bridge`s
16,331 different street names
231,991 - number of buildings,
47 % of them added in 2013
27,237 buildings with names
44,608 buildings with type of building defined
Distribution of Point of Interest
5,400 fixme TAG`s
Over 30.000 POI`s in total
869 locations with opening_hours tag
745 locations that provide information for access for persons with disability