OpenStreetMap

baditaflorin's diary

Recent diary entries

New GitHub Repository : osm postgis scripts

Posted by baditaflorin on 12 January 2016 in English (English)

Openstreetmap Postgis Script Repository For Osmosis Pgsnapshot Schema - QGIS compatibility scripts

Github code here https://github.com/baditaflorin/osm-postgis-scripts/tree/master

Youtube tutorial here ( 4 minutes ) https://www.youtube.com/watch?v=vhJQbKey9EI

The licence is AGPLV3+

History

Here at Telenav , in the Map Analyst Team we use Postgis a lot.

In the real world, Postgis and OpenStreetMap seems to be used more only in Academics or by people that know a medium or high degree of programming.

I think that we should change that and make more simple the availability of Postgis as a tool that can help us visualize information's from the OSM database much faster.

3 Ways of making Postgis more easy to use with OSM data

I. - SCOPE

For this i will release SCOPE ( databaSe Creator Osmosis Postgis loadEr ), a command line tool for easy loading of a osm.pbf file into Postgis, via osmosis

Scope

  • You will find scope in the repo

II. A central place for osm-postgis-scripts

The idea is to have a central place were we can find different Postgis scripts, so when we want to do a statistic or something regarding the OSM data, to have a place were you can download, fork and store the scripts that you are using in your Projects The same is when you want to make a map, visualize some things, etc

screenshoot

https://github.com/baditaflorin/postgis-openstreetmap-script-repository-for-osmosis-pgsnapshot-schema

III. Qgis + Postgis = Speed

Because people from different parts of the world can use the same script, this will mean that we can also use the same QGIS Style , so in the repo you can also upload your Qgis Style that you have used with one of the scripts, and even if i will work on a different country, i will use the same DB schema and the same script, so the style will be compatible to use.

image

Check it out here https://github.com/baditaflorin/osm-postgis-scripts/tree/master

Bonus IV. Let`s use standards and templates so we can easily modify and create something new.

I have made this example as a blueprint for creating a new sql script file. https://github.com/baditaflorin/osm-postgis-scripts/blob/master/template.sql

Need input from you, how should we do with the names of the files, so that they respect some sort of standard And also if people want to add a QGIS Style, to respect a format

If the sql script is called nodes_show_adresses.sql the Qgis Style should be nodes_show_adresses_QGIS_description.eml

I am bad at the last part, so need your input so we can create something nice for the future.

Preannouncement for a new project with a long name

Posted by baditaflorin on 8 January 2016 in English (English)

The Long name is :

Postgis Openstreetmap Script Repository for Osmosis Pgsnapshot Schema

History

Here at Telenav , in the Map Analyst Team we use Postgis a lot.

In the real world, Postgis and OpenStreetMap seems to be used more only in Academics or by people that know a medium or high degree of programming.

I think that we should change that and make more simple the availability of Postgis as a tool that can help us visualize information's from the OSM database much faster.

3 Ways of making Postgis more easy to use with OSM data

I. - SCOPE

For this i will release SCOPE ( databaSe Creator Osmosis Postgis loadEr ), a command line tool for easy loading of a osm.pbf file into Postgis, via osmosis

Scope

  • You can try scope in the repo
  • Also, you will find there a Postgis example that i posted there.

II. A central place for Postgis OSM Scripts

The idea is to have a central place were we can find different Postgis scripts, so when we want to do a statistic or something regarding the OSM data, to have a place were you can download, fork and store the scripts that you are using in your Projects

screenshoot

https://github.com/baditaflorin/postgis-openstreetmap-script-repository-for-osmosis-pgsnapshot-schema

III. Give it a more sexy and short name

It also need a more simple name, i tried for to hours to think what name to give it, and i stoped and put this long name to the github repo. Please suggest a better and shorter name or acronym

Location: Gruia, Cluj-Napoca, Cluj, Romania

Turn Restriction part 2

Posted by baditaflorin on 30 December 2015 in English (English)

I will not elaborate now on the difficulties that i had getting ( either osmconvert or osmfilter removes the timestamp, user_id, leaving me with a extracted file that is 50 MB osm.pbf but cannot do nothing more then do some static statistics.

I cannot see how many where imports, how is evolving in time, nothing.

I could try to get the information via Overpass API that this would mean to query the database fro 500.000 relations

Until i figure out what i can do, i was able only do do reverse geocoding using POSTGIS, tried with the http://www.gadm.org/ dataset in QGIS to count the points in each polygon, but it got stuck after 2 hours at 94 %

I will try to load the GADM dataset in postgis and do reverse geocoding from POSTGIS, so i can have a better lever or details about the spread of the turn restriction in each country

Country ISO Count DE 89105 IT 53901 US 45986 RU 42827 ES 34697 BR 32212 PL 12125 FR 9974 GB 9311 AR 6701 AT 6570 HU 5584 AU 5505 CA 5037 DZ 4516 CH 4453 NL 4357 UA 4121 SK 3942 GR 3932 CZ 3422 PT 3371 RO 3219 HK 3145 BE 2818 LT 2603 CO 2295 SE 2169 FI 2016 BY 1893 JP 1815 HR 1688 ID 1659 CU 1543 DK 1497 UY 1449 CL 1444 VE 1416 KZ 1379 IN 1239 LV 1226 ZA 1185 IL 1157 CN 947 MX 900 EE 807 EC 766 KR 763 IE 744 SI 703 RS 683 NZ 654 MK 650 NO 650 MY 626 PH 603 BG 582 MD 544 AE 529 AZ 491 TR 487 AM 465 OM 383 GE 381 BA 361 CR 351 QA 348 TH 345 LU 338 SA 333 BO 329 TW 313 IR 296 TJ 256 CW 252 IS 232 BN 227 UZ 207 -99 179 CY 154 TT 153 SG 147 BH 144 AL 139 MA 136 PE 135 LB 127 IQ 125 JM 124 ME 117 ZW 116 TN 113 PG 111 NA 96 KG 89 PS 89 VN 86 AO 80 PY 74 SV 73 NC 66 EG 62 KE 61 CI 61 SY 57 MN 53 MT 50 NI 47 LK 41 KW 38 MZ 38 UG 37 PR 36 SM 36 MU 35 DO 35 LY 32 NP 31 GT 27 IM 27 KP 26 PK 19 AD 16 GN 16 KH 16 JO 16 TM 15 SN 14 ML 13 GA 13 KN 13 LI 12 NE 12 TZ 12 GH 12 GQ 12 LR 12 BT 11 TL 11 SR 11 LA 9 HT 9 BF 9 MG 8 BB 8 TD 7 AI 6 MM 6 MO 5 RW 5 CV 5 BI 5 GW 5 GM 5 VA 5 CM 5 DM 5 TG 5 ST 4 BD 4 MW 4 LS 4 BZ 4 BW 3 KY 3 GG 3 ET 2 YE 2 BS 2 AG 2 FJ 2 GD 2 KM 2 MR 1 ZM 1 SZ 1 CG 1 MF 1 GU 1 VU 1 JE 1 BJ 1 NG 1 HN 1 VC 1 BL 1 (blank) Grand Total 451977

10.000 broken Turn_Restriction in the OSM Planet File

Posted by baditaflorin on 29 December 2015 in English (English)

I have extracted all the turn_restriction that exist in the planet OSM file In total there are around 500.000 in the OSM Database

2% of them, or around 10.000 relations are broken, most of them because of other people that had added/ modified streets and deleted parts of the restriction relation.

A turn restriction is made of a TO - VIA - FROM relation That means that the relation should be of 3 different elements.

Found over 1000 relations that have a turn restriction relation made of at least 4 members, and going up to 100 members, when the absolute value is a 3 members relation

I have found 9700 turn_restriction relations that are invalid, because users deleted part of the relations, either in editors like ID that does not work people that they will break a relation, or other possibilities

You can see one example here http://www.openstreetmap.org/relation/1110481/history

I had made a editable Google SpreadSheet where you can see all of the 10000 broken relations, you can download them and report the ones that you had fixed

https://docs.google.com/spreadsheets/d/1ADbJl1QeSQmkfDadwEXtiYbDrYB8a1IT9FPU0ars7U8/edit#gid=0

Location: Gruia, Cluj-Napoca, Cluj, Romania

My christmas gift for the OSM Community - JOSM Keyboard Shortcuts Cheat Sheet 300 DPI

Posted by baditaflorin on 22 December 2015 in Spanish (Español)

Aim

From my point of view, if 1.000 OSM editors that use JOSM will learn at least one new shortcut that will allow them to save 5 minutes each day, we have 5000 minutes each day. Time that people can use to relax, or they can spend this extra 5000 minutes adding more things to the map.

JOSM Keyboard Shortcuts Cheat Sheet

Print and Download

You can download the full version ( 300 DPI ) from this dropbox link Happy Printing

Request

After you print it, p[ease send a photo with it. It would be glad to see in what parts of the world this CheatSheet will Travel.

Technical Details.

I have used Inkscape to design the JOSM Keyboard Shortcuts Cheat Sheet As a start, i have used my 5 year old attempt at a cheat-sheet for Inkscape https://openclipart.org/detail/93745/inkscape-keyboard-layoutv048-colored-path

Final Words

I would like to thanks to Telenav, the company were i work right now, for the non-formal approach that they have. You see, from time to time, we have a thing called Learning Friday, were we can learn what ever we want, related to our specific line of work. I wanted to learn even more about the Keyboard shortcuts in JOSM, So i decided that for my LF, i will do 2 in 1. I will learn a new thing, but i will also create a material so that other persons can also learn, in this way, all the OSM community can benefit out of my learning.

Thanks and happy holidays

Location: Andrei Mureșanu, Cluj-Napoca, Cluj, Rumanía

JOSM Turn Restriction Enhancement Suggestion

Posted by baditaflorin on 18 November 2015 in Spanish (Español)

You can see the Video Explaining the idea here http://youtu.be/YgzuX8juTW4

Text Transcript :

We can make the Turn Restriction tool more automated, by the order that we select the ways that will make out the Turn Restriction.

The first time we select the From The second time we select the Via The Third time we select the To

If you look now, the order is ok But you still need to type the 3 roles, losing ~10 seconds for each TR. My suggestion is to make the Turn Restriction autocomplete the FROM,VIA,TO fields if it detect a logical path between them

If you don`t wanna code that much: Put a experiential feature that allow the autocomplete function to exist, only on the logic :

What was selected first becomes "From", second becomes "Via", third becomes "To". Without a logical check in place. Expert Users will still benefit from this function.

Steps involved in the actual process :

  • Move mouse to the "from" Box -Click on the "from" Box
  • Type at least 2 words, so you will get the "from" suggestion, and not "forward"

  • Move mouse to the "via" Box -Click on the "via" Box

  • Type at least the "v" so you will get the autocomplete suggestion for the "via"

  • Move mouse to the "to" Box -Click on the "to" Box

  • Type at least the "t" so you will get the autocomplete suggestion for the "to"

  • Move mouse to the ok button

  • Click the ok button

A special thanks to all JOSM developers that are making this great tool even greater.

Estatus del proyecto de importación de México

Posted by baditaflorin on 27 October 2015 in Spanish (Español)

Estatus del proyecto de importación de México

Este documento explica el proceso que tuvimos que usar para importar más la mitad de las municipalidades en México.

  • admin_level=4 significa Provincia (Estado)

  • admin_level=6 significa Municipalidad

Estadísticas rápidas:

En números : Números de nodos/caminos/relaciones borradas

  • 500K / 2k / 500

Números de nodos/caminos/relaciones agregadas

  • 1000K / ~4k / ~1050

Número de horas dedicadas :

  • 200+

Número de municipalidades agregadas:

  • 1000+

Este anuncio está dividido en los pasos que se tuvieron que hacer para lograr la importación. Desde el inicio me gustaría mencionar que es un proceso de aprendizaje y todo el tiempo nos encontramos aprendiendo. Si encuentras algo que no consideras óptimo o si tienes una sugerencia de cómo hacerlo de manera distinta por favor avísanos.

Paso 0 : Transformación de Datos

El proceso está ilustrado en la página Wiki Mexico's Administrative Divisions Import Project

Paso 1 : Pensar en el flujo del proceso

Debido a cuestiones de tiempo decidimos importar por el momento 21 de las 32 provincias

Tuvimos ayuda de las personas que hicieron la importación de Puerto Rico y me gustaría agradecerles el apoyo y ayuda que nos han dado para esta importación

http://wiki.openstreetmap.org/wiki/Puerto_Rico_Projects/Puerto_Rico_Legal_Boundaries_Import

La diferencia fue que tuvimos los datos un poco más complicados porque no empezamos con un mapa limpio, tuvimos un mapa que también tenía ya límites en admin_level=4 , admin_level=6, admin_level=7|8|9|10

Una de las dificultades técnicas que tuvimos al principio fue esta:

  • Encontrar una solución de cómo manejar los caminos que también tienen otra relación en el camino (digamos admin level 4 y 6). Eso significa que no podemos borrarlos debido a que la relación de nivel 4 depende de la misma manera y cuando hagamos la importación se crearían caminos duplicados junto con la línea, si esto no está hecho con script necesitamos agregar manualmente 500 límites a los nuevos caminos y recrear algunas relaciones (ver la tarea en Paso 4 que necesitó 70 horas)

Se optó por el método en el cual borramos todo lo que se encuentra en admin_level=6 y manualmente se reconectaron las relaciones del admin_level=6 a los límites del admin_level=4 o en los lugares donde el admin_level=4 era diferente del admin_level=6, se movieron los límites del admin_level=4 para que se vieran en el admin_level=6. Esto es debido a que los datos que estamos subiendo son autorotativos siendo los datos oficiales del INEGI

Una de las soluciones fue usar un script que Victor de OSM Puerto Rico ha compartido con nosotros

https://github.com/vramirez122000/osm-pr-boundaries

Pasamos alrededor de 3 días haciendo la separación y fusión de los límites en QGIS. No es necesario explicar el proceso porque se desarrolló una herramienta interna llamada “Mexico Split” que convierte los Shapefiles de los límites del INEGI en un archivo OSM

Detalles técnicos: OSM no permite tener más de 200 nodos en un sólo camino, tuvimos que recrear las relaciones, dividir en cada intersección los nodos.

**Estatus del proyecto de importación de México *

“Mexico Split” es una pequeña herramienta de escritorio usada para convertir los límites de las municipalidades de México que fueron dados como polígonos en poli-líneas únicas. Las fronteras de dos municipalidades adyacentes, dadas como polígonos tendrán por naturaleza algunos caminos sobrepuestos.

First image

Esta herramienta está designada a eliminar por desconexión de sus polígonos y reemplazarlos por un camino común de los dos polígonos involucrados.

Además de su propósito principal, la herramienta también separa cualquier resultado mayor a 2000 segmentos en caminos más cortos, grupos de caminos en relaciones de acuerdo con los límites que ellos definen y agrega algunas etiquetas predefinidas para esos caminos y relaciones.

Etiquetas agregadas a las relaciones:

  • type=boundary

  • INEGI:MUNID=__

  • name=__

Etiquetas agregadas a ambos caminos y relaciones:

  • boundary=administrative

  • admin_level=6

  • source=INEGI

[](./media/image2.png)

Comparando a los datos originales[](./media/image1.png)

Límites de JOSM en estilo Map Paint

Y también un estilo de Map Paint de JOSM fue diseñado específicamente para facilitar el proceso de importación de límites administrativos

Detalles técnicos: Este estilo realza el último nodo en cada camino, haciendo esto simple para ver la distancia de cada camino

El nodo cuadrado también tiene un grado de transparencia, por eso podemos ver si hay un nodo abajo del nodo.

Para poder trabajar de manera sistemática,

Puedes probar el estilo aquí https://github.com/baditaflorin/boundaries-import-JOSM-MAPCSS-STYLE

text

Rápidamente se ven los nodos duplicados

Vean la diferencia entre admin_level=4 and admin_level=6

Seguimiento de estatus de cambios

Paso 2 : Borrar datos antiguos

Pasé casi 8 horas en un callejón sin salida, era imposible hacer un grupo para borrar los límites antiguos, porque admin_level=5 admin_level=8 admin_level=9 y otros límites pequeños que son invisibles están anexados al nivel administrativo 6 una búsqueda “overpass turbo” ayudó a tener éxito haciendo el borrado del archivo.

Tuvimos que despegar más de 2000 nodos que tienen en común un nodo con los límites. Estos eran caminos en donde los límites tenían también ríos, autopistas, desniveles, etc.

Más información acerca de los límites que comparten los nodos o caminos se puede encontrar en esta pregunta abierta que se consultó en el Talk-mail list https://www.mail-archive.com/talk@openstreetmap.org/msg54274.html

Primero se comenzó a hacer esto de manera manual pero cuando nos dimos cuenta que eran más de 2000 pedimos ayuda y se utilizó un script para despegarlos. Overpass- API está limitada a la cantidad de cosas que puede hacer, al menos para todo México, entonces acabamos creando una página de Wiki con todas las provincias y despegando provincia por provincia. Esto no pudo haber sido si toda la ayuda de Rafael Avila Coya Hackpad link describing the process

Detalles Técnicos : usamos una versión modificada del script http://overpass-turbo.eu/s/bXY

Tambén borramos alrededor de 500 relaciones antiquas teniendo un total de 500,000 nodos.

Para cada 50,000 nodos que subí o cambié tuve que esperar 30 minutos para que pasara la subida. Estas son las pequeñas cosas que no esperas y que puedes aprender en el camino. No incluí en el plan original la cantidad de tiempo para descarga, borrado y carga de tal gran cantidad.

Detalles Técnicos : El proceso de upload (carga) los nodos borrados, caminos, relaciones tomó 5 horas

Paso 3 : Agregando nuevos datos

Hicimos un upload (carga) de todos los municipios de 21 provincias, en total más de 1050 municipios, compuestos de 1.050.000 nodos.

Ten cuidado

El proceso de cargar 1 millón de nodos tomo aproximadamente 7-8 horas. Tuve que manualmente hacer algunas regresiones porque cuando se estaban todavía actualizando los nodos, un usuario borró un grupo de nodos y la carga falló al final porque no pudo hace toda la carga porque no encontró los nodos que estaba buscando. En el rango de 6-8 horas un usuario observe nodos sin ningún etiquetado en el mapa y los borró. Una solución que pudo hacer sido utilizada es agregar una etiqueta a cada uno de los 1 millón de nodos y borrar la etiqueta después de que el upload había sido completado.

Paso 4 : Limpieza / Verificar los nuevos datos agregados

Tuvimos que manualmente repara más de 500 relaciones municipales que necesitaron estar conectadas a las provincias.

No tuvimos un script que pudiese descargar todos los nodos, caminos y relación que estaba anexada a otra relación. En este sentido, algunas veces vas a tratar de borrar un límite antiguo de admin_level=4 y no puedes porque hay otra cosa agregada a él. El proceso de hacer esto debe de estar documetado mejor

Detalles Técnicos : Esta es la parte más larga, el flujo es ilustrado en videos de youtube que vamos a subir- alrededor de 1 hora de grabación

https://youtu.be/rbFGKF9ai5I _ _

https://youtu.be/dz36C-J91ec

https://youtu.be/b3vZNw7oP74

https://youtu.be/-hks7UHLC2Y

https://youtu.be/sA_0yheR70c

Había diferentes islas donde el set de datos de INEGI estaba fuera de lugar con valores hasta de 1.5 kilómetros

Alt text

Y otros errores encontrados por mi compañera Gabriela[](./media/image10.jpeg)

Pensamientos para el Futuro :

No hay una forma fácil de fusionar dos sets de datos diferentes, por ejemplo, el SHP para admin_level=4 y el SHP para el admin_level=6 en la idea de un proceso automático. Tomando esto un paso más allá, importar el admin_level=4,6 y 8 al mismo tiempo, crear un script que va a fusionar todos los datasets y cuando vas a subir para tener una subida completa.

Sería logarítmicamente más difícil hacer esto en una forma manual para los datos de AGEB donde no estás hablando de 2,400 municipios sino de 16,000 bloques de AGED. Para la municipalidad en sí misma y para todo México tendrías que volver a pegar 970 relaciones

Para el dataset del AGEB tienes que volver a unir alrededor de 7000 relaciones

Mexico boundaries import status.

Posted by baditaflorin on 26 October 2015 in English (English)

This document explain the process that we had used to import more then half of the municipalities in Mexico.

  • admin_level=4 means Province ( State )
  • admin_level=6 means Municipality

Short stats :

In Numbers : Number of nodes/ways/relations deleted

  • 500K / 2k / 500

Number of nodes/ways/relations added

  • 1000K / ~4k / ~1050

Number of hours spend :

  • 200+

Number of municipalities added :

  • 1000+

The post is broken down into the steps that we had done to accomplish the import. From the start, i want to point out that this is a learning process, and we are learning all the time. If you find something thay you consider suboptimal, or you have a suggestion of how to do something in a different way, please do tell.

Step 0 : Data Transformation

The process is illustrated in the wiki page Mexico's Administrative Divisions Import Project

Step 1 : Think at the workflow of the process

We had help from the people that did the Puerto Rico import, and I want to thank them for all the support and help that they had given to us for this import.

http://wiki.openstreetmap.org/wiki/Puerto_Rico_Projects/Puerto_Rico_Legal_Boundaries_Import

The difference was that we had the data a little more complicated, because we did not start with a clean map, we had a map that also had admin_level=4 , admin_level=6, admin_level=7|8|9|10 boundaries already in place.

One of the technical difficulties that we had in the beginning was this :

  • Find solution on how to deal with the ways that also have another relation in the way ( admin level 4 and 6, let`s say ). That means that we cannot delete them, because level 4 relation depends on the same way and when we will do the import it will create duplicated ways along the line If this is not done by script, we need to manually add around 500 boundaries to the new ways and recreate some relations. ( see the task at Step 4 that needed 70 hours )

I would like to thank Rafael Avila and all the people from the OSM_MX for the support that we had have for doing this.

We had opted for the method where we had deleted everything that is admin_level=6 and manually relink the admin_level=6 relations to the admin_level=4 boundaries, or in the places where the admin_level=4 was different then the admin_level=6, move the admin_level=4 boundaries to look like the admin_level=6 . This is because the data that we are uploading is autorative, being the official INEGI data.

One of the solution was to use a script that victor had shared with us https://github.com/vramirez122000/osm-pr-boundaries

We spend around 3 days doing the splitting and merging of the boundaries in QGIS. I will not explain the process, because, in the end we had developed a internal tool called Mexico Split that is converting the INEGI Boundaries Shapefiles into OSM file

Technical details : OSM does not allow to have more than 2000 nodes for a single way, we needed to recreate the relations, split at each intersection the nodes.

Mexico Split

Mexico Split is a small desktop tool used for converting Mexico municipality borders given as polygons into unique poly-lines. The borders of two adjacent municipalities, given as polygons, will inherently have some overlapping ways.

First image

This tool is designed to eliminate by detaching them from their polygons and replacing them with a single way common to the two involved polygons.

Besides this main purpose, the tool also splits any resulting ways longer than 2000 segments in shorter ways, groups the ways in relationships according to the borders they define, and adds some predefined tags to these ways and relations.

Tags added to relations:

  • type=boundary
  • INEGI:MUNID=__
  • name=__

Tags added to both ways and relations:

  • boundary=administrative
  • admin_level=6
  • source=INEGI

Alt text

Compare to original data First image

JOSM boundaries Map Paint Style

And also a Map Paint Style for JOSM designed specifically for easing the process of importing the admin boundaries.

Technical Details : The style highlights the last node of every way, making it simple to see the length of every way.

The square node also have a certain degree of transparency, so we can see if there is a node under the node.

To be able to work in a systematic way,

You can try the style here https://github.com/baditaflorin/boundaries-import-JOSM-MAPCSS-STYLE

text

Quickly see duplicated nodes

See the difference between the admin_level=4 and admin_level=6

Track status of changes

Step 2 : Delete old data

I have spend almost 8 hours on a dead end, it was impossible to bulk delete the old boundaries, because of admin_level=5 admin_level=8 admin_level=9 and other small boundaries that are invisible attached to the admin_level=6 , a modified overpass turbo query helped us success doing the deleting of the file.

We had to unglue over 2000 nodes that had a common node with the boundaries. This where ways where the boundaries where the same with a river, a highway, or level_crossing , etc

More info about boundaries that share nodes or ways can be found in this open question addressed on the talk mail-ling list https://www.mail-archive.com/talk@openstreetmap.org/msg54274.html

We first started doing this manually, but when we find out that there are over 2000, we asked for help and we had used a script that does the unglue . Overpass-API is limited in the amount of things it can do, at least for the whole mexico, so what we ended up doing was to create a wiki page with all of the provinces and unglue province by province . This could not have been done without the of the help of Rafael Avila Coya Hackpad link describing the process

Technical Details : we used a modified version of this script http://overpass-turbo.eu/s/bXY

Also, We deleted around 500 old relations, having a total of over 500.000 nodes.

For every 50.000 nodes that I upload or change, I have to wait almost 30 minutes for the upload to pass. This are the little things that you do not expect and you learn them on the go. I did not included in the initial timeline the amount of time needed for downloading, deleting and uploading such a big amount.

Technical details : The process of uploading the deleted nodes, ways and relations alone took around 5 hours

Step 3 : Add new data

We uploaded all the municipals for 21 provinces, in total there were over 1050 municipals, comprised of over 1.050.000 nodes.

Be careful

The process of uploading 1 million nodes takes around 7-8 hours. I had to manually do some reverts because when it was still uploading the nodes, a user deleted a bunch of nodes, and the upload failed at the very end, because the way could not upload, because it did not find the nodes that it was looking for, because in the time-frame of the 6-8 hours, a user observed some nodes without any tags on the map and he had deleted them. One solution that would burden the osm server and the history file a little more would be to add a tag to each of the 1 million nodes, and delete the tag after the upload is complete.

Step 4 : Clean / verify the new added data

We had to manually repair over 500 municipals relations that need to be connected to the province.

We did not manage to have a script that will also download all of the nodes, ways and relation that are attached to a relation. In this sense, sometimes you will try to delete an old admin_level=4 boundaries and you are not allowed because some other things are attached to it. The process of doing this should be better documented.

Technical details : This was the longest part, the workflow is illustrated in the youtube videos that we will upload – around 1 hour of footage.

https://youtu.be/rbFGKF9ai5I _ _

https://youtu.be/dz36C-J91ec

https://youtu.be/b3vZNw7oP74

https://youtu.be/-hks7UHLC2Y

https://youtu.be/sA_0yheR70c

There were different islands where the INEGI dataset was off with values as large as 1.5 kilometers.

Alt text

And some more errors, this one found by my colleague Gabriela

Future toughs :

There is not yet a simple way of merging 2 different datasets, for example the SHP for admin_level=4 and the SHP for admin_level=6 in the idea of a automatic process. Talking this one more step, to import the admin_level=4,6 and 8 in the same run, to create a script that will merge all of the datasets and when you will upload, to have a complete upload.

It will be logarithmic more harder to do this in a manual way for the AGEB data, where we are not talking of 2400 municipals, but of 16000 AGED blocks. For the municipals alone, for the whole mexico, you will have to reglue 970 relations.

For the AGEB dataset, you will have to re-glue around 7000 relations.

Location: Independencia San Martín, General Francisco R. Murguía, Zacatecas, Mexico

Open Question about tehnical issues around prepping boundaries SHP to Osm schema

Posted by baditaflorin on 30 September 2015 in English (English)

When you want to do a import, and you start with a SHP file, there are 3 challenges that i had identified :

How do you convert and split the file, in such a way that a way will not be longer then 2000 points ? How do you remove the line duplicates after you explode the file ( at first all of the municipality is a polygon, that means that at common borders, there will actually be 2 ways.

How do you elevate the properties of the shp file, now the information is on the way, and you want to move this information into the relation.

Average highway node distance between 2 points in OpenStreetMap - September 2015

Posted by baditaflorin on 24 September 2015 in English (English)

I only get the data per continents, for some types of roads. for europe i only got them for 6 types, because osmosis is not able to load the whole Europe file, at least with a server that have 16 GB RAM and octacore processor. I had a deadline, and from my estimation would have took more then 2 days to load the whole europe, so i had used osmfilter and osmconvert to extract different types of highway types

text

Average number of tags per way in OpenStreetMap

Posted by baditaflorin on 23 September 2015 in English (English)

That is, the average number of tags that each highway type have, per continents.

image

Highway ways in ASIA with over 200 tags

Posted by baditaflorin on 22 September 2015 in English (English)

There are 12 ways that have over 200 tags attached to them in asia

text

you can see them in overpass-turbo using this link http://overpass-turbo.eu/s/bAD

Thanks sandert17 for the suggestion to use overpass-turbo links

POI nodes with over 100 tags

Posted by baditaflorin on 18 September 2015 in English (English)

Ever wandered how many POI over 100 tags ? Now you can know.

There are 33 nodes that have over 100 tags, mostly if not all being lighthouse

https://gist.github.com/baditaflorin/8a30a5f45c94f4d2bf8d

I am not able to save the list with the tags also, at least not in OpenJump ( SHP will truncate at 255, other extension does not work ) , QGIS ( Error after the forth value ) ( filled bug request ) Pgadmin (don`t know how to save directly so that i could open it

[Update] A list with all the nodes tags that have over 50 tags

https://gist.github.com/baditaflorin/53db974195c2d1e95745

image

Postigs question - find out the unnecessary points that exist on the map

Posted by baditaflorin on 16 September 2015 in English (English)

question

My hypothesis is that we could reduce the planet file with 10-100 MB by only removing the unnecessary points that exist on the map.

I am trying to figure it out the correct postgis query to find out exactly this.

I am trying to compare the points of a linesting, and if the degrees between 3 adjacent points is 0 degrees, that means that the point in the middle can be deleted.

The only check in place that i see is to check that the point in the middle is not connected to a point and check that the hstore tag of that node is empty, meaning that there is no value added to the node ( pedestrian crossing, motorway_jucntion, exit_to, etc )

Osmosis Stats for loading different continents

Posted by baditaflorin on 16 September 2015 in English (English)

This is just a dump of some ideas and stats that you should not take for granted, the data its most of the time inaccurate , because i run 2 commands at the same time.

But anyhow, its a point of conversation First, i had filtered with osmconvert and osmfilter so i remained with a planet dump, composed of only the tag highway I am trying to import all the roads in the world, after i tried with a direct attempt and did not scale up, i had now spitted the world into each continent

Size = only the highway , saved as a osm.pbf

africa_highway ( 356 MB ) = 46 Minute to import into postgis Asia_highway ( 1318 MB ) = 213 Minute to import into postgis Australia_highway ( 259 MB ) = 14 Minute to import into postgis Central America ( 63 MB ) = 10 Minute to import into postgis

How many days or weeks does it take to upload the whole planet via Osmosis ?

Posted by baditaflorin on 15 September 2015 in English (English)

I am now in day 6 of waiting

There is a 16 GB RAM server, octacore environment, running ubuntu 14.10 ( don`t ask why )

I have a 317 GB file generated called copy********************n

And there is a file called copy**************w that is still being generated, growing by 10-20 GB per day , now is at 76 GB

And a file called copy*******************wn at 30 GB

i am guessing that the n means nodes and w means ways, but how much should it grow, procentualy compared with the 317 GB n file ?

6 day stats

The command that i had used to give the command looks like this

sudo osmosis --rbf latest-planet.osm.pbf --wp host=localhost database=mydatabase user=******* password=**********

Top 25 cities in the world by the number of POI

Posted by baditaflorin on 9 September 2015 in English (English)

Have you ever asked yourself what are the TOP 25 cities in the world, based on the number of POI in OpenStreetMap ?

To my surprise, the number 1 city is not a European one, but a city in Japan, that have 121.154 POI

I have calculated all the poi from the following category's : amenity, leisure, shop, sport, tourism, man_made, office

I did not include the historic POI, because when i filtered the world planet, i had extracted the history tag, instead of historic

I had converter the ways into polygons, and then combined the 2 data-sets into one.

Then i had counted, based on the bbox of the city, the number of POI in each of the cities. I will give here the TOP 25 cities by the number of POI

Waiting for your opinions, comments, etc

Yokohama 121154 Paris 113165 Tokio 97078 Kawasaki 90576 Rio Grande 85238 London 84681 Saitama 75783 Berlin 74002 Birmingham 65093 Moskau 61579 München 54533 Essen 53935 Dusseldorf 52342 Stuttgart 50712 Madrid 48335 Vienna 47396 Toronto 47324 Köln 45018 Dortmund 44534 Lyon 44441 Hamburg 44334 Sete 41723 Milan 41317 Osaka 40756 Frankfurt 38128

Location: Gruia, Cluj-Napoca, Cluj, Romania

OSM Version and tags visualization

Posted by baditaflorin on 5 August 2015 in English (English)

Short Legend : Bigger green means bigger version, bigger red ways means that the way contains more tags inside. The same with the points

Country View Link

Close Up

Close Up

Link

Bucharest City View

You can download the bucharest city view from here 4A0 - 250 DPI - 180 MB file 23405px * 16555 px

How to add a relation in OpenStreetMap

Posted by baditaflorin on 15 July 2015 in English (English)

In Romania, we are planning to add the relations for all the national Roads

I made this youtube video-clip to show how easy is to add a relation

https://www.youtube.com/watch?v=nv6HU1YxUWw

Location: Gruia, Cluj-Napoca, Cluj, Romania

OSM POI age of different cities around the world

Posted by baditaflorin on 25 September 2014 in English (English)

Playing with Mapbox Studio and the ability to use the osm_id of a POI, i managed to make a historic map of the POI of 24 selected cities around the world

In a totally random way i will show some of the different type of cities that we have in OSM, in the terms of relevance to people that are interested in going in that city and using osm for POI

Most of the POI in Ankara were added in 2006 - 2008 Alt text

Berlin Alt text

Bruxelles Alt text

Bucharest Alt text

Budapest - An example of a systematic, mapping party approach ( somebody with local knowledge can correct me if i am wrong ) Alt text

Buenos Aires - It would be useful to see how many of the new edits are from local people, to see how sustainable is that and how much the local community will continue to add after the SOTM Alt text

Cluj Napoca Another example of the power of mapping party, for this ever-growing city, that will be in 2015 the European Capital of Youth Alt text

Dublin Alt text

Frankfurt - Central, POI only around important streets, approach Alt text

Hong Kong Alt text

L.A. - From the types of POI that where added in 2009, Los Angeles had an import of date in 2009, but except from that the city is almost dead in terms of POI growth Alt text

London - is more complete then it looks on this map. In Mapbox i took only the POI that are a node . In London there are a lot of POI that they are tagged as a building. For the sake of simplicity those tag`s are only rendered as text, with a small white dot.

It was a technical stuff, i would have to calculate the age for the POI of the ways.

Alt text

Madrid Alt text

Melbourne - a example of a city center POI useful city Alt text

Moskow - A beautiful example of a complete city, where you can find POI in all the city, not just in the city center Alt text

Munchen Alt text

New York - except a import in 2009, in the last 2 years the city center started getting some kind of attention Alt text

Prague Alt text

Rotterdam Alt text

Sofia Alt text

Tokio - Is growing fast, in a decentralized, all over the map type Alt text

Wien Alt text

Zagreb Alt text

Zurich Alt text

You can download all the Full Resolution maps here https://www.dropbox.com/sh/whixmqcr14n873y/AAA5cRJev87o7dcdjAnKsylda?lst

Older Entries | Newer Entries