OpenStreetMap

baditaflorin's diary

Recent diary entries

New OSM Postgis Script - Count the number of Restaurants inside each admin_level=4 polygon

Posted by baditaflorin on 14 April 2016 in English (English)

This script works in 2 parts.

1. First it will generate from the osm.pbf file the list with all the admin_level=4 (al_4)

Then it will remove the ones that are not closed. Then it will make a polygon from each al_4, using the ST_BuildArea

After it will have a inner join that will search for all the nodes and ways that are a restaurant.

2. Then, using st_contains we are searching to see in what polygon each restaurant fits.

Took 4 seconds to do it for Romania

You can find the query here https://github.com/baditaflorin/osm-postgis-scripts/blob/master/statistics/count_number_of_restaurants_inside_each_al_4.sql

| Admin_level_4 | Count | |----------------------|-------| | Municipiul București | 505 | | Cluj | 249 | | Constanța | 217 | | Iași | 155 | | Prahova | 128 | | Brașov | 124 | | Timiș | 121 | | Mureș | 109 | | Hunedoara | 88 | | Sibiu | 78 | | Arad | 74 | | Galați | 68 | | Buzău | 56 | | Neamț | 53 | | Harghita | 53 | | Dolj | 52 | | Argeș | 47 | | Bihor | 41 | | Covasna | 38 | | Ilfov | 36 | | Brăila | 35 | | Vâlcea | 33 | | Alba | 32 | | Maramureș | 29 | | Suceava | 28 | | Bacău | 28 | | Mehedinți | 28 | | Caraș Severin | 26 | | Călărași | 22 | | Tulcea | 22 | | Olt | 20 | | Satu Mare | 19 | | Gorj | 18 | | Dâmbovița | 16 | | Giurgiu | 13 | | Ialomița | 13 | | Botoșani | 13 | | Bistrița-Năsăud | 12 | | Vaslui | 11 | | Vrancea | 11 | | Teleorman | 7 | | Sălaj | 5 |

New category for osm postgis scripts

Posted by baditaflorin on 12 April 2016 in English (English)

I have added a new category , Statistics to the OSM PostGIS Script Repository

For example, using this quick query you can get a year by year statistic about the last time when a building was edited. You can see the result for Romania here :

https://github.com/baditaflorin/osm-postgis-scripts/tree/master/statistics

Nice looking import :)

Posted by baditaflorin on 5 April 2016 in English (English)

Random OSM World Stats

Posted by baditaflorin on 25 March 2016 in English (English)

The total lenght of the pedestrian roads in the world is 2.919.220 kilometers

For the tunnels and bridges, i will also present to you the global figure here 114.949 pedestrian tunnels are in the osm planet map 351.099 pedestrian bridges are in the osm planet map

Crossings Name 48872 Paris 27647 Madrid 26310 Milan 18959 Yokohama 16748 Helsinki 15445 Kawasaki 15418 Tokio 13542 Moskau 13117 Portland 13081 Phoenix

Top Tunnels

Count tunnels Name city 4304 MĂĽnchen 2552 Moskau 2255 Berlin 1948 Essen 1792 Paris 1417 Hamburg 1366 Milan 1366 Vienna 1268 Helsinki 1205 London 1091 Zurich 1082 Prag 1021 Stuttgart

Top Bridges

Count Name 3911 Hongkong 3844 Kawasaki 3648 Shenzhen 3544 Kowloon 3535 Tokio 3327 Yokohama 3070 Rotterdam 2330 London 2256 Amsterdam

NEW. Windows version for Osm PostGIS script repository

Posted by baditaflorin on 12 March 2016 in English (English)

I made a windows version of the scope.sh script. The script helps you load a osm.pbf file into PostGIS, without needing to know osmosis or other low level tools. The fact is that you want to load and use the data, not think what command line should i use in postgres or osmosis for doing this.

Osm Diary Entry about Osm PostGIS Script Repo

Github code

YouTube Tutorial

Experimental Osm PostGIS script - "OSM nodes Grid density"

Posted by baditaflorin on 8 March 2016 in English (English)

Osm PostGIS Script Repo – Find duplicated nodes - Romania example

Posted by baditaflorin on 8 March 2016 in English (English)

Objective

Run the find_duplicate_nodes.sql script from Github repository to find all the possible duplicated nodes that exist in Romania.

Hardware

The hardware used is a Intel i7-4770 CPU @ 3.40GHz, 16 GB RAM

Numbers

The pbf file for Romania is 144 MB. Romania database have a total 18M nodes. The query took 20 minutes to run, and found 34674 duplicated nodes.

34674 duplicated nodes loaded into JOSM

1

The way 336945939 have 1500 nodes, and does not contain now any relevant information.

2 Buildings not sharing same connecting nodes

3 All the city had been imported 2 times

This means that every building have a duplicate, one of them should be deleted.

4 Duplicated Highway path

Way 370332689 and way 363957545 are the same.

5 Bad Import

Separate ways for each objects, when they should be connected and share the same way.

6 2 relations on 2 different ways

Way 164811803 and way 164682871 should be combined where they shere the same path, and one of them deleted, and mode the relation into the remaining way

7 Bad Import - Building and fence not sharing same way.

That is one of my imports. Ups.

8 - To much details

9 - Reduntant Nodes

The line is strait, so the information is redundant.

Osm Diary Entry about Osm PostGIS Script Repo

Github code

YouTube Tutorial

New Osm Postgis Script – Find duplicated nodes.

Posted by baditaflorin on 12 February 2016 in English (English)

New Osm Postgis Script – Find duplicated nodes.

The idea of using a single repository where everybody cand find and fork the same Postgis OpenStreetMap schema scripts, is allowing anybody in the world to run a query on every part of the OSM database.

Read more on the project here http://www.openstreetmap.org/user/baditaflorin/diary/37758 Github code here https://github.com/baditaflorin/osm-postgis-scripts/tree/master

Even if you want to load a city and run a analysis on that town, or you want to find out where are all the duplicated nodes in Mexico, you can reuse the code that other people had created. Or you can add your own script that will be compatible with anybody that will load the osm.pbf with Osmosis.

To make the loading of a osm.pbf more simple, i made the SCOPE loader script for linux users, that allow everybody to create a postgis database without needing to know how to use Osmosis and other command line tools. Youtube tutorial here https://www.youtube.com/watch?v=vhJQbKey9EI

Mexico Example :

In Pgadmin i have run the command to find duplicated nodes on all the nodes that exist in Mexico on a intel octacore, 16 GB RAM Machine

Mexico have 13.000.000 nodes, and the query took 550 seconds to complete. It found 43.000 rows, meaning around 10.000 instances of possible duplicated nodes, ways. For example this building appers on the same location 23 times !!! http://www.openstreetmap.org/way/376880427#map=18/20.80528/-104.92827

JOSM building

This garden way needs smoothing 369775636 http://www.openstreetmap.org/way/369775636 Also this way 369775423

The find duplicate nodes it`s available here https://github.com/baditaflorin/osm-postgis-scripts/blob/master/find_duplicate_nodes.sql

New GitHub Repository : osm postgis scripts

Posted by baditaflorin on 12 January 2016 in English (English)

Openstreetmap Postgis Script Repository For Osmosis Pgsnapshot Schema - QGIS compatibility scripts

Github code here https://github.com/baditaflorin/osm-postgis-scripts/tree/master

Youtube tutorial here ( 4 minutes ) https://www.youtube.com/watch?v=vhJQbKey9EI

The licence is AGPLV3+

History

Here at Telenav , in the Map Analyst Team we use Postgis a lot.

In the real world, Postgis and OpenStreetMap seems to be used more only in Academics or by people that know a medium or high degree of programming.

I think that we should change that and make more simple the availability of Postgis as a tool that can help us visualize information's from the OSM database much faster.

3 Ways of making Postgis more easy to use with OSM data

I. - SCOPE

For this i will release SCOPE ( databaSe Creator Osmosis Postgis loadEr ), a command line tool for easy loading of a osm.pbf file into Postgis, via osmosis

Scope

  • You will find scope in the repo

II. A central place for osm-postgis-scripts

The idea is to have a central place were we can find different Postgis scripts, so when we want to do a statistic or something regarding the OSM data, to have a place were you can download, fork and store the scripts that you are using in your Projects The same is when you want to make a map, visualize some things, etc

screenshoot

https://github.com/baditaflorin/postgis-openstreetmap-script-repository-for-osmosis-pgsnapshot-schema

III. Qgis + Postgis = Speed

Because people from different parts of the world can use the same script, this will mean that we can also use the same QGIS Style , so in the repo you can also upload your Qgis Style that you have used with one of the scripts, and even if i will work on a different country, i will use the same DB schema and the same script, so the style will be compatible to use.

image

Check it out here https://github.com/baditaflorin/osm-postgis-scripts/tree/master

Bonus IV. Let`s use standards and templates so we can easily modify and create something new.

I have made this example as a blueprint for creating a new sql script file. https://github.com/baditaflorin/osm-postgis-scripts/blob/master/template.sql

Need input from you, how should we do with the names of the files, so that they respect some sort of standard And also if people want to add a QGIS Style, to respect a format

If the sql script is called nodes_show_adresses.sql the Qgis Style should be nodes_show_adresses_QGIS_description.eml

I am bad at the last part, so need your input so we can create something nice for the future.

Preannouncement for a new project with a long name

Posted by baditaflorin on 8 January 2016 in English (English)

The Long name is :

Postgis Openstreetmap Script Repository for Osmosis Pgsnapshot Schema

History

Here at Telenav , in the Map Analyst Team we use Postgis a lot.

In the real world, Postgis and OpenStreetMap seems to be used more only in Academics or by people that know a medium or high degree of programming.

I think that we should change that and make more simple the availability of Postgis as a tool that can help us visualize information's from the OSM database much faster.

3 Ways of making Postgis more easy to use with OSM data

I. - SCOPE

For this i will release SCOPE ( databaSe Creator Osmosis Postgis loadEr ), a command line tool for easy loading of a osm.pbf file into Postgis, via osmosis

Scope

  • You can try scope in the repo
  • Also, you will find there a Postgis example that i posted there.

II. A central place for Postgis OSM Scripts

The idea is to have a central place were we can find different Postgis scripts, so when we want to do a statistic or something regarding the OSM data, to have a place were you can download, fork and store the scripts that you are using in your Projects

screenshoot

https://github.com/baditaflorin/postgis-openstreetmap-script-repository-for-osmosis-pgsnapshot-schema

III. Give it a more sexy and short name

It also need a more simple name, i tried for to hours to think what name to give it, and i stoped and put this long name to the github repo. Please suggest a better and shorter name or acronym

Location: Gruia, Cluj-Napoca, Cluj, Romania

Turn Restriction part 2

Posted by baditaflorin on 30 December 2015 in English (English)

I will not elaborate now on the difficulties that i had getting ( either osmconvert or osmfilter removes the timestamp, user_id, leaving me with a extracted file that is 50 MB osm.pbf but cannot do nothing more then do some static statistics.

I cannot see how many where imports, how is evolving in time, nothing.

I could try to get the information via Overpass API that this would mean to query the database fro 500.000 relations

Until i figure out what i can do, i was able only do do reverse geocoding using POSTGIS, tried with the http://www.gadm.org/ dataset in QGIS to count the points in each polygon, but it got stuck after 2 hours at 94 %

I will try to load the GADM dataset in postgis and do reverse geocoding from POSTGIS, so i can have a better lever or details about the spread of the turn restriction in each country

Country ISO Count DE 89105 IT 53901 US 45986 RU 42827 ES 34697 BR 32212 PL 12125 FR 9974 GB 9311 AR 6701 AT 6570 HU 5584 AU 5505 CA 5037 DZ 4516 CH 4453 NL 4357 UA 4121 SK 3942 GR 3932 CZ 3422 PT 3371 RO 3219 HK 3145 BE 2818 LT 2603 CO 2295 SE 2169 FI 2016 BY 1893 JP 1815 HR 1688 ID 1659 CU 1543 DK 1497 UY 1449 CL 1444 VE 1416 KZ 1379 IN 1239 LV 1226 ZA 1185 IL 1157 CN 947 MX 900 EE 807 EC 766 KR 763 IE 744 SI 703 RS 683 NZ 654 MK 650 NO 650 MY 626 PH 603 BG 582 MD 544 AE 529 AZ 491 TR 487 AM 465 OM 383 GE 381 BA 361 CR 351 QA 348 TH 345 LU 338 SA 333 BO 329 TW 313 IR 296 TJ 256 CW 252 IS 232 BN 227 UZ 207 -99 179 CY 154 TT 153 SG 147 BH 144 AL 139 MA 136 PE 135 LB 127 IQ 125 JM 124 ME 117 ZW 116 TN 113 PG 111 NA 96 KG 89 PS 89 VN 86 AO 80 PY 74 SV 73 NC 66 EG 62 KE 61 CI 61 SY 57 MN 53 MT 50 NI 47 LK 41 KW 38 MZ 38 UG 37 PR 36 SM 36 MU 35 DO 35 LY 32 NP 31 GT 27 IM 27 KP 26 PK 19 AD 16 GN 16 KH 16 JO 16 TM 15 SN 14 ML 13 GA 13 KN 13 LI 12 NE 12 TZ 12 GH 12 GQ 12 LR 12 BT 11 TL 11 SR 11 LA 9 HT 9 BF 9 MG 8 BB 8 TD 7 AI 6 MM 6 MO 5 RW 5 CV 5 BI 5 GW 5 GM 5 VA 5 CM 5 DM 5 TG 5 ST 4 BD 4 MW 4 LS 4 BZ 4 BW 3 KY 3 GG 3 ET 2 YE 2 BS 2 AG 2 FJ 2 GD 2 KM 2 MR 1 ZM 1 SZ 1 CG 1 MF 1 GU 1 VU 1 JE 1 BJ 1 NG 1 HN 1 VC 1 BL 1 (blank) Grand Total 451977

10.000 broken Turn_Restriction in the OSM Planet File

Posted by baditaflorin on 29 December 2015 in English (English)

I have extracted all the turn_restriction that exist in the planet OSM file In total there are around 500.000 in the OSM Database

2% of them, or around 10.000 relations are broken, most of them because of other people that had added/ modified streets and deleted parts of the restriction relation.

A turn restriction is made of a TO - VIA - FROM relation That means that the relation should be of 3 different elements.

Found over 1000 relations that have a turn restriction relation made of at least 4 members, and going up to 100 members, when the absolute value is a 3 members relation

I have found 9700 turn_restriction relations that are invalid, because users deleted part of the relations, either in editors like ID that does not work people that they will break a relation, or other possibilities

You can see one example here http://www.openstreetmap.org/relation/1110481/history

I had made a editable Google SpreadSheet where you can see all of the 10000 broken relations, you can download them and report the ones that you had fixed

https://docs.google.com/spreadsheets/d/1ADbJl1QeSQmkfDadwEXtiYbDrYB8a1IT9FPU0ars7U8/edit#gid=0

Location: Gruia, Cluj-Napoca, Cluj, Romania

My christmas gift for the OSM Community - JOSM Keyboard Shortcuts Cheat Sheet 300 DPI

Posted by baditaflorin on 22 December 2015 in Spanish (Español)

Aim

From my point of view, if 1.000 OSM editors that use JOSM will learn at least one new shortcut that will allow them to save 5 minutes each day, we have 5000 minutes each day. Time that people can use to relax, or they can spend this extra 5000 minutes adding more things to the map.

JOSM Keyboard Shortcuts Cheat Sheet

Print and Download

You can download the full version ( 300 DPI ) from this dropbox link Happy Printing

Request

After you print it, p[ease send a photo with it. It would be glad to see in what parts of the world this CheatSheet will Travel.

Technical Details.

I have used Inkscape to design the JOSM Keyboard Shortcuts Cheat Sheet As a start, i have used my 5 year old attempt at a cheat-sheet for Inkscape https://openclipart.org/detail/93745/inkscape-keyboard-layoutv048-colored-path

Final Words

I would like to thanks to Telenav, the company were i work right now, for the non-formal approach that they have. You see, from time to time, we have a thing called Learning Friday, were we can learn what ever we want, related to our specific line of work. I wanted to learn even more about the Keyboard shortcuts in JOSM, So i decided that for my LF, i will do 2 in 1. I will learn a new thing, but i will also create a material so that other persons can also learn, in this way, all the OSM community can benefit out of my learning.

Thanks and happy holidays

Location: Andrei Mureșanu, Cluj-Napoca, Cluj, Rumanía

JOSM Turn Restriction Enhancement Suggestion

Posted by baditaflorin on 18 November 2015 in Spanish (Español)

You can see the Video Explaining the idea here http://youtu.be/YgzuX8juTW4

Text Transcript :

We can make the Turn Restriction tool more automated, by the order that we select the ways that will make out the Turn Restriction.

The first time we select the From The second time we select the Via The Third time we select the To

If you look now, the order is ok But you still need to type the 3 roles, losing ~10 seconds for each TR. My suggestion is to make the Turn Restriction autocomplete the FROM,VIA,TO fields if it detect a logical path between them

If you don`t wanna code that much: Put a experiential feature that allow the autocomplete function to exist, only on the logic :

What was selected first becomes "From", second becomes "Via", third becomes "To". Without a logical check in place. Expert Users will still benefit from this function.

Steps involved in the actual process :

  • Move mouse to the "from" Box -Click on the "from" Box
  • Type at least 2 words, so you will get the "from" suggestion, and not "forward"

  • Move mouse to the "via" Box -Click on the "via" Box

  • Type at least the "v" so you will get the autocomplete suggestion for the "via"

  • Move mouse to the "to" Box -Click on the "to" Box

  • Type at least the "t" so you will get the autocomplete suggestion for the "to"

  • Move mouse to the ok button

  • Click the ok button

A special thanks to all JOSM developers that are making this great tool even greater.

Estatus del proyecto de importación de México

Posted by baditaflorin on 27 October 2015 in Spanish (Español)

Estatus del proyecto de importación de México

Este documento explica el proceso que tuvimos que usar para importar más la mitad de las municipalidades en México.

  • admin_level=4 significa Provincia (Estado)

  • admin_level=6 significa Municipalidad

Estadísticas rápidas:

En números : Números de nodos/caminos/relaciones borradas

  • 500K / 2k / 500

Números de nodos/caminos/relaciones agregadas

  • 1000K / ~4k / ~1050

Número de horas dedicadas :

  • 200+

Número de municipalidades agregadas:

  • 1000+

Este anuncio está dividido en los pasos que se tuvieron que hacer para lograr la importación. Desde el inicio me gustaría mencionar que es un proceso de aprendizaje y todo el tiempo nos encontramos aprendiendo. Si encuentras algo que no consideras óptimo o si tienes una sugerencia de cómo hacerlo de manera distinta por favor avísanos.

Paso 0 : Transformación de Datos

El proceso está ilustrado en la página Wiki Mexico's Administrative Divisions Import Project

Paso 1 : Pensar en el flujo del proceso

Debido a cuestiones de tiempo decidimos importar por el momento 21 de las 32 provincias

Tuvimos ayuda de las personas que hicieron la importación de Puerto Rico y me gustaría agradecerles el apoyo y ayuda que nos han dado para esta importación

http://wiki.openstreetmap.org/wiki/Puerto_Rico_Projects/Puerto_Rico_Legal_Boundaries_Import

La diferencia fue que tuvimos los datos un poco más complicados porque no empezamos con un mapa limpio, tuvimos un mapa que también tenía ya límites en admin_level=4 , admin_level=6, admin_level=7|8|9|10

Una de las dificultades técnicas que tuvimos al principio fue esta:

  • Encontrar una solución de cómo manejar los caminos que también tienen otra relación en el camino (digamos admin level 4 y 6). Eso significa que no podemos borrarlos debido a que la relación de nivel 4 depende de la misma manera y cuando hagamos la importación se crearían caminos duplicados junto con la línea, si esto no está hecho con script necesitamos agregar manualmente 500 límites a los nuevos caminos y recrear algunas relaciones (ver la tarea en Paso 4 que necesitó 70 horas)

Se optó por el método en el cual borramos todo lo que se encuentra en admin_level=6 y manualmente se reconectaron las relaciones del admin_level=6 a los límites del admin_level=4 o en los lugares donde el admin_level=4 era diferente del admin_level=6, se movieron los límites del admin_level=4 para que se vieran en el admin_level=6. Esto es debido a que los datos que estamos subiendo son autorotativos siendo los datos oficiales del INEGI

Una de las soluciones fue usar un script que Victor de OSM Puerto Rico ha compartido con nosotros

https://github.com/vramirez122000/osm-pr-boundaries

Pasamos alrededor de 3 días haciendo la separación y fusión de los límites en QGIS. No es necesario explicar el proceso porque se desarrolló una herramienta interna llamada “Mexico Split” que convierte los Shapefiles de los límites del INEGI en un archivo OSM

Detalles técnicos: OSM no permite tener más de 200 nodos en un sólo camino, tuvimos que recrear las relaciones, dividir en cada intersección los nodos.

**Estatus del proyecto de importación de México *

“Mexico Split” es una pequeña herramienta de escritorio usada para convertir los límites de las municipalidades de México que fueron dados como polígonos en poli-líneas únicas. Las fronteras de dos municipalidades adyacentes, dadas como polígonos tendrán por naturaleza algunos caminos sobrepuestos.

First image

Esta herramienta está designada a eliminar por desconexión de sus polígonos y reemplazarlos por un camino común de los dos polígonos involucrados.

Además de su propósito principal, la herramienta también separa cualquier resultado mayor a 2000 segmentos en caminos más cortos, grupos de caminos en relaciones de acuerdo con los límites que ellos definen y agrega algunas etiquetas predefinidas para esos caminos y relaciones.

Etiquetas agregadas a las relaciones:

  • type=boundary

  • INEGI:MUNID=__

  • name=__

Etiquetas agregadas a ambos caminos y relaciones:

  • boundary=administrative

  • admin_level=6

  • source=INEGI

[](./media/image2.png)

Comparando a los datos originales[](./media/image1.png)

Límites de JOSM en estilo Map Paint

Y también un estilo de Map Paint de JOSM fue diseñado específicamente para facilitar el proceso de importación de límites administrativos

Detalles técnicos: Este estilo realza el último nodo en cada camino, haciendo esto simple para ver la distancia de cada camino

El nodo cuadrado también tiene un grado de transparencia, por eso podemos ver si hay un nodo abajo del nodo.

Para poder trabajar de manera sistemática,

Puedes probar el estilo aquí https://github.com/baditaflorin/boundaries-import-JOSM-MAPCSS-STYLE

text

Rápidamente se ven los nodos duplicados

Vean la diferencia entre admin_level=4 and admin_level=6

Seguimiento de estatus de cambios

Paso 2 : Borrar datos antiguos

Pasé casi 8 horas en un callejón sin salida, era imposible hacer un grupo para borrar los límites antiguos, porque admin_level=5 admin_level=8 admin_level=9 y otros límites pequeños que son invisibles están anexados al nivel administrativo 6 una búsqueda “overpass turbo” ayudó a tener éxito haciendo el borrado del archivo.

Tuvimos que despegar más de 2000 nodos que tienen en común un nodo con los límites. Estos eran caminos en donde los límites tenían también ríos, autopistas, desniveles, etc.

Más información acerca de los límites que comparten los nodos o caminos se puede encontrar en esta pregunta abierta que se consultó en el Talk-mail list https://www.mail-archive.com/talk@openstreetmap.org/msg54274.html

Primero se comenzó a hacer esto de manera manual pero cuando nos dimos cuenta que eran más de 2000 pedimos ayuda y se utilizó un script para despegarlos. Overpass- API está limitada a la cantidad de cosas que puede hacer, al menos para todo México, entonces acabamos creando una página de Wiki con todas las provincias y despegando provincia por provincia. Esto no pudo haber sido si toda la ayuda de Rafael Avila Coya Hackpad link describing the process

Detalles Técnicos : usamos una versión modificada del script http://overpass-turbo.eu/s/bXY

Tambén borramos alrededor de 500 relaciones antiquas teniendo un total de 500,000 nodos.

Para cada 50,000 nodos que subí o cambié tuve que esperar 30 minutos para que pasara la subida. Estas son las pequeñas cosas que no esperas y que puedes aprender en el camino. No incluí en el plan original la cantidad de tiempo para descarga, borrado y carga de tal gran cantidad.

Detalles Técnicos : El proceso de upload (carga) los nodos borrados, caminos, relaciones tomó 5 horas

Paso 3 : Agregando nuevos datos

Hicimos un upload (carga) de todos los municipios de 21 provincias, en total más de 1050 municipios, compuestos de 1.050.000 nodos.

Ten cuidado

El proceso de cargar 1 millón de nodos tomo aproximadamente 7-8 horas. Tuve que manualmente hacer algunas regresiones porque cuando se estaban todavía actualizando los nodos, un usuario borró un grupo de nodos y la carga falló al final porque no pudo hace toda la carga porque no encontró los nodos que estaba buscando. En el rango de 6-8 horas un usuario observe nodos sin ningún etiquetado en el mapa y los borró. Una solución que pudo hacer sido utilizada es agregar una etiqueta a cada uno de los 1 millón de nodos y borrar la etiqueta después de que el upload había sido completado.

Paso 4 : Limpieza / Verificar los nuevos datos agregados

Tuvimos que manualmente repara más de 500 relaciones municipales que necesitaron estar conectadas a las provincias.

No tuvimos un script que pudiese descargar todos los nodos, caminos y relación que estaba anexada a otra relación. En este sentido, algunas veces vas a tratar de borrar un límite antiguo de admin_level=4 y no puedes porque hay otra cosa agregada a él. El proceso de hacer esto debe de estar documetado mejor

Detalles Técnicos : Esta es la parte más larga, el flujo es ilustrado en videos de youtube que vamos a subir- alrededor de 1 hora de grabación

https://youtu.be/rbFGKF9ai5I _ _

https://youtu.be/dz36C-J91ec

https://youtu.be/b3vZNw7oP74

https://youtu.be/-hks7UHLC2Y

https://youtu.be/sA_0yheR70c

Había diferentes islas donde el set de datos de INEGI estaba fuera de lugar con valores hasta de 1.5 kilómetros

Alt text

Y otros errores encontrados por mi compañera Gabriela[](./media/image10.jpeg)

Pensamientos para el Futuro :

No hay una forma fácil de fusionar dos sets de datos diferentes, por ejemplo, el SHP para admin_level=4 y el SHP para el admin_level=6 en la idea de un proceso automático. Tomando esto un paso más allá, importar el admin_level=4,6 y 8 al mismo tiempo, crear un script que va a fusionar todos los datasets y cuando vas a subir para tener una subida completa.

Sería logarítmicamente más difícil hacer esto en una forma manual para los datos de AGEB donde no estás hablando de 2,400 municipios sino de 16,000 bloques de AGED. Para la municipalidad en sí misma y para todo México tendrías que volver a pegar 970 relaciones

Para el dataset del AGEB tienes que volver a unir alrededor de 7000 relaciones

Mexico boundaries import status.

Posted by baditaflorin on 26 October 2015 in English (English)

This document explain the process that we had used to import more then half of the municipalities in Mexico.

  • admin_level=4 means Province ( State )
  • admin_level=6 means Municipality

Short stats :

In Numbers : Number of nodes/ways/relations deleted

  • 500K / 2k / 500

Number of nodes/ways/relations added

  • 1000K / ~4k / ~1050

Number of hours spend :

  • 200+

Number of municipalities added :

  • 1000+

The post is broken down into the steps that we had done to accomplish the import. From the start, i want to point out that this is a learning process, and we are learning all the time. If you find something thay you consider suboptimal, or you have a suggestion of how to do something in a different way, please do tell.

Step 0 : Data Transformation

The process is illustrated in the wiki page Mexico's Administrative Divisions Import Project

Step 1 : Think at the workflow of the process

We had help from the people that did the Puerto Rico import, and I want to thank them for all the support and help that they had given to us for this import.

http://wiki.openstreetmap.org/wiki/Puerto_Rico_Projects/Puerto_Rico_Legal_Boundaries_Import

The difference was that we had the data a little more complicated, because we did not start with a clean map, we had a map that also had admin_level=4 , admin_level=6, admin_level=7|8|9|10 boundaries already in place.

One of the technical difficulties that we had in the beginning was this :

  • Find solution on how to deal with the ways that also have another relation in the way ( admin level 4 and 6, let`s say ). That means that we cannot delete them, because level 4 relation depends on the same way and when we will do the import it will create duplicated ways along the line If this is not done by script, we need to manually add around 500 boundaries to the new ways and recreate some relations. ( see the task at Step 4 that needed 70 hours )

I would like to thank Rafael Avila and all the people from the OSM_MX for the support that we had have for doing this.

We had opted for the method where we had deleted everything that is admin_level=6 and manually relink the admin_level=6 relations to the admin_level=4 boundaries, or in the places where the admin_level=4 was different then the admin_level=6, move the admin_level=4 boundaries to look like the admin_level=6 . This is because the data that we are uploading is autorative, being the official INEGI data.

One of the solution was to use a script that victor had shared with us https://github.com/vramirez122000/osm-pr-boundaries

We spend around 3 days doing the splitting and merging of the boundaries in QGIS. I will not explain the process, because, in the end we had developed a internal tool called Mexico Split that is converting the INEGI Boundaries Shapefiles into OSM file

Technical details : OSM does not allow to have more than 2000 nodes for a single way, we needed to recreate the relations, split at each intersection the nodes.

Mexico Split

Mexico Split is a small desktop tool used for converting Mexico municipality borders given as polygons into unique poly-lines. The borders of two adjacent municipalities, given as polygons, will inherently have some overlapping ways.

First image

This tool is designed to eliminate by detaching them from their polygons and replacing them with a single way common to the two involved polygons.

Besides this main purpose, the tool also splits any resulting ways longer than 2000 segments in shorter ways, groups the ways in relationships according to the borders they define, and adds some predefined tags to these ways and relations.

Tags added to relations:

  • type=boundary
  • INEGI:MUNID=__
  • name=__

Tags added to both ways and relations:

  • boundary=administrative
  • admin_level=6
  • source=INEGI

Alt text

Compare to original data First image

JOSM boundaries Map Paint Style

And also a Map Paint Style for JOSM designed specifically for easing the process of importing the admin boundaries.

Technical Details : The style highlights the last node of every way, making it simple to see the length of every way.

The square node also have a certain degree of transparency, so we can see if there is a node under the node.

To be able to work in a systematic way,

You can try the style here https://github.com/baditaflorin/boundaries-import-JOSM-MAPCSS-STYLE

text

Quickly see duplicated nodes

See the difference between the admin_level=4 and admin_level=6

Track status of changes

Step 2 : Delete old data

I have spend almost 8 hours on a dead end, it was impossible to bulk delete the old boundaries, because of admin_level=5 admin_level=8 admin_level=9 and other small boundaries that are invisible attached to the admin_level=6 , a modified overpass turbo query helped us success doing the deleting of the file.

We had to unglue over 2000 nodes that had a common node with the boundaries. This where ways where the boundaries where the same with a river, a highway, or level_crossing , etc

More info about boundaries that share nodes or ways can be found in this open question addressed on the talk mail-ling list https://www.mail-archive.com/talk@openstreetmap.org/msg54274.html

We first started doing this manually, but when we find out that there are over 2000, we asked for help and we had used a script that does the unglue . Overpass-API is limited in the amount of things it can do, at least for the whole mexico, so what we ended up doing was to create a wiki page with all of the provinces and unglue province by province . This could not have been done without the of the help of Rafael Avila Coya Hackpad link describing the process

Technical Details : we used a modified version of this script http://overpass-turbo.eu/s/bXY

Also, We deleted around 500 old relations, having a total of over 500.000 nodes.

For every 50.000 nodes that I upload or change, I have to wait almost 30 minutes for the upload to pass. This are the little things that you do not expect and you learn them on the go. I did not included in the initial timeline the amount of time needed for downloading, deleting and uploading such a big amount.

Technical details : The process of uploading the deleted nodes, ways and relations alone took around 5 hours

Step 3 : Add new data

We uploaded all the municipals for 21 provinces, in total there were over 1050 municipals, comprised of over 1.050.000 nodes.

Be careful

The process of uploading 1 million nodes takes around 7-8 hours. I had to manually do some reverts because when it was still uploading the nodes, a user deleted a bunch of nodes, and the upload failed at the very end, because the way could not upload, because it did not find the nodes that it was looking for, because in the time-frame of the 6-8 hours, a user observed some nodes without any tags on the map and he had deleted them. One solution that would burden the osm server and the history file a little more would be to add a tag to each of the 1 million nodes, and delete the tag after the upload is complete.

Step 4 : Clean / verify the new added data

We had to manually repair over 500 municipals relations that need to be connected to the province.

We did not manage to have a script that will also download all of the nodes, ways and relation that are attached to a relation. In this sense, sometimes you will try to delete an old admin_level=4 boundaries and you are not allowed because some other things are attached to it. The process of doing this should be better documented.

Technical details : This was the longest part, the workflow is illustrated in the youtube videos that we will upload – around 1 hour of footage.

https://youtu.be/rbFGKF9ai5I _ _

https://youtu.be/dz36C-J91ec

https://youtu.be/b3vZNw7oP74

https://youtu.be/-hks7UHLC2Y

https://youtu.be/sA_0yheR70c

There were different islands where the INEGI dataset was off with values as large as 1.5 kilometers.

Alt text

And some more errors, this one found by my colleague Gabriela

Future toughs :

There is not yet a simple way of merging 2 different datasets, for example the SHP for admin_level=4 and the SHP for admin_level=6 in the idea of a automatic process. Talking this one more step, to import the admin_level=4,6 and 8 in the same run, to create a script that will merge all of the datasets and when you will upload, to have a complete upload.

It will be logarithmic more harder to do this in a manual way for the AGEB data, where we are not talking of 2400 municipals, but of 16000 AGED blocks. For the municipals alone, for the whole mexico, you will have to reglue 970 relations.

For the AGEB dataset, you will have to re-glue around 7000 relations.

Location: Independencia San Martín, General Francisco R. Murguía, Zacatecas, Mexico

Open Question about tehnical issues around prepping boundaries SHP to Osm schema

Posted by baditaflorin on 30 September 2015 in English (English)

When you want to do a import, and you start with a SHP file, there are 3 challenges that i had identified :

How do you convert and split the file, in such a way that a way will not be longer then 2000 points ? How do you remove the line duplicates after you explode the file ( at first all of the municipality is a polygon, that means that at common borders, there will actually be 2 ways.

How do you elevate the properties of the shp file, now the information is on the way, and you want to move this information into the relation.

Average highway node distance between 2 points in OpenStreetMap - September 2015

Posted by baditaflorin on 24 September 2015 in English (English)

I only get the data per continents, for some types of roads. for europe i only got them for 6 types, because osmosis is not able to load the whole Europe file, at least with a server that have 16 GB RAM and octacore processor. I had a deadline, and from my estimation would have took more then 2 days to load the whole europe, so i had used osmfilter and osmconvert to extract different types of highway types

text

Average number of tags per way in OpenStreetMap

Posted by baditaflorin on 23 September 2015 in English (English)

That is, the average number of tags that each highway type have, per continents.

image

Highway ways in ASIA with over 200 tags

Posted by baditaflorin on 22 September 2015 in English (English)

There are 12 ways that have over 200 tags attached to them in asia

text

you can see them in overpass-turbo using this link http://overpass-turbo.eu/s/bAD

Thanks sandert17 for the suggestion to use overpass-turbo links

Older Entries | Newer Entries