Mapping Mexico with INEGI, you and...Cygnus
Posted by mvexel on 29 July 2015 in English. Last updated on 3 June 2022.Mexico released a huge amount of open data not too long ago.
A lot of this data is geospatial, so I say yummie! Alex Barth wrote about this data, that comes from the Mexican national statistical agency INEGI, on his diary before, with a nifty map to show how rich this data is:
(My mediocre animated GIF skills really don’t do it justice - check out Alex’s blog post to see an interactive map.)
So now the question becomes: how do we get some (or all?) of this data into OSM? This is not straightforward - OSM already has rich data in many places in Mexico we would definitely want to keep.
Here at the Telenav OSM team, we have come up with an answer to this question. We call it Cygnus - The Bringer of Balance. Let me explain in a few visuals what Cygnus does.
Consider this area in the Aguascalientes region. There is some OSM data there:
If we look at the Bing aerial images, we can clearly see that there’s an entire village there that is not mapped though!
But INEGI has most if not all the roads in this village in their now open dataset Conjunto de Datos Vectoriales de Carreteras y Vialidades Urbanas.
After converting the original data attributes mapped to OSM-appropriate tagging, saving the result as an OSM file, and loading it into JOSM, it looks like this:
OK, that is nice, but we still have two separate layers that are unconnected, and even if we merge them, we will still have to manually resolve duplicate ways and connections between the ways from OSM and those from the INEGI data.
This is where Cygnus comes in - a new conflation technology we developed at Telenav specifically to tackle this.
Cygnus, as the Bringer of Balance, takes as its input a base OSM file in PBF format, as well as what we call an ‘enhancing’ file, also in PBF format. It will then compare and conflate the two inputs and output one JOSM XML file that can be merged with OSM base data straight away. This detail shows the merged layers with all the INEGI ‘imported’ ways connected to the pre-existing OSM way:
Even though Cygnus does a pretty amazing job merging OSM data with an ‘enhancing’ layer, you will still need to check the result before you upload. Take this example here:
The highway=secondary
was the pre-existing OSM data, and the highway=residential; oneway=yes
came from INEGI data. It is clear from Bing imagery that the two ways should be connected, yet they are not. Cygnus has a (tweakable) distance threshold it uses when it decides if two ways should be connected or not. In this case, the INEGI way was too far away, so it remained disconnected.
There are a few other things to consider when you work with a Cygnus-produced change file:
- Cygnus will never degrade existing OSM data
- When both OSM and the enhancing layer have the same way, Cygnus will always keep the original OSM geometry, but it will optionally import
name
and other useful tags.
We are currently working on an import plan for INEGI data that will make heavy use of this new technology. More about this very soon! In the mean time, watch our SOTM US talk on the INEGI data and OSM in Mexico.
Discussion
Comment from Vincent de Phily on 30 July 2015 at 08:13
Looks great :) I can see that tool being used in plenty of other places. Time will tell how much manual post-processing is required.
Comment from CENTSOARER on 30 July 2015 at 20:38
Amazing tool! Congratulations! I have two questions not too related each other:
1) Is it implemented as a JOSM extension? 2) Considering we are talking about a Telenav work applied on open data and OSM, is Cygnus free and/or open source?
Thank you for this great work and I am waiting for your answers.
Comment from TOGA on 31 July 2015 at 22:27
Not sure, but at the moment I see two problems:
So, just based on the given example an import of INEGI data would probably need a lot of manual checking or you’d end up with something like TIGER all over again. Cygnus might be a nice tool, but it definitely seems to rely on the quality and completeness of the dataset. If that’s not the case, it just fails as shown above.
Comment from Geonick on 8 August 2015 at 07:46
This software could be very helpful in many use cases. I assume Cygnus is a JOSM Plugin. Is it possible to test and possibly use it?
Comment from richlv on 17 December 2019 at 17:01
The images have gone missing here… Can they be restored?
Comment from mvexel on 3 June 2022 at 14:12
@richlv sorry this took 2.5 years but yes, I restored them :/
Comment from richlv on 6 June 2022 at 15:57
Ahh, that’s great, thank you :)