Mapping Mexico with INEGI, you and...Cygnus

Posted by mvexel on 29 July 2015 in English (English)

Mexico released a huge amount of open data not too long ago.

Huge: Mexico's statistical institute INEGI goes open data @INEGI_INFORMA (via @rodowi — Alex Barth (@lxbarth)

A lot of this data is geospatial, so I say yummie! Alex Barth wrote about this data, that comes from the Mexican national statistical agency INEGI, on his diary before, with a nifty map to show how rich this data is:


(My mediocre animated GIF skills really don’t do it justice - check out Alex’s blog post to see an interactive map.)

So now the question becomes: how do we get some (or all?) of this data into OSM? This is not straightforward - OSM already has rich data in many places in Mexico we would definitely want to keep.

Here at the Telenav OSM team, we have come up with an answer to this question. We call it Cygnus - The Bringer of Balance. Let me explain in a few visuals what Cygnus does.

Consider this area in the Aguascalientes region. There is some OSM data there:

base osm

If we look at the Bing aerial images, we can clearly see that there’s an entire village there that is not mapped though!

base osm with bing

But INEGI has most if not all the roads in this village in their now open dataset Conjunto de Datos Vectoriales de Carreteras y Vialidades Urbanas.

After converting the original data attributes mapped to OSM-appropriate tagging, saving the result as an OSM file, and loading it into JOSM, it looks like this:


OK, that is nice, but we still have two separate layers that are unconnected, and even if we merge them, we will still have to manually resolve duplicate ways and connections between the ways from OSM and those from the INEGI data.

This is where Cygnus comes in - a new conflation technology we developed at Telenav specifically to tackle this.

Cygnus, as the Bringer of Balance, takes as its input a base OSM file in PBF format, as well as what we call an ‘enhancing’ file, also in PBF format. It will then compare and conflate the two inputs and output one JOSM XML file that can be merged with OSM base data straight away. This detail shows the merged layers with all the INEGI ‘imported’ ways connected to the pre-existing OSM way:

cygnus merged

Even though Cygnus does a pretty amazing job merging OSM data with an ‘enhancing’ layer, you will still need to check the result before you upload. Take this example here:

cygnus miss

The highway=secondary was the pre-existing OSM data, and the highway=residential; oneway=yes came from INEGI data. It is clear from Bing imagery that the two ways should be connected, yet they are not. Cygnus has a (tweakable) distance threshold it uses when it decides if two ways should be connected or not. In this case, the INEGI way was too far away, so it remained disconnected.

There are a few other things to consider when you work with a Cygnus-produced change file:

  • Cygnus will never degrade existing OSM data
  • When both OSM and the enhancing layer have the same way, Cygnus will always keep the original OSM geometry, but it will optionally import name and other useful tags.

We are currently working on an import plan for INEGI data that will make heavy use of this new technology. More about this very soon! In the mean time, watch our SOTM US talk on the INEGI data and OSM in Mexico.

Comment from Vincent de Phily on 30 July 2015 at 08:13

Looks great :) I can see that tool being used in plenty of other places. Time will tell how much manual post-processing is required.

Comment from CENTSOARER on 30 July 2015 at 20:38

Amazing tool! Congratulations! I have two questions not too related each other:

1) Is it implemented as a JOSM extension? 2) Considering we are talking about a Telenav work applied on open data and OSM, is Cygnus free and/or open source?

Thank you for this great work and I am waiting for your answers.

Comment from TOGA on 31 July 2015 at 22:27

Not sure, but at the moment I see two problems:

  1. All those oneway streets look highly suspicious to me. Just look for example to the north, there is a corner which basically is a deadend. Maybe you could examine the original data again, if these are really marked as oneways? If yes, I’m inclined to question the quality of this dataset.
  2. I’m not sure, if the automatic connections done by Cygnus are of any use, at least in this case. A quick look at bing satellite imagery shows that only the main street is linked to the primary in the south. All the other north-south orientated streets connect to a minor, unpaved road which parallels the primary and doesn’t even seem to be in the INEGI data.

So, just based on the given example an import of INEGI data would probably need a lot of manual checking or you’d end up with something like TIGER all over again. Cygnus might be a nice tool, but it definitely seems to rely on the quality and completeness of the dataset. If that’s not the case, it just fails as shown above.

Comment from Geonick on 8 August 2015 at 07:46

This software could be very helpful in many use cases. I assume Cygnus is a JOSM Plugin. Is it possible to test and possibly use it?

Login to leave a comment