Recent diary entries
I've just finished a cleanup and validation of field survey data collected in Bangassou, Central African Republic as part of the Missing Maps Project. I drafted a procedure for the global validation of OSM projects. I would love some input from others who have completed similar projects so I can refine the procedure and publish on the OSM Wiki.
Global Validation of a Project Area
- Define the area to be validated – this will ideally be predefined in a gpx file
- In JOSM, open the gpx file containing your area border
- Zoom in to an “acceptable” size area and click File-->Download from OSM. Make sure the “Download as new layer” box is checked. There will be a message in the bottom right corner of this screen telling you if the area selected is too big for the server. Sometimes it will say the size is okay, but you will get a server error when you try to download – this means that the area was too big and you need to try again with a smaller area.
- Once this is downloaded, go to a different area within your border and repeat with the “Download as new layer” box UNCHECKED. Do this until OSM data is loaded for your entire area.
- Go to Imagery and select the appropriate provider for your project (usually Bing)
- Scroll across the map and trace broad areas of problem imagery (low resolution or obscured by clouds). Tag the problem, the imagery provider and the date, like "name=low-res Bing image border as of August 4 2015"
- Check major roads for tagging consistency. Refer to the OSM wiki for guidance on tags. Highway Tag Africa and individual country pages are helpful.
- Look at the list of villages by going to Presets-->Search for objects by preset-->Geography/Places/Village
- Scan the list to identify naming issues, tagging issues, duplicates, or other errors. For example, you may see “Village” as part of the name, a village split and labelled 1 and 2, spelling variations for the same village, or overt spelling mistakes.
- Repeat for other features as needed, also checking the Tags/Memberships panel for consistency
- Make sure all villages have limits traced using landuse=residential (not name=residential) and that the whole village area is a single object rather than having neighborhoods traced separately.
- Run the validation tool within JOSM and address the problems it finds
- Upload changes to OSM
I have always wanted a map style that can be printed on a single A4 sheet that I can keep in my wallet anytime I need a quick reference for cycling or using the bus across the city. India lacks any maps in public spaces and I rarely carry a phone, so this would be quite handy.
Tried to skin OSM last night with this in mind and I think i've got off to a good start. The roads are styled similar to a traffic map to give me an idea of Bengaluru's complex oneway system. The labels are prioritized for suburbs, neighborhoods and streets.
Did a test print and pretty happy with the grayscale result. Need to tweak the colors a bit more though to make it more visually appealing.
I live next to a hotel, De Pluimpapaver. I frequently get visitors asking me where to park their car. I marked the parking of the hotel on the map. Let's hope people will find it now :-)
Just started using this, mainly because my NEXUS 7 only has GPS and WiFi, and I needed an offline map for Venezuela. Now I am editing my part of Caracas ! Awesome work here, glad to be part of this...
Where do we leave our Garbage?
Taginfo is a great tool to see where and how a given key is used on the map. It also gives you some nicely formatted tables with statistical data of all the tags (a tag is a key=value pair).
Did you know that the most used key is source=*?
On 25 july 2015 it appeared 162.428.193 times with a total of 143.491 different values. You can find the most common tags here.
But I was more interested in keys that appear just once in the database, because I expected many of those "solo" keys to be erroneous. To find that out, I downloaded the taginfo database (there is a link on the taginfo page to do precisely that). Be warned: after downloading and expanding that database, you have a 5 Gb file in Sqlite format to handle! But doing so, I could do my research with more details and faster than using the tools on the taginfo site. I opened it in my Sqlite client and after 10 minutes:
I had 74.569.089 records on my computer to research!
For every record in the database you have numerous tables with information available, one of them gives the information of all the keys that are in the database:
and this helped me to find what I was looking for: the count_all field.
Below is this "keys" table with the first 20 entries: The count_all field is the one I needed and after the necessary code I produced a table with all the keys that appeared once (had a value of count_all=1).
Here is the beginning of that table, after I sorted it (well, my computer was so kind to do it for me!). Rather weird names for a key, don't you think? What would "+++" denote? Or "129/"?
The first entry (source:name) starts with a space character! If you enter that string (or any of the others) into the search field of taginfo, you can get all the details about that key: its value, how many times used (1) and where it is used if you click on the tab "map". You can also click on the overpass link to see its precise location.
So, for that first entry I did all that and it turned out to be this: somewhere down-under in Australia. Try it for yourself!
Keys in a database are not supposed to start with a space character, but the OSM database accepts anything and does not do any check on what you enter, save for the length (max 255). Also keys are supposed to contain alphabetic characters and may contain digits as well. Some special characters are allowed also, like e.g: "_" and "-".
But a key with just numbers? What is that? Let's see for 09200: It seems to be the postal code for a village in France. But then it should have been:
A lot of the (faulty) keys I found are of the uppercase/lowercase type:
Name when name was meant for instance. Almost any regular key (amenity, shop, tourism, highway, landuse etc) appears in a misspelled version in the database (tourims, land-use etc). Also added interpunction (name; or name, or name-) counts for quiet a number of those one-time-only keys.
All in all 19.037 keys appear only one time in the database and out of a total of 54.382 keys, that is more than a third!
Not all of those keys are "wrong", but too many of them are, and will never be used again.
Is that a problem? Not really I think. It does not consume very much of disk/memory space, certainly not if we compare that to the huge amount of data that is also in the database. But sometimes it leads to unexpected results with software that consults the database.
So, the answer to "Where do we leave our Garbage" is: just where it is.
But if you ever come accross such a situation, please correct it and remove what is not necessary or redundant.
Along Umeälven there is a cycleway which was broken off some years ago due to constructions of new buildings. There's also a waterway=stream there that has peaked my interest earlier due to a totally failed urban exploration adventure. The stream begins on the other side of Botniabanan and comes out towards Umeälven.
It was a long time ago this new way route was built and it really feels good to write it off from my checklist of "things I should probably do sometime".
I have fun mapping even though I cannot see the place because of the clouds covering the place. But I put some notes and comments about my mappings. This is a good tool which is very useful in plotting places and the institutions around the place. It makes us knowledgeable on how to make an accurate details on the place. This mapping is very useful on different purpose especially during calamities.
I had fun this day mapping my location. Locate my own house and start a new trace? Yeah, it's funny but cool thing to trace roads and some unknown streets. Maybe this could help somehow :) :) smile and :)
OpenStreetMap in 2007 and today...
Today is the day that I put my house in the map using this sites. This is really great! cool experience.
having fun mapping my place this time..
- Change Korean (English) style nodes to use proper properties (not sure if this is correct term)
- Change zipcode (5 digits since 2015-08-01)
http://wiki.openstreetmap.org/wiki/Lanes http://wiki.openstreetmap.org/wiki/Proposed_features/lanes_General_Extension http://wiki.openstreetmap.org/wiki/Key:lanes http://wiki.openstreetmap.org/wiki/Key:turn http://wiki.openstreetmap.org/wiki/Key:destination http://wiki.openstreetmap.org/wiki/Relation:waterway http://wiki.openstreetmap.org/wiki/Relation:destination_sign
I want to share with you my blog post about mapping turn lanes
If you have any suggestions, you can write it here in comments
Mapping turn lanes in OpensStreetMap
Complex intersections often involve lane-specific turn restrictions. See for instance these overhead signs on Utah State Route 92, crossing Interstate 15 just south of Salt Lake City.
Lane signage on Utah State Route 92. Photo: Garrett.
We need turn restrictions for every individual lane to provide precise directions for drivers. In OpenStreetMap we model turn lane information with two tagging schemes:
Key:turntags to map indicated directions
type=turnlanes:turnsrelations to map turn restrictions on lanes
Here is a guide on how to map turn lanes with OpenStreetMap's JOSM editor. Note that
type=turnlanes:turns is a proposed feature. You can join the proposal process by submitting your review on the relevant talk page.
Mapping Turn Lanes
As an example, let's look at how to map turn lanes in detail on this intersection alongside US 101 in South San Francisco.
Lane-specific turn restrictions along US 101 Exit 425B.
Enable the "Turn Lanes" panel in the "Windows" menu.
Identify the number of lanes on all the roads leading to and from the junction.
Add the number of lanes to the appropriate ways as a regular tag - for example,
Split the ways that will be parts of relations. Then, for each turn lane restriction, select the nodes and ways involved to define a rule. You should see the junction modeled as shown once the lane count for all the roads have been set.
Let's start with the motorway link from US 101 leading into the junction.
exit_to=Oyster Point Boulevard;
Initially this way has one lane which splits into three before the junction. At 141 meters before the junction, add the left lane. At 92 meters, add the lane on the right side. These lanes can be added by clicking the white plus button and setting the start position by dragging the blue marker back from the junction.
Add a lane to turn left for a small section of Dubuque Avenue. You can pan the model by dragging with the right mouse button.
Now create rules for the thoroughfare from motorway junction #425B with rules for each lane by dragging a route across the relevant junctions and lanes.
Add similar rules for all directions where applicable.
Now let's check our work by selecting nodes and reviewing traffic flow. To cycle through all directions press
Ctrl+A for an overview of all restrictions on the junction.
I hope you find these instructions useful to map detailed junctions in your area. If you have any questions or ideas on mapping turn lanes, drop me a line on Twitter or through my OpenStreetMap profile.
Header photo: Interstate 10 and Interstate 17 Interchange at Night by Alan Stark
I am working on improving the data availability for some of the major northern Canadian arctic communities, as improving this data in OpenStreetMap will also improve the Arctic Web Map project that is using OSM data.
I have downloaded the shapefiles from NRCan and I used
ogr2ogr to convert them from EPSG:4140 to EPSG:4326. Then I am importing them a layer at a time using JOSM. I am using the CanVec Feature Catalogue and OSM CanVec Feature Guide for mapping values.
For the vegetation layer "085f05_11_0_VE_1240019_2" (Wooded area classified — polygons) I imported the shapefile into QGIS and used a dissolve operation to combine all the polygons. Then a Douglas generalize filter with a tolerance of 0.0001 was applied to remove redundant vertices. This reduced the vertex count from 363,000 to 40,000. Next I opened that shapefile in JOSM and changed the tags to
natural=wood and manually merged the polygons with identical polygons in the neighbouring imported tiles.
I will continue importing one layer at a time until this NTS tile is complete.
Mexico released a huge amount of open data not too long ago.
A lot of this data is geospatial, so I say yummie! Alex Barth wrote about this data, that comes from the Mexican national statistical agency INEGI, on his diary before, with a nifty map to show how rich this data is:
(My mediocre animated GIF skills really don't do it justice - check out Alex's blog post to see an interactive map.)
So now the question becomes: how do we get some (or all?) of this data into OSM? This is not straightforward - OSM already has rich data in many places in Mexico we would definitely want to keep.
Here at the Telenav OSM team, we have come up with an answer to this question. We call it Cygnus - The Bringer of Balance. Let me explain in a few visuals what Cygnus does.
Consider this area in the Aguascalientes region. There is some OSM data there:
If we look at the Bing aerial images, we can clearly see that there's an entire village there that is not mapped though!
But INEGI has most if not all the roads in this village in their now open dataset Conjunto de Datos Vectoriales de Carreteras y Vialidades Urbanas.
After converting the original data attributes mapped to OSM-appropriate tagging, saving the result as an OSM file, and loading it into JOSM, it looks like this:
OK, that is nice, but we still have two separate layers that are unconnected, and even if we merge them, we will still have to manually resolve duplicate ways and connections between the ways from OSM and those from the INEGI data.
This is where Cygnus comes in - a new conflation technology we developed at Telenav specifically to tackle this.
Cygnus, as the Bringer of Balance, takes as its input a base OSM file in PBF format, as well as what we call an 'enhancing' file, also in PBF format. It will then compare and conflate the two inputs and output one JOSM XML file that can be merged with OSM base data straight away. This detail shows the merged layers with all the INEGI 'imported' ways connected to the pre-existing OSM way:
Even though Cygnus does a pretty amazing job merging OSM data with an 'enhancing' layer, you will still need to check the result before you upload. Take this example here:
highway=secondary was the pre-existing OSM data, and the
highway=residential; oneway=yes came from INEGI data. It is clear from Bing imagery that the two ways should be connected, yet they are not. Cygnus has a (tweakable) distance threshold it uses when it decides if two ways should be connected or not. In this case, the INEGI way was too far away, so it remained disconnected.
There are a few other things to consider when you work with a Cygnus-produced change file:
- Cygnus will never degrade existing OSM data
- When both OSM and the enhancing layer have the same way, Cygnus will always keep the original OSM geometry, but it will optionally import
nameand other useful tags.
We are currently working on an import plan for INEGI data that will make heavy use of this new technology. More about this very soon! In the mean time, watch our SOTM US talk on the INEGI data and OSM in Mexico.
One day one of my phones has shut down on a trip after 80 kilometers which I will never do again. I have checked Locus where the tracklog stopped: a few kilometers behind. Started Tower Collector, it showed zero locations! Oh my god. I had another trip previous day which was not uploaded yet.
This device was rooted, started root browser, checked the application's data directory:
In directory "databases" I have seen an empty measurements.db (a few kB), but there was another file: measurements.db.back with 600 kB! Made a backup copy of the whole directory and resumed my route.
At home I have checked the db. It is an sqlite database, but has been corrupted.
sqlite> pragma integrity_check; Error: database disk image is malformed
I have made a dump of the database:
echo .dump | sqlite3 measurements.db.back > measurements.sql
There were 2059 measurements for 76 cells, valuable data. How to restore them? Load back to Tower Collector. I did not have time for that for weeks, but continued collecting new data. I was frustrated because this phone did not remember the cells collected before the crash, displayed a most of cells as new. I have also missed the data from that two trips. Some weeks later I have tried to load data.
At first I have created a new database using the recovered data but I have failed many times. I replaced measurements.db with the new one, app crashed (SQLiteCantOpenDatabaseException). Checked directory and file permissions via Root Browser, it showed root as owner of every file, but later I have checked the same thing via adb and it showed totally different owners: every app has a userid and owner should be set correctly.
After setting owner for the database, I had another kind of crash: Tower Collector said "upgrading database" and crashed again. Stack trace had an SQL statement: "CREATE TABLE cells" which failed because cells table already existed. Why does it upgrade? I did not see any difference in schema of old and current database.
Then I started a different way. Prepared only data from the dump: kept only INSERT statements for the following tables:
INSERT INTO "measurements" ... INSERT INTO "cells_archive" ... INSERT INTO "cells" ...
Uploaded this file via adb:
adb push inserts.sql /data/data/info.zamojski.soft.towercollector/databases/inserts.sql
Then restored the last working measurements.db and logged in via adb:
$ adb shell ~ # cd /data/data/info.zamojski.soft.towercollector/databases # sqlite3 measurements.db
Here I had an almost empty database, only cells_archive table had 1227 rows. I have truncated this to make history clean:
# DELETE FROM cells_archive;
Then I imported the SQL file just uploaded:
# .read inserts.sql
Logged out and tested: app displayed 2059 measurements for 76 cells. Quickly uploaded to OpenCellID and checked newly uploaded measurements there.
Then I merged the new cells captured since the failure. Dumped cells_archive from the previously backed up version of measurements.db, replaced table name "cells_archive" to "cells", loaded into the database using the same way and then deleted them. This database has a trigger which archives rows deleted from table "cells" to "cells_archive" ignoring duplicates.
I had seen 6176 cells before the crash. Since then there were 1227 new, merging resulted 6793 cells, 617 new. The rest were duplicates.
The last step was setting total number of locations to the sum measurements before and after crash:
UPDATE stats SET total_locations = 156440+29641 WHERE row_id=1;
Now I have a database which knows all cells and number of locations since I have started collecting data. I am happy.
I've been using Merkaartor on LInux for some years. Usually I import a track, zoom in, then use "download more" to download OSM for that region. But there's a 50,000 item limit, so if I'm not zoomed in far enough, it fails.
Years ago I had read that I could get planet.osm as a default start set, but I never did. Now, when I look, it's huge. Merkaartor can't handle that, at least not with the amount of RAM I have. It can't even handle all of Canada without swapping.
I see that the US is split into states, but all of Canada is one big file. Is there any easy way, preferably command-line on Linux, that would let me split out a region from the Canada OSM - one province, or a smaller area ?
How do we deal with multiple values for a key?
We all know this situation: you need to add a telephone number to a node and add the line:
Then you find out that there is a second phone number for that node, but you can't add a second phone= tag because OSM doesn't allow that.
The general question is: how do I tag multiple values for one key?
Let's investigate how mappers have solved that problem sofar. The screenshots all are made with OpenPoiMap.
This example is the Eiffel tower in Paris for which four architects worked together, but only one gave his name to the final product!
In the source the names are separated with semicolons:
Stephen Sauvestre;Gustave Eiffel;Maurice Koechlin;Émile Nouguier
One piece of art created by 5 artists (somewhere in Seattle).
Create a new key with a sequential number attached to it for every member of this group. In this specific case I would have started numbering with artist_number_2 because the first one is already in artist_name. Even better, I would have started with artist_name_1 and used it for Andrew Keating and would have omitted artist_name altogether.
This example is to show how to map multiple sets of related tags to one node. In this case we have man_made=mast which has attached to it 4 (mobile phone) antennas at different heigth and each working with a different technology.
An underscore could have been used as in the previous example, but there seems to be a tendency to tag situations like this with a key:N notation, where N is running over the natural numbers.
[4.] Here we see a combination of both methods. Adding a number (with underscore) at the end of the key to count them or adding the number after the colon. Again, in the case of the fuel I would have started with fuel:diesel:1 for the Biosolar.
Any different opinions on this subject?
is there somehow a possibility to visualize all GPS-tracks that were gethered for the same area? For example I use JOSM, I define my area of interest and for that specific AoI I want to fetch all available GPS-tracks. Hope that was clear!