Recent diary entries
Out of curiosity, I pulled some statistics on the US Rail network. This does cross into a bit of Canada and Mexico where the GeoFabrik extract approximated the boundary.
level_crossing TOTAL 232167 crossing TOTAL 8231 Rail_Bridge 47232 Rail_Tunnel 16394 Highway_Bridge 70811 Highway_Tunnel 8972
Total Layer Crossings 143409
A bridge or tunnel is counted as a single occurrence no matter how many rail lines are included. Each marked crossing node is counted, so a fully mapped rail yard could contain many crossing nodes.
For a closer look here is a breakdown of the top 20 categories of level_crossing - the first column is the type of 'highway', and the second column is the type of 'railway' in OSM:
residential rail 112659 service rail 29171 tertiary rail 26789 secondary rail 16717 unclassified rail 12732 track rail 10058 primary rail 6734 residential disused 1628 residential light_rail 1547 residential tram 1413 trunk rail 1122 tertiary light_rail 1114 secondary light_rail 1059 service light_rail 676 residential abandoned 660 primary light_rail 517 tertiary tram 493 service disused 435 tertiary disused 372 residential Unknown 371 residential preserved 353
A type of 'Unknown' usually means that a node that joins 'highway' to 'highway' is marked as type level_crossing for example.
Similarly, here are the top 10 'crossing' types:
footway rail 2984 footway light_rail 2083 path rail 881 cycleway rail 762 footway tram 565 path light_rail 116 cycleway light_rail 96 footway miniature 86 footway preserved 62 residential rail 56 service rail 34
Note the presence of 'residential' and 'service' which are either newly incorrectly marked 'crossing's or a rail crossing at the junction of 'residential' and 'path' for example.
To see the complete breakdown to perform custom category groupings, obtain the raw .CSV files from Rail Crossing Counts
Baltimore's Monuments to the Confederacy removed from OpenStreetMap after their removal by the City of BaltimorePosted by ElliottPlack on 16 August 2017 in English (English)
The Mayor and City Council of Baltimore acted swiftly after violence at the Unite the Right rally in Charlottesville, Va. led to a national conversation on the removal of Confederate monuments. The protests centered around the removal of prominent Confederate generals' monuments in that city. In Baltimore, the City Council passed a bill that permitting the removal of four Baltimore monuments to the Confederacy. The mayor executed the order by contacting local firms to remove the four Confederate monuments in Baltimore during the night of August 15, 2017, into the following morning.
OpenStreetMap is a database of physical features. Since the four monuments are no longer physically present, I have removed them from OpenStreetMap. The removed statues include the Roger B. Taney Monument, the Lee-Jackson Monument, the Confederate Soldiers and Sailors Monument, and the Confederate Women's Monument.
Weekend before last I spent several hours roaming Abadan and Gypjak, far western suburbs of Ashgabat, collecting street names, points of interest, and adding streets that didn't appear on the map (unplugging the SD card from the Garmin navigator forces it to collect tracks where you actually are, rather than trying to align them to the nearest street). Those towns, while not yet complete, are in much better shape in OSM now.
The research continued on names of monuments in traffic circles within the Ashgabat city limits. I count 22 monuments inside traffic circles and believe we now have names for all of them. If you are curious, enter "binasy, Ashgabat" or "monument, Ashgabat" in the OSM search box (without the quotation marks) and see what pops up.
Construction in Ashgabat is winding down as the Asian Indoor and Martial Arts Games approach. They start September 17 and the city is preparing.
This Friday, I will be at SOTM in Aizuwakamatsu, Fukushima this weekend talking about the state of validation on OpenStreetMap! I will talk about the need for making a validated error free map with OSM data, the recent efforts of the Mapbox Data team to review OSM changes and what the future of data validation might look like. If you are interested to attend the talk, grab a seat in the main hall at 4:10pm on Friday at the Aizuwakamatsu City Culture Center.
Distribution of reviewed and validated changesets using OSMCha in 2017 | View Interactive Map
There's recently been a thread on the talk-gb mailing list where someone has decided that, despite previous custom and practice there, the "name" field in both English- and Welsh-speaking areas of Wales should be a compound of both the English and Welsh names. No-one says "I'm climbing up Snowdon / Yr Wyddfa today", they'll use one name or the other, not both together.
In the Welsh-speaking areas the Welsh names are more likely to be used; in the English-speaking areas the English names. It's not a hard-and-fast rule; this peak in the Black Mountains is referred to about equally by both the Welsh and English names, despite it being in a predominantly English-speaking area.
Wikipedia gives an idea of Welsh-language take-up here. That's a bit broad-brush; for example I don't think there's an isogloss between Carmerthenshire and Swansea where people gain/lose the ability to speak Welsh.
So how is it possible to extract data from OSM with the Welsh name in the Welsh-speaking areas and the English name in English-speaking ones, both when creating e.g. a rendering database for the first time and when updating it as people update OSM? Firstly we'll just consider the "loading the database" part.
Very roughly, the Welsh-speaking area of Wales corresponds to this area. That's not perfect, but it's not a bad approximation for a rectangle. I downloaded the latest Welsh data from Geofabrik and cut that area out of it:
osmosis --read-pbf wales-latest.osm.pbf --bounding-box left=-4.82 bottom=52.02 right=-3.34 top=53.69 --write-pbf wales_cy_before.pbf
Convert the "Welsh-speaking" part to names based on "name:cy":
osmosis --read-pbf wales_cy_before.pbf --tag-transform transform_cy.xml --write-pbf wales_cy_after.pbf
Create a copy of the larger file with names based on "name:en":
osmosis --read-pbf wales-latest.osm.pbf --tag-transform transform_en.xml --write-pbf wales_en_latest.pbf
Merge the two together (do it this way around and the "Welsh" file seems to take precedence):
osmosis --read-pbf wales_cy_after.pbf --read-pbf wales_en_latest.pbf --merge --write-pbf wales_merged.pbf
osm2pgsql --create --slim -d gis -C 2500 --number-processes 2 -S openstreetmap-carto.style --multi-geometry --tag-transform-script ~/src/SomeoneElse-style/style.lua wales_merged.pbf
The "osm2pgsql" command to use obviously varies depending on the data and the style used; I'm using this lua tag transform (that's unrelated to the "osmosis" tag transforms described above) and this map style.
Edit: There's an automatic script to do this (for the style I use) here.
For those that have check my previous tutorials, are aware of the benefits of using Mapillary imagery (especially the traffic signs detections), however some crosswalks don't have traffic signs nearby. By using Mapillary AI we are able to detect them fast and add them to OSM. Here's the video tutorial how you can use AI Detections for OSM.
Find out more about Mapillary AI Detections
The end of the 2017 Google Summer of Code is getting closer. In the last weeks I was working on styling the map.
The most challenging task this month was definitely the anti-aliased ways, and I ended up doing that with the help of barycentric coordinates. I also added borders to areas and paths. I did a mistake previously, and not all buildings were visible, but I have fixed that issue, and now small buildings are also visible.
There is still some work to do, I intend to concentrate on the obvious things in the next day. Looking at the pictures below one can see that dotted/stippled lines are not yet supported (in the first picture the tiny red lines should be dotted). Fixing the looks of the texts also have high priority. The size is not quite right, and it is blurry. I also want to add support for patterns before the GSoC ends.
I was wondering how to revert an edit made on political bias. This user deleted the majority SeaWorld Orlando but the rest of their edits look to on the up and up. Change set: 50681538
Chris Barrington-Leigh and I have been working for the past few years to assess the completeness of the street network in OSM. We're pleased to have now published our results in the journal PLoS ONE. Thanks to many suggestions from the OSM community on our preliminary analysis.
Here are the highlights from the paper's abstract:
We find (i) that globally, OSM is ∼83% complete [as of January 2016], and more than 40% of countries—including several in the developing world—have a fully mapped street network; (ii) that well-governed countries with good Internet access tend to be more complete, and that completeness has a U-shaped relationship with population density—both sparsely populated areas and dense cities are the best mapped; and (iii) that existing global datasets used by the World Bank undercount roads by more than 30%.
An update using the April 2017 snapshot suggests that completeness is now ~89%. Our more detailed results and all our code are available on GitHub. Here's a sample of the largest 10 countries (updated through 2017), showing the actual growth in OSM road length and our model fits:
And here's the fraction complete, also updated through April 2017:
out there is trying to run Vespucci 0.7.0 which dates back to April 2011 on a Xiaomi Redmi Note 4 with Android 6.0.
While the app will run, 0.7.0 was released a while before 64bits for OSM element ids became necessary (early 2013), and trying to parse current data will lead to a crash.
With other words: please upgrade to a current version!
Again a bit of prototyping today.
I always wondered what the best way to download all objects along some GPX track (more generally a list of lat/lon pairs) in Overpass API might be.
Usually, you don't want to create some complex poly-filter for this, or even worse upload your gpx track as OSM way just for the sake of finding out what's close by.
You might have come across the
around filter, which comes in handy to find objects around a center point. I have extended this idea a bit to also handle linestrings.
Here's a first result: a turquoise circle in the middle is our great made up GPX track, consisting of about 20 points. All the yellow stuff in the background is what Overpass API returned as highways up to 500m away from our circle. Obviously, that circle doesn't exist in OSM, so we're really looking at what's close to our virtual way.
Live demo: http://overpass-turbo.eu/s/qX9
Imagine you want to create a map with only minimum features (motorways, some important nodes), and some place names along your own lat/lon pairs. Here's an example for such a trip from Frankfurt to Basel, downloaded as 10MB PBF file, processed via osm2pgsql and rendered by Kosmtik:
Live demo: http://overpass-turbo.eu/s/qXI
Github issue: https://github.com/drolbr/Overpass-API/issues/418
Always wanted to make a nice railway map style. A well spent 20 mins before bed.
A turn-restriction defines restricted or mandatory turns at a junction and are one of the most important features to map for accurate driving directions. Thought I would quickly make a comparison of how different map data editors visually represent this feature for mappers. Pick the best:
Add me now ! Cwassim87 👻👻👻👻
Yesterday we had our anniversary OSM Time meetup, where we celebrated 13th years of OSM, and we also had a workshop for the participants, to learn how they can extract information's from OSM, using overpass API and overpass-turbo.
Attaching 2 slides examples :
You can find the slides here
You can find the materials for the participants here
Malaria in Zimbabwe - HOT
Many organizations have collaborated together in the “Malaria Elimination Mapping Campaign”. This campaign seeks to create data to help eliminate malaria in affected countries in Africa, Asia and Central America.
Mapbox has helped in various ways with visualization, coordination, communication and validation of building data. Validation was done to improve data quality in the areas where these tasks were activated.
The "To-fix" tool is used to detect errors and we use Digital Globe and Bing satellite imagery to confirm mapped buildings.
You are invited to use "To-fix", it is very easy to use and easy to download.
List of detected errors:
- Overlapping buildings
- Crossing buildings
- Buildings that do not exist in the DG or Bing Imagery
Buildings that do not exist in the DG or Bing Imagery
How to build to-fix task:
- Osmlab/Osmlint is github repository where we have an open source suite of js validators for OpenStreetMap data, to identify common geometry and metadata problems at scale.
- Clone this repository to our local machine.
- By following the instructions in the readme, download the required mb tiles from the OSM QA TILES
- By running tile reduce commands, necessary validators and filters, We can generate geojson.
- The Final geojson is used to load in to-fix and create the task..
When “To-fix” is activated in JOSM, it automatically directs you to where the error is located, and points to it with pink circles.
Once located the error is indicated in the To-fix window, as is the case:
- Fixed → fix the problem
- Not an error → is not an editing error
- Skip → I can not solve the error
This tool can be downloaded for JOSM: Preferences > Plugin > find: "To-fix" > download or consult wiki.openstreetmap.org
This GIF shows how to work with this tool, is very simple and fast.
How to use the To-fix Plugin in JOSM
Added 4,377,776 buildings and validation began on May 16th, with 18 members of the Mapbox data team. We validated more than 17,000 errors. We are continuing to validate buildings in the countries of Zimbabwe and Zambia.
Malaria in Zimbabwe - HOT
Muchas organizaciones han colaborado juntos en ”Malaria Elimination Mapping Campaign”, esta campaña busca crear datos para ayudar a eliminar la malaria en paises afectados de Africa, Asia y America Central.
Mapbox ayuda de diversas maneras con visualización, coordinación, comunicación y validacion de edificios, Se realizó la validación para mejorar la calidad de los datos en las áreas donde se activaron estas tareas.
Se utiliza la herramienta “To-Fix” para detectar errores y utilizamos imagenes satelitales de Digital Globe y Bing para confirmar los edificios mapeados.
Lista de errores que detectamos:
- Duplicado de edificios
- Cruce de edificios
- Edificios que no existen en imagenes sateliateles Bing o Digital Globe
Duplicado de edificios
Cruce de edificios
Edificios que no existen en imagenes sateliateles Bing o Digital Globe
Cómo construimos una tarea en "To-Fix":
- Clonar este repositorio en nuestra máquina local.
- Siguiendo las instrucciones del archivo "README", descargue los archivos MBtiles necesarias de OSM QA Tiles
- Para correr los comandos de tile-reduce, validadores necesarios, Nosotros generamos un Geojson.
- El Geojson final es subido a to-fix para crear la tarea.
Flujo de trabajo de validación:
Cuando se activa esta herramienta, automaticamente te dirige hacia donde se ubica el error, y lo señala con circulos color rosado.
Una vez ubicado el error se indica en la ventana To-Fix, segun sea el caso:
- Fixed → se soluciono el problema
- Not an error →No es un error de edicion
- Skip →No puedo resolver el error o no dudas en el error
Esta herramienta tiene que ser descargada JOSM: Preferences > Plugins > Buscar: “To-Fix” > Descargar o consultar wiki.openstreetmap.org
Este Gif muestra como se trabaja con esta herramienta, es muy sencilla y rapida.
Cómo utilizar el plugin to-Fix en JOSM
agregó 4.377.776 edificios y validación comenzó el 16 de mayo, con 18 miembros del equipo de datos de MapBox. Validamos más de 17.000 errores. Continuamos validando edificios en los países de Zimbabwe y Zambia.
Are you bored of looking at your neighbourhood in OSM, trying to find the last unmapped tree, the last sidewalk still left to be drawn? Are you tired of trying to come up with even more complex relations so that landuse near your town is perfectly represented with multipolygons? Are you getting cross-eyed with the galaxy of POIs in your nearest city centre?
Wouldn't you prefer the thrill of being the first person to put entire villages and towns on the map? Discovering hundreds of kilometers of new roads, never before touched by OSM? Or relaxingly drawing completely rivers and canals, shooting straight across open plains?
If only the frontier was still open. If only there was such a place of vast possibilities. If only there was somewhere where life, pardon me, mapping, was still simple, honest, and true. Well, fear not! Such a place does exist. Come join us over at WikiProject_Iraq and talk-iq and claim your place among the pioneers.
Just look what you can do. No features visible on these images are drawn AT ALL:
The group of railroad crossing challenges listed in the previous diary entries is now complete. Many people worked on these challenges to make this happen. The tasks seemed to get more difficult as the challenge neared completion, since each task would correct a single crossing at a time.
Many of the tasks involved original TIGER highway crossings with poor alignment. I also worked on geometric alignment of the crossing approach roads to match the aerial imagery. Although almost none of these would be cross-checked against GPS traces, Bing imagery is almost always a factor of 10 more precise than the worst of the TIGER data. The resulting edits may have an effect on "TIGER desert analysis" - the study of large untouched areas in the US. Those "TIGER deserts" still exist but would be smaller. My personal contributions heat map now mostly shows areas in the US where there were rail crossings that needed fixing:
I have fond memories /nightmares of a task landing in West Virginia or Kentucky, and realizing that the only way I could fix the crossing called out by the task would be to untangle the surrounding 30 miles of rail and roads.
The challenges took a total of 10 months to complete. I attempted to keep the quality of tasks relevant by re-running the analysis every week. Thus any fixes completed outside of MapRoulette or edits of surrounding crossings would automatically be marked complete in MapRoulette.
In the role of the "Monday Morning Quarterback", (with perfect hindsight) there are things that ideally would be done differently or included in a more detailed task definition page:
- Separate tasks into additional challenges to reduce the time needed to analyze each one. For example, a 'duplicate rail crossing nodes' task would define a work flow even if there is already a visible 'X'.
- Create a video tutorial for the most popular editors to show the workflow for all the common tasks and how to fix. Even having screen shots of how to identify most common US bridges and tunnel types would help.
- I was able to concentrate on tasks marked as Skipped or 'Too hard' by querying the API. If the crossing was completely invisible from aerial imagery or alternate aerial imagery such as being under a bridge that might be 3 layers, the final task completion was to remove the task and leave an OSM Fixme or OSM Note. It would be useful to add a preference for those task status types in MapRoulette.
- Define a helpful sequence of alternate imagery: latest TIGER, NCOneMap (for North Carolina off-leaf imagery), use of SRTM Elevation to detect the difference between a tunnel entrance VS barriers on a decommissioned highway crossing, USGS Large Scale Imagery / NAIP for latest imagery for road realignments.
- Define a sequence of analysis when encountering a crossing that looks correct: was it recently fixed by someone else, or is the problem difficult to see? (1. Try a node "J"oin in JOSM. 2. Check date of last Crossing node edit - mark as already fixed if corrected in the past week. 3. Check the last edit date of both the highway and railway - mark as already fixed if corrected in the past week. ) The trickiest of these was a bridge - road crossing where all edits were much older than the last OSM synchronization. The problem was a 'duplicate bridge', two 2-node ways where one 'bridge' had no layer or bridge tag.
As a side note about the internal detail of MapRoulette, it seemed as though the Postgres SQL RANDOM statement was not truly random - as though there was an internal optimization or spatial caching of a previous RANDOM statement. My only evidence of this was that when encountering a 2-track crossing on a task where both level crossing nodes were corrected for the first task, the second crossing task would appear after only a small number of other tasks.
Because OSM is constantly being edited, there are now new unmarked crossings identified. I have not extended the Rail Crossings challenge because many of them come from new construction where there is no imagery to determine whether the crossing should be a level crossing or bridge. I may add these if I can find a way to ensure that just the solvable crossings are identified.
The end result of these challenges is that the Rail-Road network intersections in the US are much more accurate. They could serve as a reference to routing apps that can generate an alert for level crossings. This application has one case that doesn't fit easily in the OSM tagging convention: busy tram crossings in cities, where there may be many crossing alerts generated over a short distance. Although they are actual rail crossings that would result in serious damage in an accident, continuous alerts are a case of 'crying wolf'. This was briefly discussed in railway=level_crossing with in-street trams? , but without any definite resolution. For now, the application developer also needs to examine the crossing rail type to screen out tram crossings - probably not an ideal solution.
Pic4Carto is a great tool for finding lot of urban features using street-level pictures. I was able to add some vending machines remotely, in Aizu-Wakamatsu (where the next State of the Map 2017 will happen). This map (made with MapContrib) shows all the machines in the city.
You might know that soon the annual OpenStreetMap conference will take place in Aizu-Wakamatsu. As some other lucky mappers, I'm going to this amazing event (thanks to the French OSM association) and give a talk about a tool I've made called Pic4Carto. This post is a real-world example of what can be achieved using this tool.
I've been last year to Tokyo, it was such an amazing trip. Japanese people and culture are so great. I was amused by the amount of vending machines you can find in the city. Here in France, we have only a few vending machines, and most of them are indoors. So seeing a bunch of machines pretty everywhere was really fun.
As I will talk about Pic4Carto, I thought it would be great to do a real showcase of the tool with these vending machines. But first, what is Pic4Carto ?
Pic4Carto ( http://projets.pavie.info/pic4carto/ ) is a web viewer of all open-licensed, street-level pictures available online. It was created to make re-use of pictures from Mapillary, OpenStreetCam, Flickr and Wikimedia Commons easier for mapping. In fact, you can easily find on these pictures a lot of urban objects, like fire hydrants, benches, shops, speed signs, and... vending machines !
Add all the vending machines !
So, the idea of the showcase is to see how you can map remotely urban objects using Pic4Carto and other tools like MapCraft. Before I started, there was already some vending machines in Aizu-Wakamatsu. 33 nodes to be precise (thanks Overpass Turbo !).
Meanwhile, I can see that there are a lot of recent pictures (last 6 months) using the Pic4Carto main map, which shows picture availability statistics in a city (just wait two seconds for stats to load).
So we're good to go: a theme, a good source of data, and all the tools ready to do some efficient micro-mapping !
Pic4Carto + MapCraft = <3
In order to make what I call exhaustive mapping, meaning map all objects in an area using all available pictures, we can use the power of both Pic4Carto and MapCraft. MapCraft is a tool for work in team over an area, as you do in Humanitarian OSM. You provide to the tool a set of geometries, each one will be handled by one contributor.
Good news, you can directly export the Pic4Carto main map grid to use it in MapCraft. Use the export button in the top left corner (see below), and save the grid as "OSM XML" format on your computer.
Then, in MapCraft, you can create your own "Cake", which is mapping project. Just log-into MapCraft, then you will have the menu "New cake". There, you have to supply some informations: project name, description, and geometry. The geometry file is the grid we just exported from Pic4Carto.
Then create it, and here you are ! You have your freshly-baked cake ready to use. Here can start the productive work: each one of your team mates can take a slice of the cake, and start mapping over it.
Adding vending machines
Now that all the tools are ready, we can do some productive mapping. As I was working alone on this project, the MapCraft cake is just a way to track which pictures I already have looked at. The process is the following:
- Take a slice of MapCraft cake (set yourself as owner)
- Go back to Pic4Carto, on the same cell
- Watch every picture, and when you see an interesting object, click on the Edit button (either with iD or JOSM)
- Add it on the editor. A good practice is to link the picture to the added object using either mapillary=*, flickr=*, wikimedia_commons=* or image=* tag depending of the picture provider.
- Then watch other pictures, until you've done the cell
- Go back in MapCraft, set the progress of the slice to 9 (which is 100%), and remove yourself of owning the cell
Not so hard, eh ? I think it's a good way to spend time when weather isn't good in your city, it lets you discover amazing places and improve our database.
So you have wondered that all this can work for pretty any kind of objects: street lanes, benches, traffic signs, shop, amenities, building description... You can use pictures for mapping whatever you like, and it have never been that easy. You can work alone, or in team. You have access simply to all available street pictures of the world.
Oh, what about these vending machines ? I created a map showing them (using MapContrib which is a great online editor for new mappers), and you can add the ones you know in your city directly there: https://www.mapcontrib.xyz/t/8a6e32-Vending_machines
I hope that this story was interesting to you, and will inspire you for doing some remote mapping, or uploading pictures to Mapillary or OpenStreetCam ! And if you go to the State of the Map this year, see you soon there ! :-)
Want to go further ?
Came across a brochure today naming some previously names-not-known monuments in Ashgabat, so did a little ground truth to verify them and entered their names (no, the brochure does not bear a copyright!) Also added a kindergarten, updated some streets, added one (1) street name discovered while exploring a neighborhood to discern changed traffic patterns. There are still street names to be discovered but these days instead of collecting a dozen or two dozen on a trip, I'm lucky to find one new street name.