Recent diary entries
I'll be participating in the Building Canada 2020 community-led initiative.The goal of this project is to map and digitize every building in Canada on OpenStreetMap by 2020. Creating a free publicly accessible, non-proprietary source of GIS information that stretches from coast to coast to coast.
There is an estimated 10-15 million buildings in Canada that need to be digitized in OpenStreetMap. Over 2 million Canadian buildings have already mapped!
I came across this while browsing the RSS feed, and I think I can recognise a certain website that is central to the humanitarian mapping effort.
After some experimenting and help from the Belgian OSM Community, I figured out how to tweak the parameters for the routing engine and steer the navigation process. It seemed as if the technical documentation was missing, so lets write it here.
The parameter file: routing.xml
All parameters for the routing file are stored in a single XML.
This document should be placed in a place where OsmAnd will find it. As I chose 'external storage' to store my maps (
general settings ->
Data Storage Folder (under miscellaneous title), adding this custom routing is easy.
Plug your phone in your computer and go to the path
Reading the parameter file
routing.xml is actually documented quite good! Read the docs there to get some general idea.
Adding an option to a profile
The parameters are split per profile. I edited the bicycle profile, as that is what I use most (also see my previous diary entry).
Over there, add a toggle, e.g., if you want to avoid sett surfaces:
<parameter id="avoid_sett" name="Strongly prefer asphalt and paving stones" description="Tries to route around sett and cobblestone, ideal if riding with a cycling cart" type="boolean"/>
id is the name you'll use in if-statements, the
name is what shows up in OSMAnd. Save the file as described above and restart OsmAnd. Start navigation to somewhere. If you click the settings (the cogwheel on the bottom when starting navigation), your new parameter should show up!
Routing considers three major aspects:
Each of those aspects has its own section in the xml; you can't miss them.
Access deduces if this road can be used or not. This is very hard: yes or no. If you want to create a profile to cycling with your toddler in the cart, and that toddler wakes up from the sett stones, you can add the following to avoid a condition in all cases:
<select value="-1" t="surface" v="sett"/> -- never ever enter a set road
Of course, you might want to avoid unknown surfaces too:
<select value="-1" /> -- Default: no access <select value="1" t="surface"/> -- only if a surface is known, allow access
And if you mix in the toggle, you'll get:
<if param="avoid_sett"> <select value="-1" t="surface" v="sett"/> </if>
The other aspects are tweaked similarly:
Speed is the maximal practical speed on a certain road. For example, if sett slows you down to 5 km/h; then you can tell the navigation:
<select value="5" t="surface" v="sett">
This is relative to other speeds. The default routing takes a default speed of
14.5 for cycling, so this would be way slower.
Priority is the preference of a road. If two routes are equal, the route with the highest priority is taken. However, as I modified both
priority in a similar way, it is hard to state the differences.
A sample use case
This custom routing can be tailored for your needs. See my other diary entry as an example and for the routes taken.
The 26th of september was an OSM-meetup in Bruges. The few guys which where there - and especially Joost, must have gotten some weird and extremely contagious disease, called 'mapperitis'. It has been the start of a growing involvement in the mapping efforts of OSM.
Falling in love
Although I've been using OSM as my navigation of choice for years now (prob. 5), it's only recently I've actively begun mapping. Altough my first mapping were the ski routes on holiday, that was a one-off - until now.
Using streetcomplete, I've been surveying the surfaces of Bruges, which contains - as you guessed it, a lot of sett (aka 'neat cobblestone'). In the meantime, I've been spreading that contagous mappiritis around as well - until it reached someone of the Fietsersbond. One of them had recently seen the talk by Escada and was wondering if it would be possible to map all cycling routes, their surface area and width, to test if they meet the vademecum. "Yes, this is possible" was my answer of course!
I followed by a short showcase of OSM, overpass-turbo and how easy it was to enter all this info. Although he was impressed, he was a bit scared of how much work it seemed to be - and that it is an older and not so tech-savy person didn't help either.
That was a week ago*. By now, the north of Bruges (the center) is surface-mapped, and around 50 cycleways are measured.
*(To be honest, it was two weeks. However, I was on holiday one week, so I couldn't map)
Routing around sett
In the mean time, I was confronted with another problem. Quite often, I'm cycling around with a cycle cart. These things are slowed down tremendously by sett stones. So I wondered, wouldn't it be possible to tweak the OSMAnd routing to avoid those nasty cobblestones?
After a few inquirements on the community riot channel, it turned out to be possible. And not only possible, it was easy as well!
So I added an option to bicycle routing - avoid setts if possible. Not a hard ban, but if a detour is acceptable.
For example, if I want to go from my home to my favorite youth movement, the shortest route is through the city - and over cobblestones. Not that great with the cart:
However, with tweaking the weight of sett, I got this route:
A really cool result. It seems like I have the only navigation system in the world that knows where all that cobblestone is and that actively avoids it...
Next step: promote OSM to the 'Bruhhelingen'.
And because I want to keep mapping, I even added a mode to prefer routes where the surface key is missing... Giving the following result:
In conclusion, a very usefull and extremely fun experiment!
If you want to create your own routing, I wrote a small getting started for OSMAnd here.
Some of you will know that I recently conducted a survey of OSM user demographics. The survey closed at the end of August. Since then I have been collating and cleaning the data ready for analysis. The survey generated a fantastic response and I would like to extend my thanks to everyone who took the time to participate.
As promised, sixty respondents have been randomly selected to receive a £15 Amazon voucher as a small gesture of gratitude for participating. I have now sent the voucher codes out via the OSM messaging system so please check your in box to see if you are one of the lucky recipients.
I hope to be able to share some preliminary results in the coming weeks and look forward to seeing exactly what the data reveals about the impact of gender bias in OSM contributing.
Due to China strict law of map concerns of China, stuff including military area on OpenStreetMap is not welcomed by China mapping law. Some China Mappers are thinking about removing sensitive military area on OpenStreetMap. One of the mapper FreedSky is censoring map stuff on OpenStreetMap, and start removing military landuse on OpenStreetMap
I revert the changeset 53508675 and changeset 53508564. It's the military area on Fujian, which is the standing opposite Taiwan.
In the previous experience deal with national security, military landuse in Israel and South Korean was removed by local mappers. But the the removed stuff was recovery when someone discovered and talk on the mailing list.
Which is important, the national security of certain country, or the global goal of mapping everything on the world?
Time : 11:16 AM (Bangladesh) Location: @my_students_home Mapping Location :#1075 - Bamako (Mali)
Hi diary, I am bored sitting and doing nothing !!! My student is giving exam so I thought of using this idle time to map @osm. I feel to contribute from beginners level and this mapping location seems fun. Good luck to me and to my student for her quiz !! ;)
Best Regards, Akhi Agatha Jetra
Open Street Maps for emergency services.
One of the issues to be dealt with during a tragedy is mapping information. When you setup emergency shelters, or if you are setting up mobile command centers it is important to share that information with everyone so everyone knows where resources are located.
I think that OSM can fill that gap for real time mapping updates for a changing situation.
Wondering what everyone else might think about that as well.
After the new law of 2014 for the administrative boundary it was a lot of work to split the border line to the new one.
A good help was this source:
- The official law
- Counties of Albania, wikipedia
- Municipalities of Albania, wikipedia
- Administrative Division of Albania, wikipedia
It was checked/added/updated the nomination (name), boundary line, zipcode, population, wikidata, wikipedia!
We are pretty close to overcome the 1M (mega) count of opening_hours values in OSM. Also the amount of values which are machine readable is increasing from day to day. Almost all of those 1M values (97.8 %) follow the specification which greatly extends the use case of opening hours.
Thanks to all of you! 10 million is not far now :)
I've been busy lately and haven't had time for much OSM mapping. But yesterday I went on a mt bike trip. I had planned the route so that I could check out some old quarries and a couple of possible paths as well.
The temperature was a couple of degrees above 0, but there was still ice on the small pools of water on the trail. Sometimes the ice was so thick that the bike would not simply go through but first skid on the ice and then go through. It was also very muddy and wet, so the biking conditions were quite tough. But if you go mtb-biking in Sweden in november, these conditions are pretty normal.
The first quarry had not been used for some time, but it was quite easy to find. It was used for stone material, not any metal extraction.
Shortly after visiting the first quarry, I put the bottom half of the bike inside a lake, to remove some of the mud from the gear. When I continued, the bike broke however. The bolt that hold the saddle to the seatpost broke. Of course I did not have a spare bolt with me, so I had to decide whether to call it a day or not. But since I don't often get the time to go out biking/mapping, I decided to continue even though it meant biking standing up for more than 20 km.
I could not go as fast as before, but it worked quite ok. After a while I found the second quarry. This was a granite quarry, and had been open in the 80s. The nice thing about it was that there were still lots of large blocks of stone stored inside the quarry, as can be seen on the pictures. I climbed a couple of the to take pictures, but did not stay very long.
On my way back home I mapped 2 paths I hade planned for some time. But the last kilometers in the forest were quite hard since I was quite tired after biking standing up. Especially the technical climbs were tough since it was difficult to maintain the right torque so that the bike would get up the climb without losing grip on the wet roots and rocks.
Even though I was a bit disappointed in the bike braking down, I was quite content with the mapping, and happy that I had decided to carry on anyway....
I had the honour of being "mapper of the month" for October 2017: http://www.osm.be/2017/10/23/en-motm-seppe_santens.html
Mapping location : Rajshahi, Bangladesh Date 11/1/2017 Time: 6:14
Sitting with my student at her room, no homework today Me: lets teach you mapping. My student: On google? Me: No, on OSM. Student : What's OSM? Me: Openstreetmap ! You can edit on osm , you can draw buildings, lakes etc. Student: wow ! Then lets do it.!! AND THE JOURNEY BEGINS.
We are pleased to announce the International Federation of Red Cross and Red Crescent Societies is joining the Missing Maps Partnership. Missing Maps is an open, collaborative project in which you can help to map areas where humanitarian organisations are trying to meet the needs of vulnerable people – creating open data that can be used for planning, awareness, and analysis. Over 33,000 people have contributed 33 million edits to OpenStreetMap since Missing Maps was founded in 2014. These contributions have been used to support health interventions, emergency response and resilience programs. Volunteers, organizations, and humanitarians collaborate in a number of ways: remote mapping, field mapping and humanitarian activities.
Founded by the American Red Cross, British Red Cross, Médecins Sans Frontières and Humanitarian OpenStreetMap Team, Missing Maps connects a global community of volunteers to humanitarian response. Other Movement members also include the Netherlands Red Cross and the Canadian Red Cross. Some of the mapathon activities initiatives include mapping tasks in support of work of the International Committee of the Red Cross (ICRC), Nepal Red Cross Society, Tanzania Red Cross Society and Malawi Red Cross Society. Mapathons have been hosted by many national societies including Finnish Red Cross, German Red Cross, and others. As we grow this program, we anticipate more collaboration across the Movement. Read more...
You can definitely say things have happened since previous post.
Currently I am working in Kuwait, on something that will be one of the largest oil refineries when opened. Almost daily driving between this project construction site, and the corporate apartment I have been given in Salmiya, almost 100km away. Kuwait is quite good mapped, but there are construction work on several locations, so the map will need a series of updates and corrections in the near future.
Further I have also taken an active role in preparing GTFS data for bus routes in Greater Vitoria area, and together with the osm2gtfs team have assisted in improving the data model of that project, so it will not be too challenging to create a data scraper and use OSM data to create a valid GTFS data file usable in public transport routing apps.
In addition to this, my gpxupload project have received a complete rewrite, and with a little more attention will be ready for release of v2.0. The script have become 10x faster, and much easier to maintain. It will also start using configuration files, so that options that should be user configurable no longer are hardcoded into the script.
I also have a server in construction, due to the project in Kuwait, I do not have direct access to it, and several of the services are still not in place, but upon completion here I will have the server completed. The server will not be placed online, so it will only run personal services, but will allow me to crontab several tasks I run manually now, and test out a lot of stuff I want to work with, as well as freeing up a lot of disk from my notebook computer.
Planned services for the server:
- cron job for gpxupload
- OSRM for Brazil with daily updates
- PostGIS or OSM mirror
- Mapnik server with Brazil specific and Nautical Chart tiles (TMS with raster and/or vector tiles)
- TMS/WMS/WMTS with official nautical charts compatible with OSM license
- cron job to periodically update GTFS of Greater Vitoria public transport networks
- cron job to process and push my Mapillary/OSC data
- some automated/semiautomated service to create a TMS with free satellite images obtained through my Planet.com partnership.
I am looking into the posebility of a VPN solution, as I think my ISP have a firewall barrier preventing direct connection from internet, even with DynDNS solutions. Access from outside is mostly desired for myself so I can manage the server, aslo when spending long time away.
The Global Partnership for Suitable Development Data (GPSDD) announced one of the biggest initiatives to support collaborative data innovations. 400 applications were summited and after a very close decision, 10 projects will be piloted with the support of the World’s Bank Trust Fund for Statistical Capacity Building (TFSCB). All ten proposals have a strong background in data production, dissemination and use, most of them aiming to try their approaches in low and middle-income countries across Africa, the Middle East and Asia.
The diversity of selected projects covers all kind of innovations, water level monitoring and alert using telecommunication networks, utilising satellite waves to detect illegal fishing and counter human traffic, improving vital registration for Syrian refugees in Lebanon, predictive machine learning methodology to help aid workers foresee patients’ behaviour, etc. Read the complete list here.
Within the selected projects, the Netherlands Red Cross in partnership with Malawi Red Cross Society and CartONG will launch an initiative to use building collaborative data to support Sustainable Development Goals (SDGs) on Health and Water, Sanitation and Hygiene (WASH). This innovation will be tested in Malawi and in the Democratic Republic of the Congo. Malawi Red Cross volunteer working in the field collecting data using his mobile phone
The Malawi Red Cross Society volunteer working in the field collecting data using his mobile phone
The objective is to make better health and WASH decisions and interventions using data collaboration between development and humanitarian actors, academia, government and digital volunteers. To start building and strengthening information bridges between all these organisations, collaborative digital tools and dashboards will be used. The project builds on open data and open source technologies, to facilitate interaction among involved parties.
One of the highlighted innovations of the project is identifying with partners the best practices for data collection for WASH and Health data, and to agree on common standards and tools to collect these data collaboratively. The expected outcome is to facilitate the availability of new data sources and an increased data literacy of the stakeholders.
The project will be supported by 510, the Netherlands Red Cross initiative for smart use of data to make humanitarian aid faster and more (cost)effective. Find out more about 510 here. For the next 18 months, you will certainly hear more about all these innovations, which will improve by learning from successes and failure and start shaping the future of humanitarian response.
[ed. note: Data @ IFRC is a blog series to share highlights from the Red Cross Red Crescent Secretariat and National Societies. We will include learning opportunities and thought pieces on all things data from ethics to evidence.] Maps have always been storytelling devices. And, the art of mapmaking is a community-building exercise. Missing Maps and YouthMappers are two programs which provide unique ways to connect humanitarian work to mapmaking. YouthMappers, with university chapters around the world, connects humanitarian mapping needs to education programs. Their main target group is university-level geography (GIS) students. Missing Maps is both a global and local program that supports mapathons around the world. Inspired by both concepts, data scientists Heather Leson and Guido Pizzini co-hosted a Missing Map event for youth at Ecole Internationale Geneva. See below the complete narration;
Maybe I think creating a new map style is quite easy for someone who is experienced with coding new styles. To be honest I really do not know the amount of work nor if you need server to render a new map style. Since quite a time I am looking for a map style that shows only buildings which are ilustrated in black. Not more, not less. No streets, no water bodies, no green areas. In architecture its called ground map (in german: Schwarzplan) and should give you a better visualisation of the structure of a town. Due to these structures it gives you also a reference of the historical development of a city. The often refered website (www.der-geograph.de) is a good posibility to render areas with the above mentioned style. My question to the OSM-Community is if it is not possible to create a map style which makes it obsolete to render areas tile by tile which takes ages like on that website.
Obviously but very importantly:
- Before copying anything into OSM the license needs to be properly vetted.
- If you are planning to perform any kind of bulk update, the full extent of the Import Guidelines applies.
After comparing Tiger 2017 for 16 US states until now, using shp2cygnus, a tool that I written at Telenav to be able to compare big datasets in a automatic way, to identify roads that are present in the dataset, but not in OSM
After posting on the talk-us, some people told that it would be more useful to run this tool on other datasets, from state or local authorities.
In this article, I want to teach you how to create your own translation file, so that if you have a shp file, you can use Cygnus to compare this with OSM, using this link http://cygnus.improve-osm.org If you have a shp were the size is bigger then 50*50 km, you can send me a email and I can process it for you.
But you need to learn how to create your own translation file.
So, first step.
- # Find your data
We will use as a example Philadelphia Each city or local authority should have a open data portal. After a google search for "Philadelphia Open data", I found this site https://www.opendataphilly.org/
Search for either street or road
Click the data-set, and identify the shp, that you need to download
Extract the archive and open the shp into JOSM.
To be able to open shp in JOSM, you will need to install the OpenData Plugin
If we look at the tags, we see that they don`t have any connection with what we are used to, like highway=residential, tertiary, etc or oneway=yes
To be able to understand how to translate the shp to osm, you need the Street Centerline Metadata
Click Download, and you will see, at least for the Class, what is the highway type.
To see how you can create the translation for osm, you can look here for the OSM highway tags http://wiki.openstreetmap.org/wiki/Key:highway
After doing this, run this script, and then load the output.osm file into JOSM.
python ogr2osm.py -t translation_philadelphia.py --add-timestamp --add-version --positive-id input_shapefile.shp -o output.osm
It will not be perfect, this is after the first try.
Comparing this with osm data I see that I have to much yellow roads (tertiary)
Also, the trunk roads are actually motorway on OSM
We look at the shapefile and identify that Class=4 should be highway=residential, instead of highway=tertiary
We run again the converter
And we get now a more better result, more accurate with what we have in OSM.
If we are happy with the result, we need to save the osm XML file to PBF, so that we can use them with Cygnus.
We do this by installing the pbf plugin in JOSM
Save the file as Pbf
We next go to the http://cygnus.improve-osm.org website, and load the pbf file.
Also, you can select some tags of interest, for me, this was oneway.
In the end you should see something like this
You can download the translation file for Philadelphia from here
It's been possible to run Linux software on Windows for some time using Microsoft's "Windows Subsystem for Linux". Here's what needs to be done to run a simple tile server there, based on the existing instructions here .
Firstly, install "Bash on Ubuntu on Windows". There's are some intructions here (but don't worry where that says "This won’t work with server software"). Continue down to where it says "You’ll be asked to create a user account and password for use in the Bash environment". It's easiest if that user account is the one that you use in the rest of the switch2osm instructions.
Once installed, at a shell prompt "lsb_release -a" should say:
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
What yo get is surprisingly similar to a standard Ubuntu server machine. Some things don't work (e.g. "screen"), and you'll have to manually start some of the services that will need, but otherwise things are very familiar.
Run "sudo apt update" and "sudo apt upgrade" to upgrade to the latest version of everything.
Next, go over to the switch2osm instructions and follow those as normal, except for the caveats pointed out below:
If you want to ssh in (you probably will) you'll need to "sudo service ssh start".
You'll likely want to allow access to ports 22 (for ssh) and 80 (for http) and possibly others at the Windows Firewall (inbound rules / new rule / port / tcp port 80 / allow / all / "name").
After installing postgresql but before doing e.g. "createuser renderaccount" you'll need to start postgresql manually. "sudo service postgresql status" will say "down", but after "sudo service postgresql start" it will say "Starting PostgreSQL 9.5 database server" and "service postgresql status" will then say "9.5/main (port 5432): online".
Where the switch2osm instructions say "service apache2 reload" you'll need to do two things. One is to work around a "Protocol not available: AH00076: Failed to enable APR_TCP_DEFER_ACCEPT" error that is described here, and the solution is to add "AcceptFilter http none" to the end of the /etc/apache2/apache2.conf file, and the other is to start Apache2 manually via "sudo service apache2 start".
Pointing a web browser at http://yourserveraddress/ should display an Ubuntu Apache "It works" page. If it doesn't, check firewall access and any error messages displayed when trying to start apache.
Before you run "renderd -f -c /usr/local/etc/renderd.conf" you'll need to ensure that postgresql and apache2 are both started (they'd start automatically on Ubuntu but not here). Requests are passed from Apache and mod_tile to renderd, and here's an example of what you might get (this is with Hereford, an English county, loaded):