Recent diary entries
Early this month, Butuan City was again the site for a crowd-sourced mapping workshop using OpenStreetMap (OSM), part of the Coordinating Roads and Infrastructure Investment for Development (CR+ID) initiative for industry mapping series run by The Asia Foundation.
The Industry Mapping Workshop proper was held in Y Hotel, Butuan City, on 13th-15th January, while the OpenStreetMap workshop track started in the afternoon of the 14th. This OSM workshop is the sixth of a series of workshops being carried out by the Coalitions for Change (CfC), of the CR+ID project meant to familiarize participants with the OSM platform, and other Open Source mapping technologies they can utilize in their respective communities. It is also meant to promote the development of interest by local government authorities, formal organizations, local volunteer groups, and informal associations to map their communities and other areas of interest using the OpenStreetMap platform. In this particular case, the activity is focused specifically in teaching the participants the rudiments of mapping establishments, infrastructure, and other points-of-interest (POI) that relate to the tourism industry.
The workshop series is an introduction to the OpenStreetMap platform, and builds upon the participants prior knowledge about mapping. Many of the participants already have working knowledge of geomatics and geographic information systems.
Participants were also introduced in the use of Smartphones and the OsmAnd application for mobile field data collection. Early next morning, participants were give an opportunity to exercise their field data collection skills in downtown Butuan. The late morning was spent on editing and processing the field data, and sharing their learning experience.
An Overpass-Turbo query reports the following changes by the participants during the workshop day itself: nodes: 219, ways: 18. The edits are mostly in downtown Butuan, where new POIs had been added to the local map during field work. Some mappers, made updates in their own neighborhoods, as well.
Kudos to the new mappers. I'm looking forward to seeing more editing activity from that area of Mindanao.
This handy feature in JOSM fetches the titles of visible imagery layers that you have loaded into JOSM and adds it as the changeset source.
Never knew it existed till someone pointed it out to me. Hope you find it useful.
Last Saturday, the U.S. Embassy in Manila hosted a workshop and mapathon to add data to Project 1129: Missing Maps: Leyte, Philippines. Organized by Celina Agaton/Map the Philippines and OpenStreetMap Philippines(OSMph) advocates. I took a few photos, too.
During the closing activity, the top new mappers were awarded with prizes, but only new mappers were taken in consideration, and I neglected to give due recognition to two "old" mappers who contributed significant number of edits in the area of interest during the mapathon, namely curran74 and Sea Tea Zen.
Cheers for these two awesome mappers. Their edits, combined, is responsible for one third of the total mapathon edits made in the Leyte task. Guys, I hope to see you both in the next mapathon. Do remind me that I owe you a bottle of beer each. ;)
Many thanks to the Mr. Allen Phelps and his team for hosting the mapathon at the American Embassy, including the refreshments, Celina Agaton for organizing the event, and the usual suspects from OpenStreetMap Philippines who were physically present - seav, Rally, schadow1, and Ge-Mapper, and two remote participants jmbangante (from chilly France) and maning (from warm Bengalore) for the support.
For related posts, check your favorite social media stream for the
#ManilaMapathon hashtag. Group photo above is snipped from Celina's tweet about the event.
The spike in mapping activity over the weekend is attributed to this mapathon (from Pascal Neis' Neis' One country stats):
Out of curiosity I had a short look at various map formats how much space they need for a certain country. I picked (for no certain reason) Bulgaria.
- 30 MB Garmin (openmtbmap, with contourlines (5MB))
- 41 MB mapsforge
- 57 MB Geofabrik osm.pbf extract
- 67 MB osmand
- 73 MB maps.me (streetmap (42MB)+routing (31MB))
- 98 MB openandro (with srtm)
- 100 MB Geofabrik osm.bz2 extract
PS: Of course various maps may contain a various degree of details in the data which I did not check.
PPS: Fixed OSMAnd and mapsforge after Zverik's hint (used a third party website before with obviously outdated data).
Just found that the raw satellite imagery from ESA's Sentinal-2 project is now open. The resolution of the imagery is 10m/pixel, so not good enough for street level detail, but can be quite useful to monitor vegetation changes and infrastructure construction progress since the update cycle seems to be around 1 month.
Band 8 imagery with false color mapping showing the 5km long Bandra-Worli Sea Link in Mumbai
This is a great source for multiband imagery that you can use to do some interesting raster analysis in QGIS.
EU law grants free access to Copernicus Sentinel Data and Service Information for the purpose of the following use in so far as it is lawful4:
- communication to the public;
- adaptation, modification and combination with other data and information;
- any combination of points 2 to 4.
Where the user communicates to the public or distributes Copernicus Sentinel Data and Service Information, he/she shall inform the recipients of the source of that Data and Information by using the following notice:
- 'Copernicus Sentinel data [Year]' for Sentinel data; and/or
- 'Copernicus Service information [Year]' for Copernicus Service Information.
Where the Copernicus Sentinel Data and Service Information have been adapted or modified, the user shall provide the following notice:
- 'Contains modified Copernicus Sentinel data [Year]' for Sentinel data; and/or
- 'Contains modified Copernicus Service information [Year]' for Copernicus Service Information.
The users' rights on their personal data are protected under European law6. Such data will only be used by the European Commission and the providers of the said Data and Information for providing services to the user and for statistical as well as evaluation purposes.
Since NC D.O.T. may and have realigned a few roads in North Carolina, so, I had to realign the paths, delete redundant ones and added planned ones.
See what you guys think!
It has been a few weeks since I wrote about the public beta release of Cygnus, the Telenav conflation engine for OSM data. Since then, I have since been approached by a few folks who wanted to take it for a spin. One of them is long time OSM contributor MikeN. He is preparing an import for Holt and Atchison counties in the U.S. state of Missouri. We worked together on scaling some technical hurdles. Here's a report of what we (well mostly he) did.
Mike obtained the source data from Holt and Atchison counties from their official GIS:
I obtained an updated road network from their official GIS, extracted and translated the tags, and followed up with a review against current aerials as well as checking for connectivity, glomming like road segments, and simplifying geometry. The final goal is to obtain permission to import, go through the import process steps, and merge new data onto the existing OSM data.
The next step was to convert the data into the OSM PBF format that Cygnus requires. This is when Mike got in touch with me to work through some technical difficulties:
Since Cygnus required the PBF format, I used Osmosis to convert. This failed because the nodes did not "have a version attribute as OSM 0.6 are required to have". I have learned from Martijn that OsmConvert works without a version attribute, and was able to verify this on my second county.
The next catch was that PBF doesn't accept negative node numbers. The simple workaround is to just use a text editor to remove the minus sign from
ref='-. This seems a bit dangerous - would that file upload if accidentally selected? If so, many low numbered objects would be corrupted around the world. Hopefully, the conversion from OSM to PBF can be moved to the Cygnus chain so that it can accept zipped OSM since most users will start with .OSM data.
That is a great suggestion. On the one hand, we don't want to make Cygnus too easy to use. (Cygnus is in the end a tool to help with imports. It should never be easy to just import data in OSM. There are strict guidelines, and any tool to help with imports should make the user consider the process very carfully.) On the other hand, handling the conversion from JOSM XML (including the negative IDs) to valid PBF is a mechanical step that most any Cygnus user would need to perform, so I would like to include that in a future version.
With that out of the way, we worked together to produce the Cygnus JOSM XML change file.
The result was that it did pick up the new roads and they appear to connect properly into the existing road network. The cases of modified geometry were also detected. And although the node placement was different, the rest of the roads were properly untouched.
It turned out that the number of changed / new roads was fairly minor. A future import would therefore not be too invasive. Here is the OSM base data versus the updates suggested by Cygnus for one of the counties Mike is working on:
In total, Cygnus suggested 68 updates. 31 entirely new geometries, and 37 updated geometries. The updated geometries were mostly caused by connecting the new ways to the existing network, adding a node to the existing way where that happens.
Mike is still working with the counties and the community to move this import forward. Working with Cygnus gave some good insights and will hopefully help prepare the actual import when it happens. This is pretty much an ideal use case for Cygnus, and I hope to see more of them.
Mike also has a wish list for Cygnus:
Future enhancements - In my case, I extracted the surface tags from the GIS source. It would be interesting to have more control over tag merging - such as taking surface tags from the 'new' ways if there is no current surface tag. And in the case of renamed roads, to be able to give the new name a priority for the merge.
I already discussed this with my team and this is high on our list of Cygnus improvements - together with support for POI type nodes.
Get in touch with me if you are ready to give Cygnus a try with local data you have!
At Open Labs we wanted to kick off the local OpenStreetMap community in Albania since the first year of our hackerspace in Tirana. For many unknown reasons (aka lack of time) and different this did not happen until Open Source Conference Albania 2015, where we organized a competition for the best OSM editor of a specific area in Tirana. This was followed by meetups in September and November inside and outside our hackerspace. After the first baby steps I strongly believe that 2016 will be even more intensive and hopefully more OSM contributors will join us. If you want to be part of the community ping us at our mailing list. Have a look at some of the OpenStreetMap meetups at Open Labs Hackerspace in Albania
Here is for an OSM-er 2016!
When I learned mapping OSM from one of my closest friends Dr. Ganesh Dhamodkar, I searched for Ramtek town (because my hometown Kachurwahi was already mapped by him). It was completely blank except some roads. Then I started adding things, residential roads, POIs, Hospitals, clinics, Schools, colleges, etc and my map became pretty good over a week. Still many things are to be added there. I hope you would love it. Just have a look at it. I still need lot of guidance and suggestions. Map of Ramtek
OSM project is famous for its quality assessment tools. We have tools to track changesets, to check properties of objects against certain reference databases, we have Achavi to visualize changesets, we have KeepRight to analyze geometry, check website links and to do lots of things. There is also a mechanism of notes, which works as an issue tracker for both bugs and "feature requests". Ability to comment on changesets is nice, but it's a kind of limited: only author of changeset gets a notification and there is no way to bind a comment to an object to turn it into another issue tracking tool. But all these tools are for data: geometry and properties (tags).
Infrastructural elements of OSM, such rendering styles, websites, editors, being an opensource sub-projects, usually have issue trackers on GitHub.
But there is absolutely nothing for tagging schemes. I mean, yes, you can write something on Discussion page of Wiki, or complain somewhere on mailing lists or forum. But nobody is checking discussion pages constantly, while posts on mail lists and forums have a tendency to be buried pretty quickly by other stuff posted there.
For me, it seems to be logical to have some place, where everyone can tell about an issue of tagging scheme:
- overlapping meaning,
- linguistic issues (different meaning of word, used for key or value in different cultures),
- systematic misuse of tags,
- lack of way to tag particular entity.
Definitely, it should not be a place to ask questions about tagging - it should be a place exclusively for reporting issues and requesting features, just like the most of issue trackers.
It is well-known fact, that acknowledging a problem is the first step towards fixing it. And currently there is no common way to make this step. For example, people are showing up on forum or mail list complaining about something and getting typical reply: "It's well-known problem, but we don't have any solution yet." That's pretty much all that happens. Then, after a month or a year, someone asks about the same issue.
Good issue tracker with labels, ability to merge issues and so on, may help collecting valuable information about bugs, imperfections and incompleteness of OSM tagging scheme. And contributors, interested in improving it, could use this information to set better objectives, to have better understanding of major issues. It also may probably help to connect those who have an issue and those who can fix it, because many contributors do not have any experience in editing Wiki to clarify a documentation, while those who can edit Wiki are not necessarily aware of the same issues.
I'm not sure, if GitHub issue tracker can actually fulfill all requirements since I don't have enough experience running it as an administrator, but if yes, it might be a good choice.
Feel free to leave your comments and suggestions.
We're well on the way for the next Vespucci release, a rough list of what I'm working on can be found on github. Numerous changes have already been implemented, here I just want to touch on the most interesting one for now.
Long time users may remember when the tag editing UI looked like:
The image is from version 0.7.0 which dates back to 2011/2012 (before I even knew it existed), I haven't been able to find older screen shots, but I suspect it looked similar in 2009 in the first version. Mid last year I refactored the tag editing code to support a multi-page view and lots of other improvements, resulting in:
One of the goals of that effort was to make extending the tag editing UI easier and that has now beared fruit in allowing a simple preset-driven form based editing interface to supplement the old key-value editing:
This might sound straightforward, however there is no guaranteed one-to-one matching of presets to OSM elements which may have attributes for multiple real-world objects tagged (finding the correct preset is key to determining the correct values for each tag). I've taken an iterative approach to addressing this by trying to find the best match, adding all tags that belong to that preset, then adding all tags from linked presets and then repeating this for the remaining tags and so on. So far the results look good.
There are still a couple of rough edges which need to be smoothed out, but I expect a test build with this feature to be available in a couple of days.
Update: first beta with new UI https://github.com/MarcusWolschon/osmeditor4android/releases/tag/0.9.8-beta-1
Quadrant routes in Pennsylvania have 4-digit route numbers prefixed with
SR. OSM recommends that these route numbers be tagged with the
ref:penndot key rather than the
ref key in the following manner:
- Total incorrectly tagged ways in Pennsylvania: 6819. Of these:
- Incorrectly tagged ways in Pennsylvania excluding routes with no space: 3132. Of these:
I'm looking into re-tagging these incorrectly tagged ways into
ref:penndot=SR **** to prevent any problems with mappers and data users getting confused by the mention of a reference highway that isn't normally meant for the public being in the
The work-flow I'll be following is listed below:
- Load overpass turbo link and run the query
- Export the results into JOSM
- In JOSM, find objects with the filter
- Rename key for selected objects from
- The changeset comment i'll be using is
Changing key of quadrant routes in PA which are not signposted from ref to ref:penndot as per OSM Wiki guidelines https://github.com/mapbox/mapping/issues/149
For further details on the progress please refer to this repository.
I live in Mechelen and work in Brussels.
I do Mozilla Stumbler, Mapillary and a little OSM.
In OSM I mainly maintain the notes for Brussels. I take picture for mappers. I do not do too much effort to resolve bad notes. I am not going to Google Search to understand what the lodger means. So I delete most of them. This is mainly because OSM allows notes to be made by people who are not logged in, and this for publicity reasons.
A few months back in the OSM India community, we were curious to know how much of the
trunk network of National Highways has been mapped compared to the official statistics released by the government.
We started off by aggregating all the statistics we could find from various official sources on the road lengths for different types of roads into a common spreadsheet.
Next was to calculate the corresponding length of roads in each state mapped on OSM. To do this, I used the osm-coverage tile reduce processor with added support for Admin-1 regions from natural earth vectors.
All it takes to run this for the Indian subcontinent is:
node download.js --all
node index.js --area=[56,-12,116,42] > output.json
To generate per highway tag road length in kms for each state in the bbox. The processing should not take more than 5 minutes.
These numbers were used to compare with the official figures and generate a list of states where the national highway coverage needs to be improved.
Try it out for your area and see if it helps detect missing highway networks. The code is in the
regions branch of the processor:
Happy to hear feedback on how it worked and how the processor can be improved.
There is an article How to invent tags written mostly back in 2010, which is a kind of official guideline for making new tags and tagging schemes. Official, at least because it's linked from Any tags you like article, describing one of fundamental principles of OSM. But it seems like very minimalist guide, because it covers only a small fraction of mistakes. It even gives certain unclear tips, which could be easily misinterpreted. It also tells about "don'ts", but just a bit about "do's". I'd like to discuss it here, just because writing something on Discussion page of that article will not bring anyone's attention.
First thing anyone must do before introducing or proposing a new tag is to do a research. There are tons of officially approved, just documented, poorly documented and undocumented tags introduced by unknown contributors. Obviously, it's not so easy to find common entity which could be added to OSM database, while there is no tag for it. Therefore, the best way to do initial research is to use Taginfo, Wiki search and regular search engine queries with special keywords such as
site:wiki.openstreetmap.org for Google. It's better to use different keywords, even barely related to tag you want to introduce. Then, it's important to thoroughly examine everything you've got from search: keys, values, descriptions, statistics of usage and schemes/methods of tagging.
Good example of skipping this step is recent proposal of
leisure=aquatics_centre. There already is the way to tag it with
sports=swimming (however, it's not ideal, but it doesn't matter in this exact case, because proposed tag doesn't address any issues of current scheme). Author either wasn't aware of this existing method or ignored it (these are just two possible reasons, I'm not speculating about his intentions here). In general case, research should help anyone to discover any existing methods of tagging.
Speaking of different keywords, it's important to understand, that English is quite diverse. And it's not only about color and colour, center and centre. Sometimes, semantics is quite different. Good example is
shop=furnace. Likely, this tag was introduced by someone from the United States, because it was supposed to be a heating equipment store, not industrial metal melting/heat treatment equipment store. Should anyone be surprised, that contributing into an international project by introducing new terms requires certain linguistic skills? I don't think so. But for people, who invented
megalith_type=grosssteingrab it wasn't that obvious.
Regarding of any new key, it should be required to explain, which exact property of real-world object it describes. Otherwise, we will get more keys like
memorial= which specifies a "kind" of memorial (could anyone, please, explain, what is a "kind" exactly?), so we have values for both design (
plaque) and for an event it commemorates (
war_memorial). Usage of kind, type, sort of anything in descriptions is extremely bad manner, even if you giving an example.
Similar thing applies to values: any new value of key should be homogeneous with the existing ones. Therefore, no
war_memorial as values of the same key. Just think about it: how would you tag a stele, built in memory of war?
memorial= key is an example of semantic redundancy, because "memorial" is repeating. Another bad practice, not covered by original guide.
Same article also recommends to use descriptive names like
hiking_trail, instead of just
trail. This example looks misleading - there could be probably trails for other purposes, therefore, having the whole bunch of
*_trail tags is not a good idea. Would it be better to have separate key for trail purpose? Probably, yes. Here, that preliminary research helps to find out, if there are other purposes or something.
Original article says: "Avoid elaborate classifications". Elaborate is not bad, if objects, described by it, are complex enough. Example given there illustrates redundant, not elaborate classification. Complex classifications could be required to describe certain things, just because of nature of these things. You can't have less classes (values, for example) for something which actually has many different variants. Generalization works sometimes, but it's not necessarily acceptable decision. For sure, redundancy is a bad thing, but it's better to explain that properly.
This article also doesn't tell anything about extending an existing schemes. Another (very common) issue, not covered there, is a problem value lists. Key
sport= gives a good example: if there is a
leisure=pitch, it's impossible to tell, that you can play both volleyball and basketball there. Only option for multiple sports is
multi. Theoretically, there is a way to have value lists by using semicolon separator, but people rarely do that because it's rarely supported by styles and converters. Good practice is to have several keys in the same namespace with
no/unset values, just like Healthcare 2.0 uses for different medical specialists.
There are many other common issues, but I think, I already said enough to explain what I'm talking about.
As usual, I just have to say, that my intention is not to discuss specific tags (mentioning any bad scheme often leads to that), but to discuss the problem of better guide for making new tags.
I downloaded josm, but it has authentication problem. I authenticated it with my google account, but still while uploading edited data it shows errors. Data are not being uploaded. So I am mapping through Vespucci android app. It takes a lot of time to map things.
This week, I just realized it's been 10 years since I signed up for an OSM account! I never realized I continue to contribute and learn from this community. Some personal anecdotes of the past 10 years below.
My earliest fieldmapping in OSM. CC-BY-SA
Around 2004-05, I was doing my MS research on forest fragmentation. Back then, we had to do all our remote sensing analysis in the university lab because we only use proprietary software which we can't use at home. For my research, I decide to use FOSS tools so that I can do it outside the lab. I stumbled into GRASS GIS and QGIS and started compiling from source. Through trial and error, I learned enough of the tools that I can do any mapping related using FOSS. Armed with new found knowledge, I started looking for free geographic data. Surely, if there are free tools you can also get free data. The only free data I found was LANDSAT imagery and low resolution Digital Chart of the World. I continued my quest and later found out about OpenStreetMap. I signed-up hoping I can get data for the Philippines. Only to find out that nothing not even a country name node exist! :(
But OSM gave me one hope, you can create your own data! So I decided, if I can't get free data, let's build it from scratch and share to anyone!
I tried using the online JAVA applet editor, but it was too painful. It was very slow and I can't find any features to trace from the coarse LANDSAT imagery.
Metro Manila early coverage. CC-BY-SA
Nearly 10 months later, I discovered JOSM which you can use offline and upload later. This encouraged me to start doodling nodes/segments/ways but there's nothing much you can do with a smudgy LANDSAT imagery. The only way I can contribute more is to get a GPS device. So I borrowed one, those good old yellow Etrex. I had to rig a data cable from old credit card and PS/2 mouse to get the data from the device.
DIY GPS data cable
Know, I can contribute more, I started walking around my neighborhood and tried to map as much as I can. My mapping patch started to take shape. First roads, then POIs and so on.
GPS tracks around Marikina, mostly mine. CC-BY-SA
As I map my own little patch, I hang around the talk@osm list asking n00bs questions. I found out that in the UK, there is am a small but active community around the project. They organize mapping parties, meetups in pubs, etc. I realized we should also build a local community here. I tried shouting to the list if there are other Filipino or anyone mapping the Philippines. One guy replied, it was Mike Collinson. It turned put he travels a lot in the Philippines and was also looking for fellow mappers. He started the PH wiki page and we setup the mailinglist.
When Yahoo! gave permission to use their imagery for tracing, more contributors started mapping in Metro Manila and more people joined our mailinglist.
I was able to save some money and finally bought a Garmin device where you can load custom maps. I experimented with mkgmap and started building our own OSM derived Garmin maps for anyone can download for free.
OSM in Garmin. CC-BY-SA
During one of Mike's trip to Manila, we met and talked about how to sustain build the local community. GPS was really expensive at that time so we decided if we can get some and share to anyone interested so they can map their own neighborhood. I applied for the GPStogo program by OSMF and was able to get 4 GT-31 loggers.
Building the community
We slowly grew and started doing outreach to other interest groups, first with GPS enthusiasts to demo how they can use the map in their own devices. Later with universities and mapping groups.
Rally, teaching university students how to use a GPS. CC-BY-SA
We ran mapping parties and piggyback on other events to promote the project.
First Mapping Party, Tagaytay City. CC-BY-SA
The map continues to grow as well as the community. Despite all this, we get criticisms from so called professionals:
"I suggest that you should invite a mapping professional into your group. Mapping is not a kiddy thing. It requires knowledge on Geometric Geodesy and GPS Systematic Errors. Without the required knowledge, doing mapping is just like wearing a blindfold blindly."
Most of the volunteers see this OSMing as something fun. We use it mostly for our personal travels. It wasn't until Typhoon Ondoy when we realized we can use the data for disaster response.
Mapaction's first use of OSM data in the Philippines during Typhoon Ondoy/Ketsana, CC-BY-SA
This started my passion for contributing to the Humanitarian OpenStreetMap Team. First for every disaster in my country,
OSM poster maps during Typhoon Yolanda/Haiyan. Photo by Joe Lowry-IOM.
and later, to other parts of the world.
Lower Shire Mapping Team, Malawi. Emir Hartato, CC BY-SA
I learned a lot from this project and continues to share to those who want to listen. Whenever there is chance, I talk to events and conferences about the project. I've met and collaborated with many people within my community and around the world.
SOTM-PH Lightning talk at SOTM 2011 in Denver.
More than 3 months ago, I started working with the data team at Mapbox. I moved to Bangalore to join a young group OSMers improving the map daily. Although we sometimes receive criticisms from a few community members being paid mappers, I know my team cares about the project as much as any other OSM contributor.
Being away from home, I cannot contribute to the Philippines as much as I would like to other than fixing and cleaning my previous cruft, I maintain my connections with the local community through the list and chat.