OpenStreetMap

Users' diaries

Recent diary entries

My mapping experience

Posted by Zainab Sengeri on 1 December 2017 in English (English)

Hi mappers ! I am happy to be one among youth mappers who contribute in mapping different part of Africa. mapping helps a lot, especially when it comes about saving affected people of specific area. In other words I can say that mapping area helps to reach them easily.I started mapping since June of 2017. I have mapped more than 100 buildings and some high ways of course.

Location: Kilombero, Morogoro, Coastal, Tanzania

points d'intérêt sur une carte

Posted by kersten_vincent on 1 December 2017 in French (Français)

Bonjour, Pour pouvoir utiliser efficacement les cartes d'openStreetMap, j'ai besoin d'identifier certain site sur une carte, est-il possible de créer des cartes "personnalisées" en enregistrant les points dont j'ai besoin ? Si c'est possible, pouvez-vous me dire comment je peux m'y prendre.

Merci pour votre aide.

Location: Leumont, Wanze, Huy, Liège, Wallonie, 4520, Belgique

Leaving the HOT US Corporation

Posted by sev_osm on 1 December 2017 in English (English)

Here is the letter of resignation that I had sent to the HOT US Inc members on May 5, 2016 (more context can be found in my next diary entry):

Hi,

I hereby resign from as a HOT US Inc member with immediate effects.

Some recent members may not know me, but I have been active in HOT US Inc, as well as other OSM projects for more than 5 years ago now, both in remote activations, training and field work over more than 20 countries.

My main motivation is that HOT US Inc, despite its storytelling, is not a community driven project, it is a just a very classic Corporation run by a board whose aims are far from openness, truthfulness and respect of differences that should lead an OSM project involved in humanitarian and development fields. And as I do think it even represents a real danger for OpenStreetMap, I do not want to endorse and be in collusion with it. Considering how much I worked to make HOT US Inc successful in some ways, I let this organization totally sickened by what it became. I guess I am not the only one: some huge contributors have already silently stepped out for the exact same reasons.

The triumvirate that quickly emerged within the native English speakers from the 2015 board made the meetings a real pain for a non native EN speaker like me. Talking as fast as they can, which does not allow you to catch everything as you would like or quickly enough to everything that is implied by the decision to make. Not agreeing with them and sometimes opposing their views means disrespectful behaviors being frequently cut when you try to express yourself in their native language, your past contribution totally diminished, your ideas sometimes qualified as ridiculous and every time of the reproach the meetings are unpleasant because of you.

Be really aware of this: you simple members are not informed of the most crucial topics, that are decided and managed backstage. Yes working groups have been really launched in 2014 after 15 members or so cosigned a Manifesto to make things change radically. WG were one of the proposed actions items, but not only. They now do act, but on a limited frame, the one of the Corporation. Nothing that could renew it. And when something is considered as being crucial, comes the incredible point where there is a discussion within the board about how to inform the members the most minimalistic possible way to the point they will not understand what is really at stake. And generally the one and only Corporation adviser so far is requested to provide feedback and rewording on this. Easy to do: since 2014, absolutely no board discussion are opened to anyone outside the board, whatever the matter, the notes are very short to the point you cannot understand well what are all the components at stake and of course, anything that could make react the members (I do not talk about confidential personal matters, but crucial community points) is quickly labeled as confidential. It was really surprising and unpleasant for me to discover that when I have been elected in the board in 2014 that secrecy and lies were core within the board toward the membership.

First example : the HOT trademark. It has been a long time HOT US Inc distorts some basic OSM components, eg with the hot mailing, basically an @openstreetmap.org list that became a corporate list where people can be blacklisted just because their mails (whatever their arguments may have sense or not) could un-please partners of the NGO and not a mailing list depending on the OSMF and ruled by classic mailing rules, this being actually decided by one single person. The HOT Trademark is one big step forward. It has never been discussed or even explained to the members. An initiative of Mikel Maron backed by the majority of the board in 2014. This has not even been presented first to the OSMF, that has been informed when the process was still started. The OSMF has been really upset with it and sent a clear cease letter to the HOT Board. Nonetheless the process has been continued (!), and the majority of the 2015 board confirmed the decision of the 2014 board not to inform the HOT US Inc membership, because it could (obviously, and hopefully!) generate strong reactions. This process has been disclosed only by the OSMF itself on its list inside a thread about HOT US Inc. The people mapping with HOT US Inc tools still do not know they help creating an corporate OSM trademark. With the increase of Hotties in the OSMF board, the tactic seems now to make it adopt from the OSMF itself.

Second example: the budget gap.

If you read the report from the last Membership call you can find these words:

“Found a budget gap in 2015 found, but also note funding gaps are very common and normal, just not ideal”:

It reads as if everything was normal, under control and that the situation would be totally safe for the last and the current years. This is totally typical how HOT US Inc has been building storytellings that do not match the facts. This also does not reflect the current state of HOT US Inc.

Yes the Board had voted a negative budget for 2015, assuming expected grants that would have covered the missing money. But does this have been monitored and under control all along the year? Absolutely not.

On September 11, a big financial issue arose, when basically it has been figured out the org had only a bit of money left in its account, but was supposed to run activities for months with this.

This is not something that can be qualified as “normal”. This is basically, by far far far, the biggest issue HOT US Inc has faced from its beginning. It was totally unexpected, and of course the board was totally freak out and had an exceptional meeting during which two main things were decided:

1 Mitigate the situation as best as possible and drastic cuts have been very quickly decided within paid positions when possible and some expenses canceled. Eg no in person board meeting last year.

2 Inform clearly the membership about the situation. From my experience of how the organization has been communicating with the membership for the last two years, I emphasized that the board should not “hide” it. The triumvirate was almost offended by this word. Unfortunately, this is exactly what happened.

Here is the original message that was supposed to be sent:

We are writing to update you on changes and challenges that are facing the community.

We have a significant cash flow problem that the Board and Executive Director are working hard to solve. Unfortunately, this has lead to some very difficult decisions and hard choices. HOT is releasing some of its staff to get on a better financial foundation. We are also investigating other ways to pay for administration needs to continue to support HOT.

No one involved in the process is happy with the situation, we want and expect this to be a temporary situation and look forward to carrying on our important work in 2016 and beyond. Added: we will be asking for your help to build some plans. It is our dream to have a larger HOT Summit in the coming year. While we are focused on the immediate tasks, we are also keep supporting this amazing global community.

But on October 2, the message eventually sent to the membership was the following:

We are writing to update you on changes and challenges that are facing the HOT community and organization.

NGOs often have times when there are funds to raise and plans to change. HOT is no different. We are working to improve our financial oversight and organizational development. And, we are making some changes to our policies and practices to build a more sustainable organization. Over the past 5 years, HOT has been primarily a project funded organization with no very limited unrestricted funds. This pattern and HOT’s operating costs have made it difficult to grow as quickly as we would like. We have drafted some initial plans on this front, which we hope you will help guide.

This makes quite a difference, isn't it? Significant cash flow problem […] HOT is releasing some of its staff to get on a better financial foundation VS improve our financial oversight and organizational development.

Since, the Operations budget gap has been being mitigated by drastic cuts + the fundraising, that has purposely been made to fill the gap, not for future projects and has never been presented as such. Once the Operation budget mitigation improved, figures have been eventually shared with the membership through the 2016 Operations Budget, along the story it was not ideal but normal, and supposedly everything was under control.

Is the situation safe now? Not really.

Apart the Operations budget, everyone should know that there are also grants for each donor funded tech or field project, and actually this is where the financial gap is the largest. End of September 2015, HOT US Inc should still have approximately USD152,000 for activities still to be done or to be returned for one large multiple years project, while the bank account was around only USD10K. As a former Senior Project Manager for 5 funded projects for HOT US Inc (STM020 in Haiti, EUROSHA in Central and Eastern Africa, CAP103 in Haiti, UB_ICT4D in Mongolia and Lower Shire Community Mapping in Malawi), it will remain amazing for me how such a budget swift may have occurred, representing almost 25% of the total grant. A USD33,000 planned payment by the donor on last November allowed to postpone the cash flow issue and temporarily hide the situation, but as the project will come to its end on October 31 this year, this is obviously a huge issue.

The membership has never been informed about this situation and I guessed it would not be even the case for this board elections,

This is totally unfair and harmful to distort the reality and hide such issues to the membership, because:

1 you do not build common knowledge and experience so that this kind of situation is avoided in the future. Over 2 years the people from the board can completely change and an Executive Director can leave in only two months. And the same lack of minimal financial oversight can occur again. As well, some skilled members could have helped to mitigate the situation (as suggested in the initial informing draft) but the organization lost this potential support.

2 you do not inform the future board candidates about a hard situation they will discover only once after being elected and have to deal with and solve.

3 basically the O of HOT comes from OpenStreetMap that includes Open, seems that many forget this, and completely distort the facts are certainly far from this.

Of course, being involved in such lies has been a real pain for me. Of course, I was in strong disagreement with the way these topics were handled, from the moment I figured out that what was not supposed to be hidden would actually be. But the fast speaking majority imposes its views and you are due to shut your mouth. Otherwise you are wrong, and you are basically a bad person, just because it was a majority board decision, whatever it can be harmful for the OSM project from people with little experience or concern about openness or OSM, that does not matter at all. At all.

You can imagine the dilemma for someone who campaigned very clearly about openness transparency to be involved in such practices. Retrospectively, I would like to have reacted differently, to organize immediately special meetings to make this known immediately.

I eventually reacted this way when I disclosed to someone unaware of a complaint made towards him by the Board president because of concerns made about many Hotties running for the OSMF election and having his case being processed without even being informed of this complaint, in the most amateur, Kafka style way I would never have been able to foresee. This was the motive immediately used by the triumvirate to ask for me being put off the board. I do not regret this disclosure at all, on the contrary I am proud of it: on a personal side, at last I acted as I should have done regarding my declaration when candidate, whatever it costs me; on a community side, this could show to some members how this board violently reacts when its bad behaviors are pointed out. This is just the beginning: the reinforcement of the code of conduct will provide easier possibilities to hammer some discording voices and let the “awesome” storytelling and the big business goes on. So, for whoever would like to renew or simply improve it, good luck. Other tried, and did not manage it, like myself. You will figure out a small group of people with loooong agendas actually run this organization. And they bite hard.

But hopefully HOT US Inc is not the only way to trigger OSM in the humanitarian and development fields. It has actually even quite reduced its potential over the three last years. And basically OSM is a sum of small initiatives. It does not need a mogul Corporation to be effective. Fly with your own wings.

Sincerely,

Severin

Bounty for rendering queue fix

Posted by Komяpa on 1 December 2017 in English (English)

OpenStreetMap.org tile rendering operates on edge of its capacity. It's possible to get more from already existing cluster if rendering queue is not continuously dropped, but instead is preserved and processed during low load hours.

Ttile.OpenStreetMap.org uses mod_tile and renderd to manage rendering queue. mod_tile is currently a core ecosystem project lacking an active maintainer for years. There are currently 19 Pull Requests that nobody merges.

I tweaked values in source code to allow for a longer queue. Pull request is available at https://github.com/openstreetmap/mod_tile/pull/152. In comments there was a proposal to make it a configurable parameter.

I offer 50 USD to person who manages to:

  • create pull request to mod_tile/renderd that makes queue depth a configurable parameter, and set defaults to preserve the queue to shift day's load to night;

  • get it depolyed to tile.openstreetmap.org adjusted accordingly.

Shift should be visible on https://munin.openstreetmap.org/openstreetmap/render.openstreetmap/renderd_queue.html graphs, most likely by Dirty queue growing beyond 1000 tiles per server, and being non-empty for the whole day, shortening during the nighttime.

This should make the map browsing experience better for everyone.

stat

Mapping surface features with high resolution DEM generated from Lidar

Posted by abejorro34 on 1 December 2017 in English (English)

Example of use at mountain relief modified by mining and agriculture

Lidar (light radar) is a technique providing a very precise surface scanning. Airborne laser scanning generates a dense point cloud (several points per square metre) representing reflections of light ray. It contains reflections as from the vegetation as from the surface under it. Vegetation and building reflections may be filtered to get relief data only and create a digital elevation model (DEM) of relief. It is usually a raster and each pixel value represents the relief elevation. This raster may be further processed for better visualisation of relief shapes:

  1. Hillshade raster – grayscale 3D visualisation,
  2. Slope raster – colour represents the rate of change of elevation,
  3. Aspect raster – colour represents the downslope direction.

Surface irregularities cause by human activities are clearly recognizable in the high-resolution DEM obtained from Lidar.

DEM Spoil

My hometown was found in the Middle Ages after discovering a rich silver ore. Old mines were reused after second world war to get uranium. The massive mining activities caused significant changes of relief. Some valleys were filled with mining deposits. Agriculture in mountains was also present. Fields at slopes were divided with little retaining walls and embankments.

Embankments

Mapping these features may be useful for more purposes. Scientists and some tourists going away from tracks want to avoid demanding walking over mining deposits, which usually consist of rock blocks and are often overgrown with trees and shrubs. Geologists and mineral collectors would appreciate the presence of these features in map to locate old mines and spoil heaps. These mines are an important part of local history and the map would tell more about them to visitors and historians. Tourists get better image about the extent of spoil heaps. Some line relief features like embankments are good for orientation. Moreover, map with such features looks much more attractive.

Spoil heaps covered with vegetation are difficultly mappable from aerial images. Mapping with GPS in mountain terrain would be very demanding. GPS signal under steep slopes and vegetation is less accurate. Spoil heaps in mountain relief are characterized by change of the downslope direction and are well recognizable in the slope raster as in the hillshade raster. Spoil heaps are mapped with polygon and tagged as man_made=spoil_heap or landuse=landfill (I'm not sure which one is better).

Embankments, retaining walls, cliffs and edges of spoil heap terraces are characterized by abrupt change of surface inclination and thus they are well recognizable in the slope raster. Sometimes it is not clear from the slope raster, what is the downslope direction and which one is the upper edge of the slope and the mapping may be confused. I recommend checking all created features with the hillshade or aspect raster.

Line representing embankments (man_made=embankment), retaining walls (barrier=retaining _wall) and cliffs (natural=cliff) should be placed to the upper edge of the slope and the right side of the line should always face downslope. When mapping these features from DEM, you should always know which one is present. In case of old agricultural retaining walls in my hometown, most of them are ruined and can be marked as embankments. Edges between spoil heap terraces and downslopes are marked as embankments as well. Roads and tracks at slopes are mostly placed over embankments. Slope raster is very suitable for mapping roads and tracks as well, especially if the track is covered by vegetation, not visible in the aerial map and the GPS signal was not accurate.

Mapping

I checked my mapped features in field and must say, that this method is very reliable and accurate, because I made no mistakes. However, I know the area very well and was always sure which features I mapped.

I suppose that getting DEMs from Lidar in most countries is not so easy because it is not available for free. Sometimes it's available as WMS and JOSM allows to add a WMS layer. Always make sure that the provider license is compatible with OSM and you have permission to load the derived data to OSM.

Pokemon Go now uses OSM

Posted by Rovastar on 1 December 2017 in English (English)

The popular game Pokemon Go now (Dec 1) is using OSM for the base map in the game for (all) the world.

Expect to get a massive increase in the users registering and editing OSM in the coming days/weeks.

Things to look out for:

People adding new roads and paths. People editing/deleting existing roads and paths, especially the small driveways, etc as they lump the mall together in their rendering and look ugly and cluttered so I expect people will be changing these to the wrong type or deleting them altogether so they can veiw there map

So roll up and begin casting an eye on the swarm of edits.

I, for one, welcome our new Pokemon editor overlords..

işklj,ğlpkojıh

Posted by beyza24 on 30 November 2017 in Turkish (Türkçe)

ğpojıhug

Location: Cesson-la-Forêt, Cesson, Melun, Seine-et-Marne, Île-de-France, Fransa, 77240, Fransa

Numeração de porta em SP

Posted by O Fim on 30 November 2017 in Brazilian Portuguese (Português do Brasil)

Vila Buarque em São Paulo completada

"The next era of maps: where technology and mental maps meet"

Posted by -karlos- on 30 November 2017 in English (English)
  • This article gives me a vision of a map, dynamical rendered with the details, I need just now.
  • That should be possible by vector tiles and client side rendering, using WebAssembly.
  • The autor says: This article stems from my work at www.Mapfit.com - a nice tool. I think they use OSM data but don't have any note to OSM :-(

https://blog.prototypr.io/mapping-the-reality-of-the-world-df7ad81ccb54

MapFit example

Liberland should be a country or a semi-independent state?

Posted by JesusitoMexico4845 on 30 November 2017 in Spanish (Español)

Liberland is an a "country" founded by a czech person. it is betwen Croatia and Serbia (i think), so...

Location: 6, Kisdobsza, Barcsi járás, Somogy megye, Transdanubio Meridional, Dunántúl, 7987, Hungría

Balancing the presence of the Humanitarian OpenStreetMap Team at the OSM Foundation in 2017

Posted by Nicolas Chavent on 29 November 2017 in English (English)

In 2017, despite the current level of influence that the US NGO "Humanitarian OpenStreetMap Team US Inc" (aka HOT US Inc) has over the OpenStreetMap Foundation (OSMF) thanks to Kate Chapman and Mikel Maron (both OSMF Board directors), Heather Lesson, another former HOT US Inc Board Officer and one of its former President is running for the OSMF Board.

Since 2015, HOT US Inc, is the only organization of the OpenStreetMap ecosystem represented at the OSMF Board with two Directors: Kate Chapman (co-founder / former Board member / former Executive Director; elected at the OSMF Board in 2013) and Mikel Maron (co-founder / former President / actual Chairman of the members; elected at the OSMF Board in 2015).

This state of things provides HOT US Inc with more power of influence over the Foundation than any other organizations which is without precedent in the history of this institution. Consequently, this diminishes the representation of the OSM diversity at the Board of the Foundation.

Extending furthermore this HOT US Inc presence and influence at the OSMF Board and shrinking accordingly the room for the diversity of other perspectives around OSM to be represented in this Body of the organization would be a matter of concerns in terms of : - balance of powers between organizations - diversity of visions, thoughts and practices (including internal democracy and conflict of interests) around OSM - board dynamics: a collective of HOT US Inc Boardees would interact with single individuals

With these above risks in minds, given HOT US Inc current influence, its members (aka “hotties”) would be best advised to support the OSM project at the Foundation outside of the Board through its Working Groups to start with allowing “hotties” to get acquainted with another organization. This would mitigate risks of power games, lower tensions and strengthen OSM diversity by saving room for other perspectives and experiences around OSM to be represented at the OSMF Board.

In 2017, like in 2015, OSMF voting members shall pay attention to representing at best OSM diversity as well as check and balances in terms of organizational powers at the Foundation prior casting their ballot this 2-Dec onwards.

PS : I am Nicolas Chavent, co-founder of HOT US Inc in 2010 and as Acting Project Manager in charge of of its Operations until 2013, co-founder of two French associations Projet Espace OpenStreetMap Francophone (Projet EOF) and Les Libres Géographes (LLG).

Routing QA [eng]

Posted by Cascafico on 29 November 2017 in Italian (Italiano)

Mini abstract

Quoting OSM wiki about routing...

"Checking your fix. After you have fixed an error on the map you will need to wait until the revised version of the map propagates into the routing engine you are using. This delay will depend for each engine...",

...one of the problems in Qualty Assurance (QA) is the lag between OSM editing and update of routing databases. As far as I know, implementation takes some hours in Grasshopper and 24+ in Mapzen and OSMRM.

Hence, in trying to solve a routing problen (ie: for an interchange) testing can only be done well beyond editing time :-(

Let's see how we can bypass this delay using an homemade router.

Choosing a router

Routino looks simple, functional and flexible. I picked out this SW because it works flawlessly even on a small device like my OrangeePi: as an example, you can route in Friuli Venezia Giulia with this live instance, updated daily. For larger areas we'd soon experience performance problems; so let's leave global routinig up to above mentioned services and focus on a kind of patchwork on-demand...

Automating patch generation

User should be able to choose the area to debug: to achieve this (and due to my zero experience in html/perl scripts) I create a Telegram bot named Routino Patcher and a bash script to manage content.

What happens is:

  • bot ask you for "Location" (from mobile device GPS or thru attachment) and triggers script;
  • script runs a query [added barriers] around coordinates received which download an .osm containing highway and restrictions only [e barrier]; preset area is square 0.002° [cos(latitude) TBD];
  • script performs some cleaning ops (routino "planetsplitter" tool abort if "meta" and "note" tags are found in .osm input) and updates extents of routable database (which grows upon to accomodate each requested location);
  • in 30" or so, bot displays a message which comfirms routing database update (details can be found in routino data page) and a link to routino webGUI map, updated with our "patch".

For each location sent, the related .osm is processed together with other areas previously requested [incremental update TBD]. Since this is a service wich aims to test "fresh" OSM edits, locations older than 24h are purged.

Conclusions

I know my low programming skills leaded to this dirty solution, but I needed it, hoping that in the meantime someone will find/build a service like this, already working and using a single interface :-)

Location: Borgo Aquileia, Paderno, Udine, UD, Friuli Venezia Giulia, 33100, Italia

Influence of Human Cognition on Data Classification... Help Science and Participate in short Study

Posted by loai_84 on 29 November 2017 in English (English)

Dear participants,

Thanks to the folks who already participated in the study. We are computer scientists concerning with human cognition influence on data classification of voluntary geographic contents like OpenStreetMap. To participate in the study (it takes 25-30 min) press here

The Study consists of 4 steps:

  • Step 1: participant information data collection. We do not collect any personal identifiable data, but we are interested of only general information like gender, education, ...etc. Step1

  • Step 2: landuse classification of some objects based on CORINE standard classification schema Step2a Step2b

  • Step 3: classification plausibility of some objects, where one object could have more the one valid categories with various degrees Step3a Step3b

  • Step 4: Task Load indeX (TLX-NASA) questionnaire developed by NASA to measure subjectivity of mental efforts of each task. We use it to see how much your mental effort influence your classification choices Step4

For example, regarding landuse/landcover classification, we could agree on that a parcel of land is covered by grass or water which is an abstract level of classification. However, whether this parcel is a park, a garden, a cemetery, or even a golf court might be perceived differently. Whether a water body is a pond, a lake or a reservoir likely requires finer measures, geographic knowledge, and particular expertise.

Therefore, we developed a study to understand more about how people perceive geographic contents remotely. Actually, assigning a proper category for an object depends on

  1. the way human interacts with this object,
  2. the geographic context of the object and its symbolic representation, and
  3. geographic conceptualization of data providers.

The study aims to:

  • find out human capabilities to provide landuse data.
  • understand the influence of classification mechanism and schema on data quality.
  • study human behaviors in providing voluntary data, and
  • emphasis challenges of data classification in voluntary contents.

The study takes about 25-30 min. We do not collect any personally identifiable information. Thank you again for your time and your contribution. To participate in the study press here

Don't hesitate to contact me for further comments and feedbacks.

Best regards, Dr. Ahmed Loai Ali loai@informatik.uni-bremen.de

Схемотехника 13

Posted by Sadless74 on 29 November 2017 in Russian (Русский)

Не успели выступить на прошлой схемотехнике?

Не беда — обведите в своём календаре вечер 14 декабря 2017 (это четверг) и приходите!

Вы не только узнаете нюансы обработки геоданных и работы с ГИС, но и посмотрите на Центральный дом архитектора изнутри, постоите рядом с известными и будущими архитекторами. Сможете пропихнуть им OSM, чтобы сами свои домики отрисовывали :)

Официальный сайт конференции: http://schemo.ru

Регистрация бесплатна: https://iz.timepad.ru/event/621394/

Илья будет рад, если вы расскажете что-нибудь про OpenStreetMap.

Пишите в тему на форуме или ему в личку

ЕженедельникОСМ 383

Posted by Sadless74 on 29 November 2017 in Russian (Русский)

Updates going forward

Posted by foxer57 on 29 November 2017 in English (English)
  • add Wild Rose trails.
  • 3rd south buffered track needs to be changed to a track.
  • remove the relation on 2nd and 3rd Avenue
  • add bike lanes to Victory Road
  • figure out why bike lanes are indicated on Wall St

  • Is there a better city resource for bike lanes other than the BikeSLC map that only displays arbitrary comfort levels?

کار

Posted by Kavan Din on 28 November 2017 in Persian (فارسی)

برای کار

Mapping the Student City

Posted by Anxhelo Lushka on 27 November 2017 in English (English)

I have spent two or three days now and have mapped a large part of Student City using correct data provided by the Municipality of Tirana and applied as a background layer to make my job easier. I am pretty happy with the results so far, have added around 2500 buildings, many businesses based on pictures I took while biking and I intend to map at least 90% of that area part of Tirana.

After seeing a before and after comparision I just love the final results, you can notice how dense that part of the city is now even compared to the city centre (still a LOT of businesses missing from the map, but I'll make sure to add some more too in the upcoming days).

Watching the Map Grow: State of the Map US Presentation

Posted by Jennings Anderson on 27 November 2017 in English (English)

SOTMUS Logo

At State of the Map US last month, I presented my latest OSM analysis work. This is work done in collaboration between the University of Colorado Boulder and Mapbox. You can watch the whole presentation here or read on for a summary followed by extra details on the methods with some code examples.

OpenStreetMap is Constantly Improving

At the root of this work is the notion that OSM is constantly growing. This makes OSM uniquely different from other comparable sources of geographic information. To this extent, static assessments of quality notions such as completeness or accuracy are limited. For a more wholistic perspective of the constantly evolving project, this work focuses on the growth of the map over time.

Intrinsic Data Quality Assessment

Intrinsic quality assessment relies only internal attributes of the target data and not on external datasets as points of reference for comparison. In contrast, extrinsic data quality assessments of projects like OSM and Wikipedia involve comparing the data directly to the external datasets, often authoritative, reference datasets. For many parts of the world, however, such datasets do not exist, making extrinsic analysis impossible.

Here we look at features of the OSM database over time. By comparing attributes like numbers of contributors, density of buildings, and amount of roads, we can learn how the map grows and ultimately improves overtime.

Specifically, we aim to explore the following:

Contributors

  • How many?
  • How recent?
  • What type of edits?

Objects

  • What types?
  • How many?
  • Relative Density?
  • Object version?

The bulk of this work involves designing a data pipeline to better allow us to ask these types of questions of the OSM database. This next section takes a deep dive into these methods. The final section, Visualizing, has a series of gifs that show the results to-date.

The interactive version of the dashboard in these GIFS can be found here: http://mapbox.github.io/osm-analysis-dashboard

Methods: Vector Tiles

Specifically, zoom level 15 vector tiles are the base of this work. Zoom level 15 is chosen because (depending on Latitude), most tiles have an area of 1 square kilometer. For scale, a zoom 15 tile looks like this:

z-15-vector-tile

Vector Tiles are chosen primarily for three reasons:

  1. Vector Tiles (specifically OSM data in the .mbtiles format) are standalone sqlite databases. This means very little overhead to maintain (no running database). To this end, they are very easy to transfer and move around on disk.

  2. They are inherently a spatial datastore. With good compression abilities, the file sizes are not dramatically larger than standard osm pbf files, but they can be loaded onto a map with no processing. This is mostly done with mbview

  3. Vector Tiles can be processed efficiently with the tile-reduce framework.

In sum, at any point in the process, a single file exists that can easily be visualized spatially.

Quarterly Historic Snapshots

To capture the growth of the map overtime, we create historical snapshots of the map: OSM-QA-Tiles that represent the map at any point in history. You can read more about OSM-QA-Tiles here.

Boulder Map Growth

This image shows the growth of Boulder, CO in the last decade. The top row shows the road network rapidly filling in over 9 months during the TIGER import and the bottom row shows the the densification of the road and trail network along with the addition of buildings over the last 5 years.

The global-scale quarterly snapshots we created are available for download here: osmlab.github.io/osm-qa-tiles/historic.html.

While quarterly snapshots can teach us about the map at a specific point in history, they do not contain enough information to tell us how the map has changed: the edits that happen between the quarters. To really answer questions such as, "how many users edited the map?" or "How many kilometers of roads were edited?" or "How many buildings were added?" We need the full editing history of the map.

Historical Tilesets

The full editing history of the map is made available in various formats on a weekly basis. Known as the full history dump, this massive file can be processed in a variety of ways to help reconstruct the exact process of building the map.

Since OSM objects are defined by their tags, we focus on the tagging history of objects. To do this, we define a new schema for historical osm-qa-tiles. The new vector tiles extend the current osm-qa-tiles by including an additional attribute, @history.

Currently, these are built with the OSM-Wayback utility. Still in development, this utility uses rocksdb to build a historical tag index for every OSM object. It does this by parsing a full-history file and saving each individual version of each object to a large index (Note: Currently only saves objects with tags, and does not save geometries). This can be thought of as creating an expanded OSM history file that is optimized for lookups. For the full planet, this can create indexes up to 600GB in size.

Once the index is built, the utility can ingest a 'stream' of the latest OSM features (such as those produced by minjur or osmium-export). If the incoming object version is greater than 1, then it performs a lookup for each previous version of the object in this index.

The incoming object is then augmented to have an additional @history property. The augmented features are then re-encoded with tippecanoe to create a full-historical tileset.

Tag History

Here is an example of a tennis court that is currently at version 3 in the database. The @history property contains a list of each version with details about which tags were added or deleted in each version.

A Note on Scale & Performance

Full history tilesets are rendered at zoom level 15. OSM-QA-Tiles are typically rendered only at zoom level 12, but we found zoom 15 to be better not only for the higher resolution, but it lowers the number of features per tile. Since many features are now much larger because they contain multiple versions, this helps lower the number of features per tile, keeping tile-reduce processing efficient.

One downside, however, is that at zoom 15, the total number of tiles required to render the entire planet can be problematically large (depending on the language/library reading the file). For this reason, historical tilesets should be broken into multiple regions.

Processing 1: Create Summary Tilesets

The first step in processing these tiles is to ensure that the data are at the same resolution. Historical tilests are created at zoom 15 resolution while osm-qa-tiles exist at zoom 12 resolution. Zoom 12 is the highest resolution that the entire planet should be rendered to osm-qa-tiles to ensure efficiency in processing. Therefore, we start by summarizing zoom 15 resolution into zoom 12 tiles.

Summarizing Zoom 15 Resolution at Zoom 12

A zoom-12 tile contains 64 child zoom-15 tiles (64 tiles = 4^(15-12), resulting in an 8x8 grid). To create summary tilesets for data initially rendered at zoom 12 (like the snapshot osm-qa-tiles), we calculate statistics about each child zoom-15 tile inside of a zoom-12 tile. This is done with a tile-reduce script that first bins each feature into the appropriate 64 child zoom-15 tiles and then computes various statistics for each of them, such as "total kilometers of named highway" or "density of buildings"

Since each of these attributes pertains to the zoom-15 tile and not individual features, individual object geometries are ignored. Instead, these statistics are represented by a single feature: a point at the center of the zoom-15 tile that it represents. Each feature then looks like:

geometry: <Point Geometry representing center of zoom-15 tile>
properties : {
   quadkey :        <unique quadkey for zoom 15 tile>,
   highwayLength:       <total length of highways>,
   namedHighwayLength:  <kilometers of named highways>,
   buildingCount:           <Number of buildings>,
   buildingArea:            <Total area of building footprints>
   ...

These features are encoded into zoom-12 tiles, each with no more than 64 features. The result is a lightweight summary tileset (only point-geometries) rendered at zoom-12.

Summarizing Editing Histories

The summarization of the editing histories is very similar, except that the input tiles are already at zoom 15. Therefore, we skip the binning process and just summarize the features in each tile. Similarly, up to 64 individual features that each represent a zoom-15 tile are re-encoded into a single zoom-12 tile. Each feature includes editing statistics per-user for the zoom-15 tile it represents:

geometry: <Point Geometry representing center of zoom-15 tile>
properties : {
  quadkey : (unique quadkey for zoom 15 tile),
  users: [
  {
    name: <user name>,
    uid: <user id>,
    editCount: <total number of edits>,
    newFeatureCount: <number of edits where version=1>,
    newBuildings: <number of buildings created>,
    editedBuildings: <number of buildings edited>,
    newHighwayKM: <kilometers of highways created>,
    editedHighwayKM: <kilometers of highways edited>,
    addedHighwayNames: <Number of `name` tags added to highways>,
    modHighwayNames: <Number of existing `name` tags modified on highways>
  },
  { ... }
],
usersEver: <array of all user ids ever to edit on this tile>

Why go through all of this effort to tile it?

Keeping these data in the mbtiles format enables spatial organization of the editing summaries in a single file. Encoding zoom 15 summaries into zoom 12 tiles is the ideal size for the mbtiles format and can be efficiently processed with tile-reduce.

Processing 2: Calculate & Aggregate

With the above summarization, we have two tilesets each rendered at zoom 12 with zoom 15 level resolution. We can now pass both tilesets into a tile-reduce script. This is done by specifying multiple sources when initializing the tile-reduce job:

var tileReduce = require('@mapbox/tile-reduce');

tileReduce({
  zoom: 12,
  map: path.join(__dirname, '/map-tileset-aggregator.js'),
  sources : [{
    name: 'histories',
    mbtiles: historicalTileset-2010-Q4,
    raw: false
   },{
    name: 'quarterly-snapshot',
    mbtiles: snapshot-2010-Q4,
    raw: false
  }]
  ...

In processing, the map script can then access attributes of both tilesets like this:

module.exports = function(data, tile, writeData, done) {  
  var quarterlySnapshots = data['quarterly-snapshot']
  var histories = data['histories']

For performance, the script builds a Map() object for each layer, indexing by zoom-15 quadkey. Next, the script iterates over the (up to 64) features of one tile and looks up the corresponding quadkey in the other tile to combine, compare, contrast, or calculate new attributes. Here is an example of combining and aggregating across two tilesets, writing out single features with attributes from both input tilesets:

features.forEach(function(feat){

  //Create a single export feature to represent each z15 tile:
  var exportFeature = {
    type      : 'Feature',
    tippecanoe: {minzoom: 10, maxzoom: 12}, //Only renders this feature at these zoom levels.
    properties: {
      quadkey   : feat.properties.quadkey //The z15 quadkey
    },
    geometry: tilebelt.tileToGeoJSON(tilebelt.quadkeyToTile(feat.properties.quadkey)) // Reconstruct the Polygon representing the zoom-15 tile.
  }

  exportFeature.properties.buildingCount_normAggArea  =  < Lookup the number of buildings on this zoom-15 tile (and normalize by area).
  exportFeature.properties.namedHighwayLength_normAggArea = < Lookup kilometers of named highway for this zoom-15 tile (and normalize by area).

  // Access the contributor history information for this zoom-15 tile.
  var tileHistory  = contributorHistories.get(feat.properties.quadkey)
  var users = JSON.parse(tileHistory.users) // Get user array back from string

  // Sum attributes across users for simple data-driven-styling
  users.forEach(function(user){
    exportFeature.properties.editCount         += user.editCount;
    exportFeature.properties.newFeatureCount   += user.newFeatureCount;
    exportFeature.properties.newBuildings      += user.newBuildings;
    exportFeature.properties.newHighwayKM      += user.newHighwayKM;
    exportFeature.properties.editedHighwayKM   += user.editedHighwayKM;
    exportFeature.properties.addedHighwayNames += user.addedHighwayNames;
    exportFeature.properties.modHighwayNames   += user.modHighwayNames;
  });
  writeData( JSON.stringify( exportFeature ) ) //Write out zoom-15 tile summary with information combined from both tilesets.
})

This script produces two types of output:

  1. (Up to 64) polygons per zoom-12 tile that represent the child zoom-15 tiles. Matching the editing-history format, these features contain per-editor statistics, such as kilometers of roads.

  2. A single zoom-12 summary of all the editing activity.

Processing 3: The Reduce Phase

When the summary zoom-12 tile is delivered to the reduce script, it is first written out to a file (z12.geojson) and then passed to a downscaling, aggregation function, described next.

Downscaling & Aggregation

Last year I made a series of similar visualizations of osm-qa-tiles. I only worked with the data at zoom 12 and kept the features very simple in hopes that tippecanoe could coalesce similar features to display at lower zooms. While this worked, there were a lot of visual artifacts in busy parts of the map and the tile individual geometries must be low detail to fit:

Last Year's Example

To address this, we rely heavily on downscaling and aggregation in the current workflow to successively bin and summarize children tiles into a single parent child. Each zoom level is then written to disk separately and tiled only at specific zoom levels. Unfortunately, this is done by holding these tiles in memory. Fortunately, however, with a known quantity of (4) child tiles per parent zoom level, we can design the aggregation to continually free up memory when all child tiles of a given parent tile are processed.

Psuedocode:

zoom_11_tiles = {
   'tile1' : [],
    ...
   'tileN' : []
 }

processTile( incomingTile (Tile at Zoom 12) ){
  z11_parentTile = incomingTile.getParent()
  tiles_at_zoom_11[z11_parentTile].push(incomingTile)
  if (tiles_at_zoom_11[parent].length == 4){

    // Aggregate, Sum, Average attributes 
    // of zoom 12 tiles as appropriate to create
    // single summary zoom 11 tile

    // Write aggregated, summarized zoom 11 
    // tile to disk and delete from memory.
  }
}

In reality, these are not done for every zoom level, but instead for zoom levels 12, 10, and 8.

To ensure this function works as designed, the order of tiles being processed by the entire tile-reduce job is modified to be a stream of tiles grouped at zoom 10. While we cannot ensure that tiles finish processing in a specific order, by controlling the order of the input stream, we can create reasonable expectations that groups of tiles finish processing at similar times and are therefore appropriately aggregated and subsequently freed from memory.

Processing 4: Tiling

The final result of the tile-reduce job(s) is a series of geojsonl files (line-delimited) representing different zoom levels. Using tippecanoe, we create a single tileset that is optimized for rendering in the browser. Recall that each geometry is a polygon representing a vector tile. The attributes of each feature are consistent among zoom levels to allow for data-driven styling in mapbox-gl.

tippecanoe -Z0 -z12 -Pf --no-duplication -b0 \
  --named-layer=z15:z15-res.geojsonl \
  --named-layer=z12:z12-res.geojsonl \
  --named-layer=z10:z10-res.geojsonl \
  --named-layer=z8:z8-res.geojsonl   \
  -o Output.mbtiles

Visualizing: Mapbox-GL

Loading the resulting tileset into MapboxGL allows for data driven styling across any of the calculated attributes. An interactive dashboard to explore the North America Tileset is available here: mapbox.github.io/osm-analysis-dashboard

Downscaling across Zoom Levels

This first gif shows the different layers (the results of the downscale & aggregation):

Since everything is aggregated per-quarter, we can easily compare between two quarters. This gif compares the number of active users in mid 2012 to mid 2017. Users active Per Quarter: 2012 vs. 2017

New Building Activity

Here is a high level overview of where buildings were being added to the map in the second quarter of both 2015 (left) and 2016 (right). We can see a few major building imports taking place between these times as well as more general coverage of the map.

New Building Activity: 2015 vs. 2016

If we zoom in on Los Angeles and visualize the "building density" as calculated in July 2015 and July 2016, we see the impact of LA building import at zoom 15 resolution:

LA Building Import

Users

The 2010 Haiti Earthquake:

This slider shows the number users active in Haiti during the last quarter of 2009 (just before the earthquake) and then the first quarter of 2010 (when the earthquake struck): Users active during the Haiti Earthquake

We can see the work done by comparing the building density of the map at the end of 2009 and then at the end of the first quarter of 2010:

Building Density increase in Haiti (Quarter 1: 2010)

Ultimately, the number of (distinct) contributors active to date in North America has grown impressively in the last 5 years. This animation shows the difference between mid 2012 and mid 2017:

5 Year Growth

Looking Forward: Geometric Histories

So far, when discussing full editing history, we've only been talking about history of a map object as told through the changes to tags over time. This is a decent proxy of the total editing, and can certainly help us understand how objects grow and change overtime. The geometries of these objects, however also change overtime. Whether it's the release of better satellite imagery that prompts a contributor to re-align or enhance a feature, or just generally squaring up building outlines, a big part of editing OpenStreetMap includes changing existing geometries.

Many times, geometry changes to objects like roads or buildings do not propagate to the feature itself. That is, if only the nodes underlying a way are changed, the version of the way is not incremented. Learning that an object has had a geometry change requires a more involved approach, something we are currently exploring in addition to just the tag history.

With full geometry history, we could compare individual objects at two points in time. Here is an example from a proof-of-concept for historic geometries. Note many of the buildings initially in red "square up" when they turn turquoise. These are geometry changes after the 2015 Nepal Earthquake. The buildings were initially created non-square and just a little while later, another mapper came through and updated the geometries:

5 Year Growth

Location: The Hill, Boulder, Boulder County, Colorado, 80302, United States of America

Tool for measuring elevation above sea level on the OSM map

Posted by Alex-7 on 27 November 2017 in English (English)

I created a tool to calculate an elevation at the point of a click on the OSM map: http://ausleuchtung.ch/elevation/

The elevation is calculated as an average from the values of the ele=* keys found around the click in the radius of 1 (or 2, or 3) kilometers.

I decided to write this simple application after I read an excellent article GPS Altitude vs Pressure Altitude.

For some areas of the world there is a lot of elevation data in the OSM database, and for some there is practically none. I think it is due to the misunderstanding about the difference between GPS Altitude and Pressure Altitude. The article makes it clear that both approaches are not perfect, but these are all what we've got, and we are to use one or another.

The application seems to be simple because all the heavy lifting is done by the Overpass API, OSM, and the Leaflet.

Elevation data could be useful for many purposes. For example, for planning a hiking or a cycling route, for planning a RPAS flight, for water management, etc. For example, if a hiking route starts at the altitude of 400 meters, and ends at 1600 meters, then it is immediately clear that it would be a hard day.

I also plan from now on to measure and add more ele=* data to the OSM.

Feedback and suggestions are very welcome.