OpenStreetMap

Zverik's diary

Recent diary entries

Maps Update: April 17 → May 13

Posted by Zverik on 16 May 2018 in English (English)

A new minor release of MAPS.ME is coming, and just now I've got the new data for it — with numbers on size and features difference with the old data. First, sizes:

  • Indonesia_Central.mwm: 107 up to 127 MB
  • Tanzania.mwm: 212 up to 225 MB
  • US_Indiana_North.mwm: 19 up to 25 MB

Basically the same as the last time, except Portugal was replaced by Indiana state. And there was some prominent activity in all of Peru.

  • Somebody has traced a few thousand highway=track around Buenos Aires.
  • In Patagonia, Argentina, ~300 hamlets (50%) were demoted to isolated_dwellings.
  • Austria_Carinthia: import of ~3000 address points, +50%.
  • Brazil_Bahia: somebody mapped ~7000 power towers, +73%.
  • Brazil_Rio Grande do Norte: from 5 salt ponds to 238.
  • Canada_British Columbia_Islands: from 14 natural=cape to 635.
  • Canada_Newfoundland_West: somebody almost doubled the number of mapped buildings: 487 → 806.
  • Chile_North: 81% of place=hamlets disappeared.
  • Congo-Kinshasa_Kivu: traced ~7000 segments of residential roads, +54%.
  • The same in Gabon: +6000 segments, +160%.
  • Ecuador_West: seems like many supermarkets were imported, 845→1923 (+128%).
  • France_Pays de la Loire_Sarthe: somebody likes tracing fences, 350→3350.
  • In French Alpes, especially in Ardeche, ~9000 intermittent streams were added.
  • In Iceland, somebody retagged ~100 footways to cycleways, and added lit=yes tag to ~700 highway segments.
  • India_Assam: +11500 buildings (+90%).
  • India_Uttar Pradesh: 65 → 247 toilets.
  • Indonesia: so many new roads mapped, most prominent +40k residential in Jawa Tengah and +32k tracks in central Indonesia.
  • Ireland_Leinster: +250 wheelchair=limited tags (+54%).
  • Japan_Chubu Region_Gifu: somebody had a great survey walk there, adding ~50 each shops, temples, information boards, graveyards, company offices etc.
  • Mauritania: from 7 to 87 castles.
  • Morocco: at least 12k new surface=* tags on roads.
  • Namibia: all highway=road and highway=living_street tags were replaced.
  • New Zealand: there's an ongoing address points import, at least ~87k new (+65%).
  • In Nothern Norway somebody imported (or mapped?) ~13k power towers.
  • In Peru, schools and kindergartens were imported: +30k / +186% and +24k / +262%.
  • In Poland, somebody is drawing area:highway=footway, at least 6k new objects.
  • Russia_Arkhangelsk Oblast_North: almost all landuse=garages were removed, 135→18.
  • Russia_Moscow: imported 2700 address points.
  • Russia_Smolensk Oblast: somebody has a good mapping night, adding, among other stuff, 870 barrier=fences.
  • Singapore: 158 → 84 landuse=military. Demilitarization or concealing?
  • In South Africa many rural features were mapped, including farmlands, tracks and especially 2750 waterway=ditch lines.
  • Spain_Catalonia_Provincia de Girona: 388 → 670 admin_level=4 boundaries.
  • Spain_Andalusia_Sevilla and Spain_Cantabria: ~4000 new building:part objects, somebody making everything 3D?
  • The same in Tanzania: +2000 building:part (+78%).
  • UK_England_Greater London: from 78 down to 4 castles.
  • UK_England_South West England_Bristol: deleted all but 99 riverbanks (-97%).
  • US_California_Sacramento_Stockton: imported ~70k buildings, +56%.
  • US_Colorado: removed 2/3 of address points, ~10k.
  • US_Indiana_North: imported 150k address points, +600%.
  • US_Texas: at least 16k new surface=* tags, doubling their count.
  • Vietnam: 73 → 343 jewelry shops.

This list has taken more than a hour to write — but I imagine it took a million human-hours to map all that. Great work!

Not Yours, OpenStreetMap

Posted by Zverik on 8 May 2018 in English (English)

This is a translation of this article in Russian. Far from perfect, but understandable, I hope. Written by Ilya Zverev, CC-BY.

Riding the wave of Google Maps pricing news, Tom Chadwin reminded everyone of benefits of using open source solutions, and concluded with these words: “You now have a concrete compelling argument to those who have always asked: “Why not just use Google Maps?”.

Have I got a platinum argument towards a metaphorical Google Maps: because your open map does not have any future, that’s why. It doesn’t even have good POI coverage, unlike Google, which has franchise owners lined up with location offerings. Because it presents not a dozen grumpy dudes turning every data contributor down, but a nice mat with “Welcome” on it.

That’s an exaggeration, of course. We’ve got a great, beautiful map, which in many areas not only excels — it doesn’t have any alternatives. Nowhere else can you get a reasonably correct road graph. No other map would allow for estimating population density. Nobody would provide you with the data to install a copy of a service in a closed network.

With that, it is hard to not notice that OpenStreetMap is dying. Not because we’ve got a database for a map, or that we don’t have moderators, or the data is not split in layers, like Serge complained. For a technically skilled person it’s impossible to believe in the fall of OSM: the data is detached and decentralized, which is eternal by definition. On top of that, it is free (as in beer) and a million editors contribute to it: why isn’t every website using it?

Because it is unreasonable to use. OSM loses to any alternative maps for one reason: no control. Nobody has it. Over anything. Since around 2012 OpenStreetMap is headed directly into abyss, and rare attempts at correcting the course are met by grumpy blokes, who protect control levers, saying: “power to the community” and “our project self-regulates”. The project’s advantage became its weakness — and, it seems, the fatal one.

No control over the map. Want to import locations for your franchise? Tough luck, your data quality does not meet our standards. Want to map your town? Meet the local gatekeeper, who would scold you for highway tags choice and then disappear, because you are unbearable. And being a watchman is hard: in fourteen years the best we could do was OSMCha. Users of which are still complaining about wide, albeit thin, changesets. We’ve successfully lost the author of OWL. DWG members are still using outdated Perl scripts for work.

No control over the website. This is familiar to anybody who’ve made a pull request to any part of the core infrastructure. You’ll never hear a thanks, but will get a full bag of comments instead. Two guardians do not let through any unconventional change: it’s like amidst a crumbling world we must hold on to what we already have. They don’t see that the power of their grip is what crumbles their world.

No control over the data model. The last effort to advance the API required money and power of the entire Cloudmade company, which means a dozen osmers, working for venture funding for a few weeks. A hope for an “area” data type was faint five years ago, but by now even the most optimistic osmers stopped dreaming of change. The only thing in plans for the new API is restricting metadata for GDPR compliancy, and that’s only because nobody wants to pay charges.

No control over tagging. The main distinction and advantage of OpenStreetMap is a free tagging model. It has grew so enormous, nobody, not even experienced users, can choose correct tags. Forums are full of humor on heath, forest and namespaces. Proposals are a joke: one side invents alien tagging schemas of hundreds kilobytes, another turns inside out in attempts to sink every proposal. Novice mappers are not freaked out by this only because all editors, even on mobile, put tags away behind the presets.

No control over the mapping style. Long ago, the main mapnik style was so complex, people were afraid to touch it. Then it was converted to CartoCSS, made prettier, and contributors started flocking in. For a few years they were improving icons and colours, updated the database structure, sorted out fonts — and the map started looking decent, like on any other popular service. Same bleakness.

But now it’s obvious that nobody knows where to go next. Well, Paul Norman gives talks exactly about this (in my opinion) for two years. It's painful to look at developers’ attempts to continue, especially this year: they fruitlessly try to change established tagging principles, because the OSM data model is incompatible with good carthography. We reached the ceiling of the rendering stack made five years ago. The only way out is to throw it all away and start anew — exactly what authors are discussing these days.

No control over developers. “The most precious resource is the community, which moves the project into its bright future.” Haha, just don’t look at developers, who move in any direction but forward. Some invent a twentieth geocoder, or tenth routing engine; some spend two weeks at data wrangling, only to come up with an unimpressive series of dots. “Hurray, I’ve managed to deploy a tile server”, we hear. Congratulations, but no. It’s 2018 outside, and we’ve got no developer environment, no integration tools, no financial support, no strategic plan. Just some lonely volunteers sitting on key elements of the infrastructure.

No control over license. Mappers want to protect their work, which is understandable. From this were born all the copyleft licenses, to make the world a better, more open place. But there was a flaw. To succeed in this world, you need to learn to bargain. To take a buildings dataset from government, and provide them with geometry updates. To allow a booking service to not open their data, as to get a million and a half verified hotels, and be able to improve position errors in a service used by hundreds millions. And so on.

Our license forbids all that, which doesn’t hurt third parties much: they’ve got enough data on their own. It is us who are hurt, because we cannot make any deals. Community members are on lookout to shut any advances. Even with trivial cases, we’ve got issues. Just this year I’ve seen half a dozen requests to use the map in various TV shows. And every time for similar questions, different answers were given. Nobody, not even the Legal Working Group, understands ODbL. But it is status quo, and in OpenStreetMap, status quo is king.

As you know, in this world to stay in place, you must run as fast as you can. I follow the news for Google Maps, Yandex Maps, 2GIS, and I see them trying new algorithms, fresh points of view. They change user interfaces, always improve data models, learn to communicate with their communities in new ways. They react to problems with structural changes. They have the power to change everything — or the opposite, to tidy up the data, smoothen edges, make it comfortable. They can buy and sell, to make their map better.

All the OpenStreetMap community can do is to gather for a weekend to trace buildings in yet another town. That’s why the main use cases for our project are humanitarian initiatives and serving as a base map layer, when one cannot afford a proper map. Try remembering, what major news we had in the past year, worthy of articles in big tech blogs? A new JOSM release with whitespace trimming in tags?

I agree that relying on a proprietary map means giving up some control to a corporation. But are you absolutely sure you want control over every part of carthographic stack? Do you have enough money for that? A commercial provider can change its terms and put you in an unconvenient situation, but unlike OSM, you can make deals with them. A company is people, who have all the power: you can call them and bargain for better limits, or ask for a help with mapping. You are a client to them; to OSM you, when wanting something for your business, are worse than nobody.

That’s why OpenStreetMap stopped growing. On the charts, you can notice negative trends brewing. Like Wikimapia around 2011, our project has tapped out its meanings. With the current direction, we have ten years, after which we would look like Wikimapia does now: lots of data, no community, who have defected to alternative projects. And then people who chose OSM as a replacement to Google Maps, would have to think again.

In the next year or two we must fix at least some of the issues and find new directions for the project. The “free as in beer” narrative, constant for the past ten years, turned from progressive to pathetic. The main question is, why would you want OpenStreetMap, when you can choose from any of alternative maps, each of which is better in some aspect. And don’t start with “but my yard is mapped better here”. Maybe we plan to revolutionize geography teaching worldwide, or become a universal base map for everyone, or become a framework for experiments in modern carthography. Any answer will do — as long as you are prepared to work.

Until then, most companies would prefer Google Maps.

Maps Update: 17th of April

Posted by Zverik on 24 April 2018 in English (English)

Once or twice a month I build data for MAPS.ME application. After the process of downloading OSM data, building coastlines, extracting objects, generating indexes and packaging mwms is done, the final stage is testing. With that, I get notified when a file size changed dramatically, or many objects of a same type were added or removed in a region. I think you might be interested in some of these changes.

So, what happened between 16th March and 17th April?

  • Indonesia_Central: file size increased from 67 to 107 MB
  • Portugal_South: decreased from 56 down to 53 MB
  • Tanzania: 183 up to 212 MB (and it was just 87 MB a year ago!)

And in objects of specific types:

  • In Argentina somebody changed many road classes, mostly up (res→tertiary, primary→trunk).
  • And number of place=town objects fell threefold.
  • In Australia, somebody added surface=aspalt tag on a hundred thousand ways.
  • Austria_Salzburg: 17 thousand more address points (24k → 41k).
  • Bangladesh: office=ngo went from 76 to 121 objects.
  • Bolivia_South: 123% increase in buildings (39k → 87k).
  • Brazil around Minas Gerais: +30k power=tower and +300 substations.
  • Brazil_South Region_West and Cape Verde: lost all highway=trunks (~50 ways each).
  • Burundi: waterways increased by 100%, e.g. streams 6k → 12k objects.
  • Canada_Ontario_Kingston: no more railway=tram.
  • Czech_Stredni Cechy_East: post boxes: 450 → 1332.
  • Germany_North Rhine-Westphalia_Regierungsbezirk Detmold: 64% of railway=light_rail converted to railway=tram.
  • India_Andhra Pradesh: imported 8000 place=village/hamlet.
  • India_Assam: mapped 6600 buildings (+108%).
  • Indonesia_Central: mapped A LOT of roads (~40k tracks and unclassifieds) and 1.6 mln buildings (+180%).
  • Italy_Marche: imported 3200 address points (+120%).
  • Kenya: mapped 6k power=tower (+130%).
  • Libya: mapped 9500 intermittent streams (up from 113).
  • New Zealand North: imported 305k address points (+172%).
  • Norway: imported many thousands of amenity=recycling all over the country, ~500% increase.
  • Peru_Lima: shop=electronics count increased from 341 to 1447.
  • Peru_South: mapped 43k buildings (+52%).
  • Portugal_South: lost around half of natural=wood, =scrub, =grassland, landuse=farmland (36k to 20k), building:part (2500 to 800), railway=station (291 to 176).
  • Russia_Saint Petersburg: leisure=stadium count went down from 172 to just 36. Yes, it's one of FC 2018 cities.
  • Rwanda and Sudan_West: lost all of highway=living_street.
  • South Africa_Western Cape: added ~7k obscure amenity=* values.
  • In Spain somebody retagged most amenity=doctors as amenity=clinic.
  • Tanzania: building:part count went up from 1081 to 2655.
  • In some regions of the United States, hundreds thousand highway=service ways appeared.

Welcome to OpenStreetMap!

Posted by Zverik on 20 April 2018 in English (English)

— Yay, new mapper! Welcome to OpenStreetMap!

— I see you are using wrong tags. Please read Map Features and don't come back until you're done. I don't care if it's too long. Go to Google Map Maker if you're intimidated.

— I don't know you. Please go to talk@ mailing list and introduce yourself and reasons for your mapping. Yes, I know you have to sign up using some obscure web page, and then receive unwanted mail for eternity. But rules are rules. Go do it, or leave.

— I see you work for a mapping company. I will be watching you, because you do not care for our map. None of you do. Yes, I am working at such company too, but I care.

— I caught you not tracing from Bing like the rest of us. Please explain your sources and acquire specific permissions for using these. Or better stop.

— I see that among 500 buildings you've traced, 20 have been recently demolished. I know that they are still on imagery and in the registry. But it looks like your quality of mapping will never reach mine. Please go away and never come back.

— Hey, I've seen more edits from you. I think we've discussed it. I don't care if they are good, I have reverted them. Please do not come back.

...

— Hi new mapper, welcome to OpenStreetMap!

The effect of the new iD release on restrictions

Posted by Zverik on 22 March 2018 in English (English)

Turn restrictions are pretty common to OSM, them being the second most popular relation type, after multipolygons. Usually a restriction is a relation with two highway ways, "from" and "to" and a "via" member connecting these. The value of "restriction" tag adds a meaning: which kind of turn is forbidden on this route. For example, "restriction=no_left_turn" (and any other no_* value) forbids going from "from" to "to" way using "via" intermediate objects. Alternatively there are "only_*" values, forbidding any routes from the "from" way except the one leading to "to".

Usually "via" members are nodes, which makes them redundant for all types of restrictions except "no_u_turn" (which has the same way in "from" and "to" roles). Thus supporting them in a routing libraries has been easy. But — a "via" member can also be a way. For example, on a dual-carriageway, like the one pictured below. Supporting ways for "via" members is hard, but they constitute less than 3% of all restriction relations, so developers have often ignored these.

Screenshot of the new restriction editor in iD

On the 5th of March, a new major version of iD editor was released. Among the changes, one stands out: the restriction editor got a big update. Now — finally — it allows adding "only_straight_on" restrictions (and others of the kind). And using ways in "via" roles. It looks awesome, and I'd like to thank Bryan for this feature. It is very intuitive, and I like that the iD team puts user experience first.

This change will obviously affect the number of restriction relations that have ways in "via" roles. And these have been hard to deal with. I don't know if all routing engines support such restrictions. MAPS.ME currently does not.

So to decide if supporting these restrictions should be a priority, I conducted a small research on how the change to the editor affected the map. First, I looked at the number of restriction relations by "last edited" date:

A chart of restrictions by four categories

The black vertical line marks the day before the announcement.

You can clearly see that after the announcement, mappers started adding all kinds of restrictions, not just the new supported types. The most popular kind are "no_*" restrictions (e.g. "no_left_turn") with a node in the "via" role. But you can notice the surge in "only_*" restrictions (e.g. "only_straight_on") after the announcement, which by now has almost receded to the pre-March levels, ~100 a day. And there is a clear rise in new relations having ways in the "via" roles, from 10-20 before the new iD version, to 50-200 (!) after. While coming in waves, the number does not seem to decline.

Let's calculate percentages of these previously rare kinds of relations:

Percentages of only_* relations and ways in "via" roles

I have extended the period a bit, to account for a "calm" two-month period. Obviously, the rate of "only_*"-type restrictions did not change at all. The increase in their absolute numbers coincided with the similar increase in relations of other types.

But the percentage of relations with ways in "via" roles has definitely started to rise. From 3-5% before the new iD to 10% and rising after. It is entirely possible this line will settle on around 15-20%, which means around 300-400 new such relations every day. Which means, if you make a routing engine, you can no longer ignore relations with ways in "via" roles.

Traditionally such relations were made for restricting u-turns on a double-carriage highways. This chart confirms it:

Number of no_u_turn relations compared to others

The main increase is still in u-turn restrictions: as many as 260 a day the week after the announcement. Naturally, a developer might think they need to implement the support for only that type of a relation with a way in "via" role. It probably could be done with some kind of a shortcut patch.

Alas, other types of restrictions with ways as "via" are no longer virtually non-existent. From 1-2 a day, mappers now add 20-40 relations of other types daily. Here is the list of restriction types for relations with "via" member ways (thanks Roland and Martin for the Overpass stack!):

  • 23429 no_u_turn
  • 690 no_left_turn
  • 442 only_straight_on
  • 386 no_right_turn
  • 240 no_straight_on
  • 185 only_left_turn
  • 114 only_right_turn
  • 7 only_u_turn

Okay, no way around this issue. Looks like any routing engine that does not support ways in "via" roles won't be able to route properly from now on. And we in MAPS.ME have to catch up.

But... Is that all? Let me show you this:

Chart of relations with more that one "via" member

Yay, iD editor supports restrictions that span two and more ways! We had 800 such relations in total before March, and in just two weeks mappers have added 150 more. I am pretty sure there are no more than two routing engines supporting such chains of "via" members, and that's being generous. So — get to work, developers: the routing over OpenStreetMap has just got much more complicated.

The Subway Validator evolves

Posted by Zverik on 19 March 2018 in English (English)

First, thank you all for improving the mapping of subway networks all around the world! In just half a year, we made more than 150 networks routable, of 180 total. That is very impressive. Today, OpenStreetMap has more data on subways than any other source, open or proprietary.

Last week, I have made a few improvements to the validator. The major one is a change to how stations are counted. We had an issue of transfer stations: for some cities they were counted once, for other — twice, depending on how they are mapped. This simplified calculating a projected total number of stations (just copy it from wikipedia), but affected mapping.

Now, thanks to disc12, it sums up numbers of stations for each line. This is more predictable and allows for different interchange mapping styles. In the spreadsheet, counts of stations have been mostly updated in form of a formula: =line1+line2+...+lineN. You can clearly see how many stations it expects for each subway line. If you find an error there, add a comment and I'll update the number.

Having trouble with missing or extra stations? Click on "Y" near the city name, and you'll get a YAML file with all stations, transfers and lines. What's new is a number of stations for each line (calculated as a number of unique stations for all its itineraries), along with a list of stations. Comparing it to wikipedia is much easier.

There are some improvements planned still. For example, handling of stations under construction: you cannot add these to routes at the moment, or you'll get an error. And there is a "nowhere near the tracks" error that is hard to track — I really should do something with it. And the preprocessor calls for a GTFS output.

Thanks for mapping, now let's finish the last cities and then monitor the world for new subway and light rail stations. If you are an app developer, please consider using the validator output for your app. Contact me if you have any questions.

Subway Routing in Maps.Me

Posted by Zverik on 25 December 2017 in English (English)

Last week we published the latest version of Maps.Me. It's got the major version number increase — 8.0. And not for nothing: there is a brand new "Discovery" button, which shows interesting places around you. There are christmas markets on the map — not from OSM though. You can register as a "local guide", meet new people and show them around your city. Hotels from booking.com can be filtered by price, rating and availability.

But that is not why the release is worth celebrating. The main thing is, we've made a metro routing!

Screenshot of an Underground route in London

It won't be an exaggeration to say that this is the first even public transport routing application that uses solely OpenStreetMap data. Anybody can employ GTFS data, but using OSM is not that easy. All these relations — "route", "route_master", "stop_area", with enourmous tables in the wiki detailing their usage. Utter mess in the data, a result of mapping for a renderer. Very few people understand public transport mapping, so how did we even use it?

We started with a simple task: visualization and route planning for every subway and light rail network in the world. There are only 180 of these: 700 lines (which require at least 2100 relations, as you might know), 11700 stations. To map all of these, you have to get your tagging straight. And that's how the "Metro Mapping" proposal was made. Then I wrote the subway preprocessor, which takes a filtered planet file and produces an easy to use structure for every network, and a validation page, so you know what to fix.

And then me and a few other mappers started improving public transport relations in many cities, mainly in Europe. When we started, there were, like, three good metro networks. Just before the MAPS.ME release, there were 78 in 74 cities. I'd like to thank Claudius Henrichs for improving many routes in Asia: he's the first person outside our company that used the subway validator to improve public transport mapping.

Help us!

What's for the future? The second proposal about rapid transit mapping is being discussed right now, and in mid-January the voting will start. Please read the proposal and if in doubt about anything, write your questions on the discussion page.

180 networks is not too much, but we need your help. Not a city in the United States has subway routing in MAPS.ME. Zero cities in China. If you live in Asia or any of the Americas and want to have subway routing on your next trip, please read the Metro Mapping tutorial, consult the validation page for your country and fix the tagging in OpenStreetMap.

We'd like to see public transport from OSM being used properly, not only in rendering of lines. If you work on an application, please consider using Subway Preprocessor to provide rapid transit navigation for your users. We in MAPS.ME strive for OSM to be used in as many ways as possible, and we continue to work on making public transport from OSM available to everyone.

I broke my streak

Posted by Zverik on 19 December 2017 in English (English)

A few days ago I forgot to spend a minute in an OSM editor. I had almost a month of consecutive edits, level 3 in OSM Streak with almost a hundred points. And then, after a hard day, I forgot about even opening a laptop. Neither an email, nor a telegram notification helped.

Participating three days in a row

The next day I submitted my first changeset in a row. How much did I lose in the game? I had been receiving five points for each changeset (I was too lazy to complete tasks), now I get three, completing the tasks. The fourth level was two months ahead, now... I don't know, two and a half? 500 points is a long way to go.

I am practicing for mapping every day in the next year. Now I have seen how the streak could be broken, and to prevent that, I must not push mapping towards the end of a day, but do it the minute I see a notifidaction. Sometimes the task intimidates me: instead of opening the iD editor on a random place and mapping a building (which takes a minute), it requires opening JOSM, finding a specific place, looking at third-party services like Mapillary, remembering some piece of tagging, adding an imagery layer and uploading changes. All of this fits in five minutes, but there are days when I cannot spare that for OSM.

But seeing the point counter increase is still fun. I invite you to register in OSM Streak and spend a few months mapping. You might like it. Or maybe you will have a few ideas for the tasks.

A year of edits with OSM Streak

Posted by Zverik on 3 December 2017 in English (English)

Hi, I am Ilya and I have been uploading changesets 15 days in a row. Not because I'm so into it or somebody makes me: I've made a tool that reminds me to do it. In a year I expect my HDYC activity chart to be completely filled. And you can have the same too.

Introducing OSM Streak: a website that gives you points for submitting changesets each day. You get 1 point for the first changeset, and then you get more: for example, I will receive 4 points for my next changeset tomorrow. And that is not all: it gives you a random task each day, so you don't stare at the map trying to come up with an idea. For completing a task, you get an extra point. And when you map many days in a row, you gain levels, which open more tasks.

Forgetting to visit a website is expectable, so OSM Streak is also a Telegram bot (find the link on the "Connect" page). With the bot, you can forget about the website: it accepts changesets and sends you tasks every day. Alternatively, you can subscribe to e-mail notifications, which will be sent on 1:00 UTC.

All the tasks and the website and the bot can (and should!) be translated into your language. We have English and Russian, and I would be very grateful for more translations.

Finally, there are 26 tasks at the moment, and you can add more by making a pull request or an issue on github. You can also use the source code as an example for writing a telegram bot, doing migrations with peewee ORM, sending e-mails with python or adding i18n with transifex.

Metro Mapping Proposal (and what's wrong with proposals)

Posted by Zverik on 18 November 2017 in English (English)

Hi. I'd like to ask you to the Metro Mapping proposal page, read it in full and leave your vote below. The proposal mostly summarises subway and light rail mapping practices and introduces a few useful changes, listed in "What This Affects" section.

You will find many opposing votes there. You might be tempted to read only these comments and skip studying the material. It is simpler, and some of the authors are well-known in the community: they must know what they are talking about, right? And there are 8 screens of text, who would read that?

That is what I don't like in the current proposal system. It does not matter what the proposal actually contains, how deep an author knows the topic or if the features in questions are already mapped in a suggested way. What a great power you have when the voting starts — strike out weeks of work by writing a few words! You must be a fool not to excercise that power. And who knows what might happen if the proposal is approved? Everybody understands you cannot edit a wiki page after that. You would need to write your own proposal, and it might be rejected, so it is safer to just oppose.

For the first time in OpenStreetMap history I started actually using the data to route users through subways. And found out the tagging is insufficient: there was no sane way to map interchanges, and you could put thousands of members into some relations. For a mapper things also looked bleak: to map a subway properly, you would have to know how underground tracks go, and where underground platforms are located. No wonder that a beta version of a validator showed that the world had like three subway systems mapped good enough, out of nearly two hundred.

So I've put up this proposal page, that makes mapping subways simpler, and makes using the data easier. You could learn some things from it you've never thought about, like types of interchanges or the importance of colours. Then I wrote about the proposal to tagging@ and talk@ mailing lists, and in the course of a month got many great comments, that helped polish most sections, and rewrite some from scratch, adding more pictures and replacing blocks of texts with simple lists.

I have also finished my subway validator, that not only validates, but prepares data structures for metro routing and rendering. With it, I have been tidying up subways all over the world, finding a few flaws in the proposals, but mostly learning about regional specifics. I've done Moscow, London, Paris, Berlin (both U-Bahn and S-Bahn), Yokohama, Algiers, Lima, Pyongyang and dozens more cities. Their subways conform to the proposal, and I have not broken any external app that used these, or uploaded any extra tag. Well, except network:metro in Germany, since they have an interesting way of tagging networks (thanks Nakaner for the explanations).

With over 60 metro networks mapped, dozens of questions answered, most of which affected the proposal, after discussions in wiki, talk@, tagging@ and in changesets, I decided that the proposal is ready and started the voting. For a moment I thought the OSM community was reasonable. It's like in my seven years I haven't learned anything. Not after the turn lanes proposal, not after hate messages about the water=* proposal, five years after it's approved, not after stalled pull requests to the OSM website.

Very few people in OSM are willing to compromise. Especially people from Western Europe, who think they own the project. Because you cannot edit a wiki page without a proposal, I assume. Because no matter how many suggestions you've accepted, layer=-3 instead of -2 makes the whole work moot. It would ruin the PTv2 schema. Because despite "discussions on the Talk page" policy, when the voting starts, you can write a page of opposing arguments right into the proposal page, and nobody would turn to the Talk page to read answers, not even the person who wrote the arguments. Because it is easier to oppose than to approve: all arguments are already laid out for you, and finding answers to these in a 56-kilobyte Talk page is too hard. For one person, it's because I should have discussed the proposal in his 3-messages-a-month mailing list, not in talk@ and tagging@.

You don't even have to understand PTv2 schema to vote for its extension. Like people who +1'd the Carto'Cité comment about stations as nodes and did not even click on the Metros link to see that OSM wiki had already told everyone to map stations just as nodes. OSM is told to be a do-ocracy, but work does not matter anywhere here. You might have studied prior art and mapped hundreds of networks, but Carto'Cité has an authority over french mappers, so you bet all of them will come and vote against your proposal, even understanding that their arguments have nothing to do with the proposal and are inspired by a potential breakage of some of their inner closed-source tools. By the way, we've talked that over, I've converted the whole Paris subway network to the new way, and nothing broke.

The voting closes in a week, and there are more opposing votes than approving. Of course it would not matter after I have finished my work on tidying up metro systems: the changes will just go into wiki with the "not approved, but widely used" status, like highway=milestone. To help novices understand mapping subways, some of us will direct them to this possibly failed proposal, because the wiki has — and will have — nothing better. The only outcome of failing the proposal will be my continuing frustration of interacting with the OSM community.

So please, if you care about our data not only conforming to some proposals accepted 6 years ago without practical usage, but also being relatively easy to use, please go to the Metro Mapping proposal page and leave your approval. I assure you the OSM database won't go down in flames after that. Thanks.

Composite keys in OpenStreetMap: ref:highway, highway:ref or highway_ref?

Posted by Zverik on 3 October 2017 in English (English)

We all used building:levels and alt_name without giving it a second thought. Why are these keys built that way? Why not levels:building? To me, it looks like there is a rule for building composite keys.

Suffixes

ref is the basic tag for storing a reference number. For a reference number in some third-party table, we add a suffix: ref:third_party. That is because the new tag still contains a reference number. We have all such numbers in ref* keys. The rule of thumb is, the meaning of a value is defined by the basic tag before the suffix. ref:third_party is still a ref, and source:maxspeed is a source.

Sometimes we cannot use suffixes for historical reason. That is the case with name: we use name:en and other suffixes for names in other languages. For that reason, we build composite keys by prepending a content with an underscore: local_name or place_name. These are still names — a reversed order from the semicolon notation.

Of course, an underscore is also used for multi-word keys: public_transport and admin_level.

Namespaces

Then, there are namespaces. The most known is addr: with addr:housenumber and so on. Without a suffix, addr key has no meaning. The same with contact: and turn:. Namespaces are used for marking a group of tags that have the same meaning, have similar value formats and they are usually described on a single wiki page.

Some namespaces are used for tying properties to a part of the object described by the main tag, and for adding more specific properties of it. For example, building:* tags describe attributes of a building, and we also have roof:type and fire_hydrant:type. These words are most often put on the same object as a key or a value, e.g. building=yes or amenity=fire_hydrant, but also can mean a part of a structure denoted by these tags, like how buildings almost always have a roof.

The definition for namespaces is very vague, and some people mistake basic tags for namespaces. For example, we have 2.6k addr tags in the database. Sometimes people try to impose an prefix on a set of established and well-used tags to group these: it improves sorting in editors and allows for introducing many more similarly-named keys without "polluting" the namespace-less set of tags. That is what happend with "contact:" prefix: it is rare to see imports using "phone" and "website" tags without it.

Suffix or a namespace?

Telling a basic tag with a suffix from a namespace of the second type is harder. For example, what would be correct, building:height or height:building? roof:height or height:roof? This depends on four things:

  1. Which of the basic tags for each of parts is used more often, hence is expected to come first? In this case, building is used 28 times more that height. roof key is virtually unused.
  2. Which of these parts is more commonly used as a namespace? height: is used as a namespace for only three popular (more than a hundred usages) keys, none of which is globally spread. For building:, the number of prefixed keys with more than a hundred usages is around 120, for roof: — around 30.
  3. When removing the suffix, will the value be meaningful for the basic tag? It definitely won't be for building=100 m and roof=100 m, but will be for height=100 m.
  4. Will the basic tag without a suffix have the same meaning for the kind of objects with other similarly namespaced keys? In case of buildings, height would be enough without a suffix, and these tags are pretty widespread. But roofs are parts of buildings, so you would have either a suffix or a namespace.

So, for building height you would use a plain height key because of the fourth point. But for roof height, you would choose roof:height because roof: is commonly used as a prefix, as per the second point, unlike height:.

A case against brand:wikipedia

The reason for this post is the recent import of thousands of brand:wikipedia and brand:wikidata tags. I argue that the better choice would be wikipedia:brand and wikidata:brand, for the same reason as source:maxspeed and ref:whatever.

I accept the introduction of separate tags for an object and its brand: we can have two links for the McDonalds brand and a single notable restaurant under that brand. That covers the item 4 in the above list, and item 2 is not applicable, since both wikipedia and brand keys have not been used for namespaces. But points 1 and 3 are in favour of wikipedia:brand: the value is still a wikipedia article, and it is processed similarly to the value of wikipedia tag. And we have four times more wikipedia keys than brand keys.

To conclude, I suggest we do a mass-retagging of these imported or automatically processed keys before this mistake creeps into the wiki. Either wikipedia:brand or brand_wikipedia would be better options.

Past mistakes

In some cases we failed to notice composite keys in proposals that are built contrary to the norm described here. Now you have to do some non-obvious tagging, which requires looking for the correct keys in the wiki:

  • bridge:name instead of bridge_name (like old_name)
  • source:ref, though the correct key source_ref is used 10 times more often. Note that ref:source would not be entirely correct, since you should be more specific in the suffix. source=tmnt with ref:tmnt=1 would be the correct choice, better than source_ref=1.
  • This whole section on *:wikipedia prompted by this edit. Thankfully, we have only 20k of these keys, including the imported brand:wikipedia, so there is still time to fix this.

OpenStreetMap Awards 2017

Posted by Zverik on 28 July 2017 in English (English)

Did you know that the community voting for the OpenStreetMap Awards is open right now? We have chosen many nominees that did a great job last year, 45 of them! Not as many as the total number of active community members, but still a lot. And nine will receive the award at the Stat of the Map in August.

All nominees will be features in the series of OSM Blog posts: https://blog.openstreetmap.org/tag/awards2017/

Help translate the website and project descriptions to your language: https://www.transifex.com/openstreetmap/osm-awards/dashboard/

The voting closes on 16th of August. Vote now — you can change your choice at any time. Nominees would be glad to hear they are supported by hundreds of voters, so we would need at least a thousand people to vote.

Promote the Awards in your local communities, and if you are nominated, do encourage your subscribers to vote for you :) That is okay, provided you don't pay for votes or anything like that.

Better Walking Papers

Posted by Zverik on 5 July 2017 in English (English)

Walking papers from the Tula Mapping Party

I have talked publicly about improvements to walking papers since at least SotM 2013. Made a blog post here in 2014 with some thoughts. But all I've seen were new ways to print tiles or atlases. While I admire the Field Papers and MapOSMatic fork improvements over the past years, a good walking paper is more than that.

For a long time I have been using a 28-step process to prepare walking papers for my mapping parties. It involved using Maperitive, Inkscape and some proprietary software. This year I finally got fed up with reanimating that old renderer, which doesn't work perfectly on Linux, and tried something else. I had always been recommending QGIS for printing maps, and I decided to try it myself. Turned out, making walking papers with it is really simple and straightforward, albeit not without issues.

I started writing another guide with QGIS and GDAL and all the new tech, but it quickly grew to 22 steps. Still too many. Having discovered the Python Console in QGIS, I started experimenting with automating a few tasks. One thing after another, and now I have automated almost everything, fixing a few issues in QGIS on the way. I present to you...

QGIS with a Walking Papers popup menu opened

Walking Papers QGIS Plugin

It is the simplest way to prepare good walking papers for your mapping party. All you have to do is sketch the pie, and the plugin does the rest. Here are the complete instructions:

  1. Install the "Walking Papers" plugin from the official QGIS repository.
  2. Click the button with blue rectangles and choose "Download OSM Data".
  3. On a layer it created draw a polygon around your mapping party area (click a pencil button, and then "area" something near it. Left mouse button adds a node, right button closes the area), and choose the same menu item again.
  4. Yay, we've got a map. Sketch the pie with lines in the "Pie Overview" layer.
  5. Having finalized the pie, activate the "Pie Sheets" layer and draw areas around quarters that go on each of the printed sheets. Usually it's 2-3 sheets per a pie piece. Name areas like "4-west", where 4 is a pie piece number, and "west" helps a mapper to locate themselves.
  6. Click the blue rectangles button and choose "Prepare Atlas". That's all, check out the sheets and print them or export them to a PDF file.

Amazing, right? For a regular mapping party this way of preparing walking papers gives you much more control, and you would need to do much less explaining when handing these sheets to participants. Here is why I prefer it to atlas-printing websites:

  • The data is very recent. It is downloaded from Overpass API, and you don't have to wait for a server somewhere to catch up. Buildings missing? Ask mappers to help drawing them, and print the papers an hour before the party, with everything they managed to draw by that time.
  • Custom map style. With online services you have basically one good choice: Stamen's Toner. It is not perfect for walking papers: labels are in English, lines are too thick and dark, buildings don't have numbers and are hatched, so you can't draw anything on top of them, and the water is awfully black.
  • Vector maps. You are not limited by zoom levels, and thickness is specified in millimeters, not pixels on some maximum zoom level.
  • Custom attributes. The bundled style prints house numbers and building heights on buildings. It is not easy to alter that at the moment, but by manually editing osmconf.ini and wp_style.yaml files in the plugin directory ($HOME/.qgis2/python/plugins/walking_papers) you can add any attributes and change the style however you want.
  • Rotation. It is frustrating when the roads in your mapping area go in 45° angle on the map, which makes most of the space on walking papers sheets unusable. With this plugin, maps on your sheets are rotated so objects on the map are as big as possible, and you have plenty of space to put down POI names and house details.
  • Speed. No more waiting for an hour while your task crawls through the queue. Click a button, get an atlas, that's all.
  • Works offline. Download a map area in JOSM beforehand, or copy it with a flash drive from a connected computer, and use the "Open OSM Data" menu item.

I hope this plugin helps you with organizing a mapping party. We know these don't help with attracting new contributors, but parties are fun, you get to know your city or village better, and the amount of data you collect is unmatchable by any other data collection method.

The 2016 Board Elections Statistics

Posted by Zverik on 23 January 2017 in English (English)

The OSMF Board elections happened a month ago, but only now I've got my hands on the anonymized ballots. Which means we are getting some statistics.

First, let's look at numbers. By the 10th of November we had 457 members eligible to vote. That is much lower than the previous year, when we had around 500 eligible members. Still, that's just a tiny fraction of active OpenStreetMap editors, so the room for improvement is huge. Other numbers have also dropped: 4 candidates instead of 11 in 2015, and 253 voters (55%) instead of 272 (~53%).

It was definitely the fastest and simplest of OSMF elections: with only four candidates, two of whom were running for re-election and other two much less known in the community, 77% of first choices were either Frederik or Kate. That is, Kate got three times as many first votes than the runner-up. I doubt there is a system in which the outcome of the voting would be different.

A chart for mentions and first places for each candidate

Let's look for the third place. Guillaume got 27 first votes and Darafei got 29. But did Darafei take the third place? By now you should know the STV does not make the answer obvious: it depends on how many people have listed these two as their second choices. And we see that twice as many Frederik's voters had chosen Guillaume for their second vote: 38, and 19 for Darafei. Kate's voters, on the contrary, preferred Darafei: he got 23 vs 18 for Guillaume.

Since the weight of Frederik's extra votes was much higher (0.4 vs 0.3) and more people gave the second vote for Guillaume, he gained most after the two transferral rounds: 60.2 vs Darafei's 55.4. Which would make him the third winner, had we another seat to fill.

Five ballots with just one candidate listed had either Frederik's or Kate's names on them. One ballot was empty, and 184 (73%) had all four candidates. Just one ballot did not have Frederik or Kate listed. 17 ballots had Frederik and Kate for third and fourth choices.

Darafei would take the third place in “Approval Voting”: if we count all mentions of candidates, regardless of a position, then Frederik and Kate would get 234 votes each, Darafei gets 217 and Guillaume 207. Incidentally, Darafei gets the most “disapproval” votes, that is, he is listed fourth on 67 ballots out of 184 that have all four candidates. Frederik has the fourth line only on 29 ballots.

A chart for number of mentions on each place for each candidate

Here is the original ballot file, which you can feed to OpenSTV to check on numbers, or analyze to get more statistics.

The OSMF Board did not change this year, which means we don't have to learn new names, but also that we won't hear new voices. I really hope next time there will be a lot more candidates, especially from under-represented countries. You don't have to be fluent in English or know accounting to serve on the Board. An urge to make OpenStreetMap better and an understanding that you would need to move in small steps is all it takes. Do push yourself for the elections this Autumn, and thanks for choosing us, the current Board members.

POI Edits by Editor

Posted by Zverik on 29 December 2016 in English (English)

The editor usage stats wiki page has two useful tables for comparing editors: by number of users and by number of edits. And while MAPS.ME reach is clearly visible in the first one (alas, it hasn't still surpassed iD in users), the second table has much lower numbers for the mobile editor. Which is reasonable: MAPS.ME does not allow editing geometry, thus a single user can make more edits in a day, than all MAPS.ME users in a month.

I was wondering if we could compare only the type of edits our mobile editor allows, that is, POI edits. And with daily replication diffs, I did:

Created and Modified POIs by Editor, total

This chart shows that while users of iD and JOSM editor create more points of interest than MAPS.ME users, it's not by a large margin.

I noticed days when 10k or 20k new POIs were uploaded with JOSM, and wondered if the picture would change if we took median values for days of each month. That is, from a sorted array of numbers we would take the middle number. Turns out, not much.

Created and Modified POIs by Editor, median day

Most of us know how hard it is to update POI data, besides fixing geometry or fixing tags. Not only you have to go out and collect new opening hours or names, but also you must know that something has changed. Mobile editors usually excel at that, so why editing numbers are higher for desktop editors? That's because I could not separate meaningful tag edits from automated, or geometry changes from tag edits. Maybe if I could, MAPS.ME's share of POI edits would be much higher.

100 € for multiple social accounts in OSM

Posted by Zverik on 14 November 2016 in English (English)

Last time it went pretty well, so let's try another long-standing issue. Namely, #1274.

You can register or log in to the OpenStreetMap website using a social account: Facebook, GitHub, Google. The thing is, most of us have many social accounts, but you can only link one to your OSM profile. If you click a wrong button, you will have to either go back or register a new account. You cannot log in to your account using different social buttons.

For me, that is a problem. I would like to not remember which of the buttons I clicked when I tested the social login, and which of these is linked to my main OSM account. For that I would like a social accounts management in the settings page. Alas, I don't have time and skills to add that, but I have a hope that somebody has. And for that I'm willing to pay a small grant: a hundred euro, like the last time.

The offer stands for two months: the pull request to openstreetmap-website should be submitted until 15th of January and merged until 15th of February. Right after merging I will transfer 100 € to any given credit card or bank account.

Tagged and Untagged Nodes

Posted by Zverik on 18 October 2016 in English (English)

I've just counted some statistics on a planet file from 14th of October. Here it is:

A table with node statistics

This table shows a number of nodes, both tagged and untagged, that are referenced by ways and relations. You can see that nearly 97% of 3.5 billion nodes are untagged, and most of these — 88% — are part of exactly one way or relation. Like, when you trace a building, you add four untagged nodes that are part of that closed way.

98.4% of all nodes are part of something, but only 12% (424 million) have two or more parent objects. This could help with designing a data storage for nodes.

There are equal amount of tagged nodes that are not part of anything, and part of an element. Interesting are these 9 million tagged nodes that are part of two or more ways. The taginfo says there are 2.5 million crossings and 860 thousand traffic signals, so that's a ⅓ of that.

Finally, we have a million of nodes with no tags not being a part of anything. I wonder when someone puts on their OSM saviour cape and a programmer's hat and rids us of these.

Short history of name editing in MAPS.ME

Posted by Zverik on 11 October 2016 in English (English)

Many mappers agree that simple and accessible editors are hazardous: the simpler editor is, the easier it is for a horde of newbies to submit wrong data. This was a main argument against Potlatch, and then iD. Now MAPS.ME built-in editor allows for changing tags and adding nodes with just a few clicks for any of our tens of millions of users. Which of course has led to a number of questionable edits.

Screenshot of the first editor with name field

The first field in any place card is name. When we released the editor in April, it was a single field for editing the "name" tag. You changed a name — the new tag value was uploaded to the map.

Complaints started coming almost immediately. Turns out, some tourists were renaming attractions to their language for easier navigation. If you look at the Questionable Edits wiki page at the time, you'll see that names in wrong languages are the most worrying kind of edits.

How do we fix that? Well, finding the language of the name from its characters could work for some languages (like Chinese vs. English), but not for most. Adding a warning that users should type only what's written on a plate is better, but it was there from the start, and nobody reads instructions. Removing the field completely, like some suggested (along with the rest of the editor) could solve the issue, at expense of the better map.

Screenshot of the editor with local names

In August, we prepared a list of native languages for each country. For example, in Finland it's "fi" and "sv", in Estonia it's a single "et". India has 11 languages, though its regions have less. We took this from the Wikidata, which may be incomplete and sometimes wrong. If you have a minute, check this list for errors. Languages should be ordered from most-used to least-used.

And with that, we completely disabled editing of the "name" tag in the 6.3 release. Mappers were asking, and we delivered. Now users were presented with one or two native language name fields, plus an English name and a name in a user's language. For example, if you are a russian in Helsinki, you'd see editors for "name:fi", "name:sv", "name:en" and "name:ru". This way it was less likely Chinese names would be entered into e.g. name:en. And since the default style on osm.org uses only the default name, changes from maps.me wouldn't be shown there.

Except for new objects: when a user creates a POI and fills any of the native language fields, that name gets copied into the "name" tag. But not when editing. Which started causing another kind of error: when a shop had changed its name, we would get old name in the "name" tag and a new name in "name:lng". It displays properly in maps.me, since we favour localized names, but not on other maps. And some mappers started complaining about equal values for "name" and "name:lng".

Screenshot of the latest editor with native name field

With the 6.4 release, we adjusted the workflow again. Keep in mind that our goal is to prevent accidental mistakes by users, not by experienced mappers who know how the application works. For the latter, we added a special language: "Native for each country" at the very bottom of the languages list. That's right: it is a way to edit the "name" tag directly.

When creating a POI and filling a name in a local language, that name will be not copied, but moved into the "name" tag, so you won't see duplicated values in tags. In my opinion, that's a drawback, but still, that's what mappers requested.

Now the complicated part: when there is only one local language for a region, like in Estonia or US, a user has a chance to change the default name. First, all empty name fields for local languages and English are pre-filled from the "name" tag. If a user have edited names in both languages, this would mean the user knows what they are doing, and the app will put the local/English/any other (whichever is not empty) name into the "name" tag.

Screenshot of MMWatch with ambigous name changes

This still means you will get discrepancies between "name" and "name:lng" values for countries with more than one local language, or with users who don't have time to edit all the fields. Know how to make name editing more safe and effective? Please share it here in the comments: maybe we could make it more transparent or even more smarter.

100€ for a subscription to diary comments

Posted by Zverik on 29 September 2016 in English (English)

Okay, I've got tired of this UI, and I'm swamped with other tasks, but there is some money left from my travel to SotM. So I am announcing a grant: 100€ for a merged pull request allowing people to subscribe to comments in OSM user diaries. (NB: 300€ now, thanks to Stereo and Mikel.)

There should be a checked by default checkbox near the "Save" button ("Receive notifications about further comments"), and a button to subscribe/unsubscribe. All notifications should go to e-mail, much like changeset comments now.

When you have your pull request merged, I'll transfer that money to your card or bank account. And of course I'll publish a big thank you :)

The offer is not indefinite: the PR must be submitted until the 1st of November and merged before the 1st of December. And yes, there might be a competition, in that case OWG will decide the winner by merging a pull request.

Just two days to vote in the OpenStreetMap Awards!

Posted by Zverik on 20 September 2016 in English (English)

OSM Stars from wiki

This Sunday, we will meet the nominees for the first OSM Award and learn who gained the most votes in each of the six categories, including Mapping and Blogging. 650 mappers have already voted, and if you have not, please head to the awards website and make your choices. All you need is an OpenStreetMap account.

The voting closes on the morning of September 22nd, when the Brussels Maptember begins with two great conferences: HOT Summit and FOSS4G.be. After that we will dive into OSM topics at the State of the Map, and on Sunday, before lightning talks and workshops, we will know the winners.

But for now, there are still ties in some categories, and your vote can decide who will get the award. Do vote now and meet us at the State of the Map!