OpenStreetMap

Vision?

Posted by SimonPoole on 27 October 2014 in English.

In 2011 the then current board had a face to face meeting in Seattle and produced a set of “audacious goals” for OSM that were

  • The World’s Most Used Map
  • More Than Just Streets
  • Cultivating Leadership of Mappers
  • Easier Contribution for Non-Geeks

Now some of these would have been directly actionable, but the first two are really “visions” under a different name. And as visions goes they are quite good, far out goal posts that nobody expects to be achieved soon.

And we are still far away from reaching the first goal . There are lots of pieces missing before we can remotely hope to achieve it, but we are on the way.

Steve has long harped about adding more addresses to OSM, matter of fact I have too. From a pure usability point of view if you want to produce a competitive device or app with door to door navigation support you are going to need address data. To be more precise this boils down to adding hose/building numbers with their associated streets (intentional pun) to our dataset in some form.

Note on the side: there are lots of regions that don’t have a conventional western way of describing locations, we don’t have good support for that yet in OSM, solving the “address problem” tends to centred around 1st world countries.

The OSM community has lots of experience with adding addresses by both on the ground surveying and imports of suitable open data, matter of fact we’ve had a full country “complete” for half a decade now. What is however undeniable is that it is slow progress, even importing a couple of million addresses (we don’t have good numbers, but it is likely that there is something between half and one billion address conventional house addresses out there) takes a lot of time if you want to provide some minimum quality and the preferable on the ground surveying tends to be even slower.

But we are making progress and looking at one of the larger countries, Germany, with 81 million population, we are now at roughly 1/3 coverage with a combination of on the ground survey supported by open data and smaller imports, completion likely in 2 years at the current rate.

There are some places where we don’t have a good handle on the issue, for example in the “original OSM country”, the UK, due to the addressing system revolving mainly around non-surveyable proprietary schemes and house numbers playing a secondary role. But we are not the only group feeling the pain there and I’m optimistic that we will find a way out, and if it is simply by replacing such proprietary systems.

To summarize: addresses are important and yes it is something that the community is working on with dedication. There is really nothing visionary about it at all at this point in time, not more than “lets map all roads in Germany”.

Hammers

Steve has a legitimate commercial itch that he wants scratched and he wants that fast. This is not unique, readers following the “imports” mailing list have seen similar requests from HOT, matter of fact one of the HOT requests had that other problematic attribute “non-editable”.

Luckily HOT hasn’t tried to stage a coup d’état to get their way. But what it does tell us is that the OSM data format, its tool chains, its editors and other applications have become extremely popular and people prefer our tools over others. It is a great testimonial to OSM.

OSM was built around the notion of mappers collecting or curating data and then adding it to our central repository, iteratively improving the quality and completeness over time. “fast” and “non-editable” are the antithesis to what normal OSM is about.

Now I’m sure ESRI doesn’t get many complaints about them not providing the ultimate multi-100 GB all free geo-data of the world shape file that includes everything, so why do we? I can only guess that it is because people are feeling left out and think that if their favourite dataset is not included that they are not part of this great community.

But, really, it is just a superficial marketing and packaging issue.

If Steve or better his employer want to create the

ultimate_free_address_superpack_US_edition.osm

containing a quickly thrown together conversion of all the open data address sets in the US, great, the OSMF might even be convinced to distribute it in the same place as the planet dumps. And we have more than enough id space to support conflict free merging.

There is simply no need to compromise the normal OSM community process for a short term gain.

We don’t only have hammers, please stop just seeing nails.

Discussion

Comment from Vincent de Phily on 27 October 2014 at 10:59

That’s very close to what the french community did with their BANO dataset : essentially a big CSV of addresses with many sources, tools to compare back and forth between BANO and osm, and the ultimate long-term goal of getting most of it merged into osm.

Comment from SimonPoole on 27 October 2014 at 11:08

Yes. But integrating additional datasets in our services goes back further, for example the central Nominatim instance uses TIGER address data not directly included in OSM.

What we haven’t done is make using additional data easy for third parties, for example somebody that wants to create an OSMAnd map with additional data. Maybe this is one of the activities that belongs in the 5% that the community wont do that Richard Fairhurst has pointed out and needs to be supported by the OSMF.

Comment from flohoff on 28 October 2014 at 07:52

“OSM was built around the notion of mappers collecting or curating data and then adding it to our central repository, iteratively improving the quality and completeness over time.”

It hasnt been like this everywhere. This might be the altruistic vision of OSM.

Just look at the Netherlands. There were huge imports of AND Data in 2008 and later with Address data just lately. Still the quality is still poor. Mistakes have been made while importing and due to the visually complete data the NL community has failed to attract a lot of mappers. So IMHO NL is a negative example of imports and fast working.

Yes - there a lot of data consumers who want full datasets and they want it now - I’d be one of them. If we had full address data for Germany that would make my life a lot easier.

Still - I am very patient. I am very confident that one day OSM will have all the data i’d need, until then i am working every day to add stuff i need. Addresses, streets, footways etc.

Back when i started with OSM everyone said - “You’ll never complete the street network”. I responded - the more attractive OSM gets the more Work will be done. Its the same now. The more people/companys start using OSM the more data will get added, the more detailed the data will get, the more addresses will be surveyed or imported. There is nothing wrong with this. Its just the reason people contribute to OSM changes. In the early days it was a purely altruistic way of “i have mapped my neighbourhood”. Now its - “damn - that address is missing - hey OSM your dataset is broken”.

Still - OSM is the best Geo-Dataset in the world and it’ll stay that way. There cant be any economical competition when you calculate your man-hours with nil.

Be optimistic and patient.

Flo

Comment from SimonPoole on 28 October 2014 at 08:06

Hi Flo

I’m fully aware of the problematic side of imports, I just didn’t want to side track the discussion. and while I personally would like to avoid imports as much as possible, it is very difficult and perhaps not a good idea to stop people from making their own mistakes.

In any case as I pointed out, data doesn’t have to be -in- OSM to be usable with our tools and apps so hopefully one day we will get to a point were the pressure on the community process will be less than it is today.

Simon

Comment from It's so funny on 28 October 2014 at 19:30

@ Flo as one of the persons behind the Dutch address and building import I’m very interested in some of the things you’re writing.

  1. “Mistakes have been made while importing” Could you be more specific on the address and building import?

  2. “due to the visually complete data the NL community has failed to attract a lot of mappers” a. What mapper% is normal for a country? b. How is the Netherlands doing compared to this measure? c. Could you be more specific on the nature of the relationship between visually complete data and % of mappers?

@ Simon “In any case as I pointed out, data doesn’t have to be -in- OSM to be usable with our tools and apps” Before the import of addresses and buildings I posted a question on the @import list how to get the Netherlands address database directly into a Geofabrik extract. Since I got now answer, I presumed this was not possible. Is it, one year later, already possible to get an address database directly into a Geofabrik extract?

Cheers, Johan

Comment from SimonPoole on 29 October 2014 at 21:05

@Johan I suspect that Flo was not referring to this years import. Wrt Geofabrik it is a privately held company under the control of its owners, I don’t see how a 3rd party could speak for them.

My point was slightly different in any case in that I was referring to non-curated datasets that we wouldn’t want to “force” somebody to use and should always be an add on. So if you wanted a geofabrik extract plus extra addrsses you would download the extract and merge a seperate address datafile for the same area before further processing. I suspect this is possible with tools that already exist.

Comment from Glassman on 31 October 2014 at 21:57

Vision

Creating a vision is more than just having a Board meeting to brainstorm vision statements. It involves listening to stakeholders, doing surveys, testing ideals, and getting feedback. Did any of that occur in Seattle? Is there any documentation?

Our vision should challenging. The vision will help us set broad goals, such as increase gender diversity, addressing, or substantially increase the number of active mappers. Let’s give those goals to the community to find ways to accomplish them. OSMF could help fund those goals

Having goals doesn’t impact the mapper that just wants to map their neighborhood. It might mean better tools to help map.

Address

Addresses have a place in OSM. They should be one of our key features much like roads. In some locations, ie. the US, addressing is much easier. Others, such as some Latin American countries, addresses, as we know them, don’t exist. That doesn’t mean that we shouldn’t attempt to find a solution.

Comment from SimonPoole on 31 October 2014 at 22:20

@Glassman you would have to ask the board that met in Seattle for an answer to your fitst question. As to the 2nd part I actually oitn that out too.

Comment from flohoff on 1 November 2014 at 15:18

It’s so funny: I have been visiting the Netherlands very regularly and since 2009 i am also mapping where i was. From the AND import the landuse is completely broken. Overlapping landuses. Single trees als landuse=forest. footways as service roads etc. The AND Data should not have been imported in the first place. We had imports in Germany too for example the postcode areas. These were offered as osm.xml files so MAPPERS visually validated the data before it ended in the OSM Database. Its a little slower than an automated import but you gain people who identify with the data and fix it where knowingly broken.

Then in 2009/10/11 i added addresses in NL - Not a lot but what i could do in 2 weeks. Those addresses are all gone now after the Adress import. Not that i think that my contribution was better or more complete than the Addressimport - But think about the psychological point - By deleting user contributed data, replacing it with imports you tell mappers that their contribution is worthless. So you destroy your community you started to gain.

Automated Imports might bring you more complete data by the cost of destroying your community.

Comment from mikelmaron on 2 November 2014 at 00:00

Non-editable requests from HOT? You must be confused. Reference please.

Comment from SimonPoole on 2 November 2014 at 00:04

@mikelmaron the UN place id stuff. Note I consider it fairly harmless, but the list could just as well be kept seperate from OSM proper. Not sure what Andrew decided on in the end.

Comment from It's so funny on 2 November 2014 at 14:15

@ Flo

I had done some (much less than you) manual address mapping in the past years. Maybe something like just 7.000 addresses or so. So I know the feeling that can arise when such a contribution is simply being deleted. After extensive discussions in the community (face to face, the forum and the wiki) a number was set: if an importer would have to delete more than a 100 manually mapped addresses that person should be contacted beforehand, to ask him/her to help in the import and to decide what to do with the manually mapped addresses. I didn’t do the import in the area where you mapped, so I don’t know if that procedure was executed correctly in your case. If not: a sincere apology for that. Since I’m not perfect I made some mistakes during the address and building import, but was glad to see that I was being contacted when a person found that out. In every such case it was possible to undo my mistake.

As about your reference to visually complete data and % of mappers and destroying a community by an automated import: I can’t find figures showing that. Personally, I have felt a lot of positive energy in the Dutch address and building import, of which I’m glad that more than 40 community members took part (1 out of every 400.000 Dutch people).

Log in to leave a comment