Recent diary entries
I'll be missing this weeks hack weekend in London with the topic of "mobile development", unluckily it turned out to be on the one weekend this year on which I really coudn't find a way to attend. In any case I wanted to at least give an update on how far I've progressed with the work on vespucci 0.9.7 which is scheduled for release early in Q4.
From a development perspective I try to get stuff that involves major internal changes done as early as possible in the ramp up to a release, for the obvious reason that we then have enough time to find issues in completely new code.
For 0.9.7 the big refactoring has been in the Notes code which in prior versions has been online only (you needed to have a working network connection and any changes to the Notes had to be committed immediately). I've reworked the whole subsystem to work more like the way vespucci handles OSM data: you can download Notes for an area manually or using a auto-download setting, work on them, save them locally for uploading at home or upload them immediately. And on top of that the same mechanisms support Osmose bug reports.
A further long time enhancement suggestions by our users was to support translated presets, for both technical and licence reasons we can't fall back on the work that has already been done for JOSM (vespucci uses JOSM format presets) and I ended up doing it rather different way and using the GNU gettext format for holding the translated strings (there is a long and boring story why the Android i8n system can't be used). The other preset related thing requested was a search function, which with the translated presets will work with those languages too.
As you can see the search seems to be a bit too fuzzy right now and will need some further tuning (some of the fuzziness is due to searching both in the original preset names and translated ones).
Back to the hack weekend and more interesting stuff. The current build of 0.9.7 contains substantially better support for generating notifications (on your phone and on any attached android wear devices, see my last diary entry) and includes support for Osmose bugs. This is fine where I live but in high error regions will potentially swamp you with notifications, some of this can be avoided by filtering the Osmose bugs, but some kind of distance vs. seriousness filter would likely work best.
The other hot topic is voice support: the current build has an experimental "pure voice" interface that IMHO doesn't really make a lot of sense and goes against the basic philosophy of vespucci that you should nearly never need to fix data after entering it on the road. What seems to work far better is a hybrid approach that I've built: mark the position where you want the new object (as always in vespucci with a long press) and then use a voice command to actually select what you want to add.
Press the microphone:
This works better than expected, particularly for names that are in the name suggestion index. Naturally likely one reason this works well is because googles voice recognition works better with well known name and terms.
If you want to play around with the current build (and please: feedback is welcome), as always it is located on google drive
Now that 0.9.6 is out, my focus is, naturally, on the next version. Hopefully 0.9.7 wont take quite as long to be ready, I'm fairly optimistic that we will be able to hit the planned release time frame of end of September. The larger operations on the guts of the app have already happened and should have enough time to stabilize till then.
If you are interested in what is planned see https://github.com/MarcusWolschon/osmeditor4android/issues?q=is%3Aopen+is%3Aissue+milestone%3A0.9.7 Two of the items have already progressed pretty far: putting the code in place for translating the presets and the integration of notification support.
The later was mainly written as a proof of concept early this year at the Karlsruhe hack weekend but needed some work to be really useful. Some images from Karlsruhe:
Alert on a watch (naturally works on phone too) from an error detected in the nearby OSM data
the same for a Note
and navigating to the location with OSMand
A test build with the inital (still needs a bit or work for production) alert support is available here: https://drive.google.com/folderview?id=0B9pKLmh8s1h8bFI5bGd4VnhYWkk&usp=sharing . You will need to turn on alert support in the advanced preferences and have "auto download" enabled in the GPS menu (if auto-download doesn't initially work try toggling the setting).
While I have your attention: one of the work items is redoing the Notes code to work more like the rest of Vespucci and allow offline use. Are there any specific wishes for how it should behave (I will likely leave a way to immediately save and close a note in the app, but the normal mode will be a bulk upload at the end of your surveying/editing session). If you have some suggestions please add them to: https://github.com/MarcusWolschon/osmeditor4android/issues/228
In case you didn't notice we have pushed 0.9.6 to the regular google play store. Release notes can be found on device and in the repository. There is likely to be a small maintenance release soon that will address a handful of minor issues.
If you do experience an issue or even a crash, please submit the crash dump if any and check our issue tracker for known issues and if appropriate open a new one. Vespucci supports a large number of different devices over a wide range of Android versions (2..2 to 5.1.1) and the issue you are expierencing maybe not be repeatable on other devices or circumstances. A post on the google play store does not get problems resolved.
In all the noise about MapBox's Series B offering and their successful bid to replace MapQuest's in house map rendering capability, it seems that our dear trade rags missed something.
Likely the most important medium term aspect of the successful bid is that it removed funding and support for a competing vector tile rendering stack that MQ was developing internally.
Anybody that has been following the developments knows that while open source and in principle freely available, the MapBox vector tile stack doesn't work "out of the box" in any reasonable meaning of the words. It follows the trend of the bits and pieces of MapBox's technology becoming increasingly more difficult to use in practice by the community. The other well known example is MapBox studio, the follow up to the widely independently used TileMill.
A recent article actually points to parts that are closed source, a not completly unexpected change of direction.
Now I think we all realize that it is just a matter of time till the open source community catches up on the vector tile front. This will address some of the issues the OSM community has been having with its map rendering and even the playing field a bit. But MapBox has clearly bought themselves some more breathing space for now.
As an experiment we have made the current release candidate available via the google play store beta program. This allows you to replace the release version from the play store with a pre-release version and back again.
To participate you need to:
- be signed up for our discussion group https://groups.google.com/forum/#!forum/osmeditor4android
- visit https://play.google.com/apps/testing/de.blau.android (at least for me this has only worked with a browser on a mobile device) and join the beta program
- you should then be able to access the beta version via the normal play store app
The underlying reason for trying this out is that we currently do not get enough feedback from the testing period and this should make it easier for users that don't want to use alternative ways of installing apps to participate.
A release candidate of 0.9.6 is available https://drive.google.com/folderview?id=0B9pKLmh8s1h8bFI5bGd4VnhYWkk&usp=sharing Note the "signed" APK is signed using the same key as used for the play store and should not need an uninstall if you are replacing a 0.9.5 install.
The only change that is forseen for release, outside off resolving any last minute issues, is inlcuding the release notes.
The last new version of Vespucci with major changes was at the end of 2014 and it has taken a lot more time than I wanted to get near to completing the next version. But now a lot of things are coming together and I'm expecting that we are only a single digit number of weeks away from a stable release.
Just some of the major features:
- multi-select mode
- restyled tag editing with split pane layout on tablets.
- selective tag copy, cut and paste.
- lots of relation editing improvements
- start a camera app directly
- go directly to preset screen when creating new elements.
Besides the multi-select support most changes have been geared towards making the user interface more consistent and easier to use.
Given that the new property editor is so much better, I've added a way to quickly and easily switch in to the "tag editing-only" mode that has been hidden in the deprecated modes preference for the last two years. This should not be confused with the current raft of "lets add a node regardless of what is already there" pseudo editors, all available OSM objects can be modified, just the geometry is locked.
For developers: one of the major hills that had to be climbed was refactoring the old "TagEditor" activity, likely some of the oldest code in Vespucci. Written by the original Vespucci author in the first months of 2009, by both OSM and Android standards it was pre-historic. I added a significant amount of code two years ago for relation support, but hadn't touched the general structure and monolithic mass of code that it had grown to, and that had started to be a big road-block to improving and adding functionality. One immediate benefit of the refactoring is that Vespucci is much improved on tablets and uses the available screen real estate a lot better.
Watch this space or twitter for the announcement of release candidates, as always any help in testing and coding is welcome.
I spent some time today producing some numbers to show how large the effect of old accounts returning and starting to edit actually is and determined the annual numbers of new mappers that created their account at least two years back. For 2014 this would be accounts created in 2011 and earlier.
Year New mappers with old accounts 2012 1'817 2013 2'241 2014 2'848
To give some perspective: 2014 missing maps and HOT together attracted roughly 1'500 new mappers. The total number of new contributors in 2014 was 105'612..
Thanks to TomH for providing some of the data required for the stats.
People who have read Gary Gales blog post on Geohipster will have noticed that one of his claims is that OSM is "business unfriendly". It is a reoccurring theme in discussion with people from the geo-industry however in many many discussions and contacts with companies outside of geo** it has never been an issue, and so the question should probably be reformulated as:
Is OSM geo-business unfriendly?
Well, my answer is, you expected this: no.
It is obvious simply by observing the many thriving businesses that would not exist without OSM and the way OSMs "business-model" is structured.
By positioning itself as a data collection project OSM has left lots of space to build businesses using OSM data and providing services on top of it. This is in stark contrast to say Wikipedia, which has always positioned itself as the one-stop shop for WP content and services.
Would a MapBox exist if OSM had chosen a more Wikipedia like model? Naturally not. Would MapBox cease to exist if OSM changed its mind today? Probably not, given that they have moved away from being a one-trick pony, but it would be the death knell for a number of other players.
But no fear, a further reason that OSM is extremely business friendly is that we have held a steady course over the decade the project has existed. Major changes have taken place over a long period of time with lots of time to adapt.
Now there is a certain slow feature creep with respect to services provided on the central OSM site which will continue to raise the bar of the minimal functionality for a viable online map portal, but anybody endangered by this should likely rethink their business model in any case.
Community run and developed software and services will likely have more impact. OSM provides a level playing field and your business model should take competing with non-commercial services in to account (note how OSMand in continuously improving).
On top of the above OSM is extremely cheap for business. Not going down the Wikipedia route has enabled OSM to produce all this good stuff with a minimum of fixed costs. The annual budget of the OpenStreetMap Foundation (the formal body behind OSM) is roughly a 1/500th of its Wikipedia counterpart, the Wikimedia Foundation. There is no obligation for a business to donate to the OSMF and a major part of its income has been from donations by individuals.
A further common complaint from the geo-business pundits is that the OSM community is business hostile, however occurrences of this can essentially always be traced back to the company in question trying to force something on the community, or trying to telling the community what to do instead of being part of it.
Matter of fact the OSM community has an extremely laissez faire attitude towards business involvement in OSM. It is difficult to find somebody opposed to building businesses on the volunteer work, and the boards of two of the most influential OSM related organisations (OpenStreetMap US and the OSMF) are dominated by industry representatives. Dominated as in: a single token non-industry involved member in each.
Garys superficial main beef with OSM is however the current distribution licence, the ODbL.
Over many decades essentially all businesses dealing with geo-data have had business models where they would obtain data from various sources, add in some self-surveyed information and sell the result either directly or services built on top of the resulting dataset., negotiating contracts at every step in the process. This goes for essentially every player from Tomtom, Nokia and google at the top, down to smaller players, excluding essentially only the national and regional government operated GIS departments. Doing business this way is deeply ingrained in the thought patterns and culture of the whole industry.
The advent of open data, mainly open government data, has not changed this. What it has changed is that the cost structures of the businesses have improved. You shouldn't be fooled by the marketing, many of the “disruptive” geo-businesses are simply using the tired, age old business model with lower costs. The benefit is that they seem to have a bit more lee way to do cool things right now, but that will change when everybody has caught up.
OSM is the odd kid on the block- Now I don't want to dive in to yet another licence discussion. As has been pointed out many times, many of the issues with the licence are make believe, the real main issue the geo-industry has with OSM is that we don't conform to their traditional business model and they are having problems to adapt. You can't simply haggle a contract with OSM, everybody gets the same.
With other words: their problem is that OSM is truly disruptive and different.
Is this “business-unfriendly”? No, it opens up lots of opportunities for geo-businesses that are willing to adapt, for the others: its capitalism, bye bye.
** it should be noted that non-geo businesses don't seem to have problems googling for the OSMF and sending mail or e-mail with inquiries and questions. The geo-industry on the other hand seems to be so ripe with google-challenged people that it was possible for MapBox to fill a full “legal” marketing piece with complaints by them.
It is time that we lay the geocoding related licence discussion to rest by forming consensus on a guideline.
It is well known that I support the concept that the results of bulk geocoding form a derived database and support the corresponding conclusions on the [Geocoding Guideline page](//wiki.openstreetmap.org/wiki/Open_Data_License/Geocoding_-_Guideline) .
However [Example 7](//wiki.openstreetmap.org/wiki/Open_Data_License/Geocoding_-_Guideline#.287.29_Address_Search_for_Points_of_Interest_Databases) glosses over a point that has been raised for example by Steve Coast in the past: are failed geocoding results really free of OSM intellectual property? For clarity: we are not discussing on the fly gecoding as there is no database created and nothing to share.
We need to resolve this to move forward on the matter.
I don't believe there is a clear and conclusive answer to the above and there is a certain danger of getting in to "how many angels can dance on the head of a pin" type of discussions, so I believe that it boils down to: with what is the OSM community happy? Naturally with the backdrop of the ODbL in mind.
I suggest something very simple: that the set of failed addresses (or more general: input data) should be shared with the OSM community. I am not saying that the failed addresses are subject to the ODbL SA clauses, just that we should treat them as if they are.
Now you might ask why would we be interested in failed addresses? On the one hand these can be mined, just as the successfully geocoded ones, for additional information, for example for house number -> post codes relationships and on the other hand the list of failed addresses is obviously helpful for quality assurance.
And I believe that this, particularly the later point, creates a win-win situation for the organisation doing the geocoding and for OSM. The win for the geocoding organisation is that more of its addresses will be found in OSM and the reliance on third party datasets will be reduced.
Now assuming that a consensus forms around the above, there is still a slightly touchy issue in that companies may not want to be identified as the source of specific addresses. To resolve this I propose providing a facility by which such input datasets can be provided to the community and published anonymously (there is at least one system in existence that could simply be cloned to provide this facility).
Note: all of the above only applies to datasets that are being publicly used so there can't be an expectation of a high level of data privacy to start with.
Two days back there was a longish discussion on the OSM IRC channel about supporting OSM on XSCE, aka on offline, potentially slow devices. Currently it seems if they (XSCE) distribute a pre-rendered set of tiles, and during the discussion (which was mainly about alternatives to distributing tiles) it was mentioned that it would be nice if they, besides a slippy map, could provide a search function.
Now given that disk space idoes not seem to be an issue in the project and keeping in sync with OSM central is not a requirement, it occured to me that Photon might be a viable way of providing a global search function.
Installing it on my PI2 (running Ubuntu 14.04) was surprisingly easy and the only painful part was downloading 30GB of search index over my not particularly fast Internet connection.
[Photon on a PI2](//wiki.openstreetmap.org/w/images/2/24/Photon-pi2.png)
Now perfomance is not great, something like 7 to 10 seconds for a query, ruling out using "search as you type", but bearable for individual queries. Potentially single language search indices would be faster, but that would needed to be tested.
Naturally alternative approaches for example using mapsforge might be better and would get around the requirement for prerendered tiles, however it is not clear if a global map would actually be feasible.
Anybody who has any interest in the growth of OpenStreetMap has probably read at least one paper or blog that has moaned about "only" a couple of 100'000 users actually having contributed out of the 2 million + that have signed up for an account.
The over 500'000 contributors are a good 25% of the total registered accounts and I'm not sure if that really counts as "only" given that we don't really have any comparative numbers, I would suspect it is actually very good.
In any case there have been calls to simply delete the "inactive" accounts as they inflate the numbers and in general do no good. I'm very much against that for two reasons: on the one hand we don't know why inactive members have joined, maybe they simply wanted to show support, maybe they wanted an OSM account for autentication in uMap or any of many other possible reasons. On the other hand they are a reservoir of new mappers, and every day we likely have dozens of old accounts starting to map.
As an holdover from the licence change I'm still running a script that produces a daily list of old accounts that have newly accepted the contributor terms and typically there are a dozen or so each day. These accounts have not been active since at least April 2011, when we had roughly 350'000 accounts total. The majority tends to be accounts that hadn't previously edited but there are always a couple that somehow didn't get the message during the licence change and had actually edited more than 4 years ago.
In any case the tl;dr version: deleting the 1.5 million inactive accounts would deny us the pleasure and fun of welcoming new contributors like https://www.openstreetmap.org/user/Geocurioius to the active mappers.
Two years ago I produced some statistics on addresses in OSM. While I did regularly re-run the scripts, I never really revisted the topic to see how things are developing,
2013-03-27 2015-04-09 Increase since March 2013 Total 20'168'470 51'903'310 257% Germany 3'659'043 8'470'716 232% USA 2'090'893 5'229'243 250%
The complete set of numbers can be found here, According to official statistics there are 18.5 million residential buildings in Germany so I would expect that we are roughly 1/3 to 1/2 of the way to complete coverage there.
The large global increase in early 2014 is mainly due to the import of 8.7 million addresses in the Netherlands which due to peculiarities in building and numbering in NL led to a far larger increase in the counts than you would normally expect of a country with a similar population size.
When I first produced the address stats the openaddresses.io project didn't exist and it it is now interesting to compare the two projects. It should be noted at this point that openaddresses.io does not make any representations wrt licences of their individual datasets and there is no guarantee that all or any of the data could actually be used in OSM.
OA currently claims to contain 115 million address, slightly more than twice the 52 million in OSM. However how large is the overlap? A very rough (and unfair) estimate based on the numbers for the US, ES, NL, PL, DK, NO and JP in OSM, would indicate that 24 million address likely turn up in both projects, but, more interesting that OSM contains 28 million addresses not in OA. Or put differently a combined dataset would contain 143 million addresses (see caveat above).
SOSM operates a fair number of services mainly for the Swiss community that up to now have been hosted on a single small server leased from Hetzner. While this is a very cost worthy solution, it doesn't have any redundancy built in and further requires cross border Internet traffic to access, which raises privacy concerns at least with parts of our community. SInce we started operating the server a bit over a year ago it was clear that a local solution would be preferable.
Due to the good relations between SOSM and Wikimedia Switzerland late last year we were pointed to the fact that Wikimedia Germany was throwing away their old hardware which used to host the "toolserver" and that while most of it was clearly not worth continuing to use, there was three couple of years old Dell R520 servers that might still be useful. While we needed to find an affordable hosting location to go with the machines (more on that soon), it was clear, if at all possible, we would like to secure the machines for SOSM.
Wikimedia operates a larger site in the Netherlands in a commercial hosting facility just outside of Amsterdam in Haarlem (yes Haarlem and nearby Breuklen gave the names to the Harlem and Brooklyn in NYC). They don't have direct on site staff and it was clear that we would have to arrange pickup when somebody was there dismantling the hardware. This turned out to be bit of a challenge and in the end, after determining that we wouldn't have to commercially import the hardware (no financial difference, just a procedural one), we decided that the easiest and most efficient, perhaps not most ecological, solution would be simply to drive up to Amsterdam short term when Wikimedia was ready and pick up the servers in person.
Last Friday was the day, Michael (user datendelphin) and myself started on our mad dash to Haarlem and back, a total of over 1'700km. To at least get some more out of it we decided to record as much as possible of the drive with Mapillary and naturally navigate with OSM only.
An old S2 for Mapillary with two empty 32 GB and OsmAnd for navigation.
I initially tried to run RTKGPS+ on the S2 with an external GPS device with the antenna beneath the sun roof (GPS signal reception is really lousy in that car), which however proved to be a bit much for the phone together with Mapillary and from Freiburg im Breisgau on we switched to using a normal bluetooth connected GPS device.
The set-up worked very well fillng a 32GB card for each direction, our longest continuous segment had something over 6200 images. Currently there are still roughly 8000 images from the drive back uploading (of a total of roughly 23'000), in any case we did cover a lot of new territory.
Routing with OsmAnd was quite painless too, and while it did take a couple of minutes to calculate the 850+ km routes that didn't really cause an issue. Very noticeable was the quite new support of destination, lanes and turn:lanes, lots of the intersections and on/off ramps were already completely tagged and only a handful had some errors. Matter of fact the only serious routing error we had along the way was due to a typo
Here exactly the same issue in OSRM:
which was caused by a short stretch of road with a wrong speed limit. OsmAnd does have an annoying tendency to try and short cut over off/on-ramps now and then, this may however be indicative of simply not penalizing these enough compared to continuing straight on. In summary the actual driving was uneventful and quite relaxing.
In the end, as we have already blogged about on sosm.ch, we picked up the servers (at 23:00 on Friday!) and had a quick and uneventful trip back on a different route on Saturday.
Special thanks to Wikimedia Germany and Switzerland and to everybody that helped getting this done. Michael is now working on getting the servers ready for deployment which we expect in a couple of weeks time.
Please consider our SOSM 2015 donation drive which should cover the initial costs we incurred acquiring the servers and any replacement and additional hardware we will need to purchase to get everything up and running.
I've updated the statistics that I regularly produce from our changeset dump, see stats page on wiki.openstreetmap.org, to the end of 2014 numbers.
The well known trends continue with growth in every category with a total of 472'925 contributors since 2005.
Looking back on the 10th anniversary celebrations in 2014 it is important to note that was celebrating the birth of the idea, not the start of success.
Success slowly started in 2007 with the project taking off in 2008, for a Web 2.0 undertaking prior to that it was a flop. Perhaps even better than a graph Martijn van Exel's "then and now" map comparision with 2007 illustrated just how much of nothing OSM was back then (unluckily the comparision is no longer available).
It is diffcult to nail down why the sudden change in course happened, likely it was more the pieces of a puzzle falling in place than just one cause. However if asked to point to a single person and event, I would clearly point to Richard Fairhurst and the Potlatch editor being the pivotal person and thing that changed the course of history for OSM.
Dear OSMF members
You have likely already seen the announcement for the upcoming general meeting on the 7th of December. I would ask you to make the small effort and either participate directly in the meeting on Sunday or, better, vote via proxy now.
As I pointed out in the original proposal for this general meeting, actually achieving the 75% votes to carry the special resolutions is a very high hurdle and unlikely to be achieved without support by the board. But even if we do not pass the 75% mark, a high as possible support for the term limits will send a clear signal to the board that we want the issues to be addressed.
It has been suggested that the main problem with the proposed term limits is that they actually affect some of the board members ability to continue to stand for re-election. Without us sending that clear signal to the board, it is very likely that the board will implement placebo limits that only serve to pacify their electorate and have no real effect.
Please make the small effort to participate in this vote and make your voice heard!
One of Steves favourite sound bites is “we have the money, lets employ somebody” in a couple of variants, used again in his current manifesto to convince people in to electing him back to the OSMF board.
During my time I simply ignored them, not wanting to derail things more than they typically were, but the obvious response would have been “how on earth would you know?” (well that is actually the polite version).
The OSMF has limited financial reporting (it has improved over the last two years) and essentially no planning, also known as a budget. It is currently simply not possible to know what the financial status of the OSMF is or should be at point in time in the future. And while the OSMF has roughly £80'000 of cash available we shouldn't be eating too much in to that given that substantial amounts of it would be needed in case of larger hardware problems (imagine one of our hosting locations catching fire) or other shortfalls and unexpected costs. Not even mentioning any kind of planning for short term cash needs.
It is one of the things that I consider a personal fail that we didn't manage to get something resembling a conventional budget in place over the last two years. It is not difficult, but it would have required cooperation from multiple board members to be actually meaningful and that was not forthcoming and so I concentrated on other pressing issues.
One of the larger planning issues is the largest line item, SOTM. Right now a couple of days before this years event, the OSMF board has no idea what has been signed in its name (if anything has been signed that is), what the potential risks are, nothing. And this was the same the previous year, the year before, the year before that and so on ….
Last year the board faced a potential GAU (German expression for the largest conceivable accident) when the local organizers for Birmingham threatened to walk out, we patched that up, but till this day we don't quite know how much we would have been in the red had the worst case happened. [I would like to point out that the local team was doing the right thing. I was simply showing that, yes, even "safe" things go wrong, and without planning you can't even do "what if"s in such a situation]
Now there is quite a lot of potential sources of funding available for OSM, but we are kidding ourselves if we think it is going to rain manna if we don't get our act together, get our financials under control and actually have projects worth funding (a separate topic).
To be clear, from a legal point of view there is nothing wrong with the OSMF finances and the way we have financed hardware purchases in the past has been low risk, leaving SOTM and our G&A as the larger variables. But it has trapped us in a vicious circle.
Rebooting (with an emphasis on boot) the OSMF board might be a way of breaking out, installing a Junta with the person mainly responsible for the current state of affairs in charge, not.
In 2011 the then current board had a face to face meeting in Seattle and produced a set of "audacious goals" for OSM that were
- The World’s Most Used Map
- More Than Just Streets
- Cultivating Leadership of Mappers
- Easier Contribution for Non-Geeks
Now some of these would have been directly actionable, but the first two are really "visions" under a different name. And as visions goes they are quite good, far out goal posts that nobody expects to be achieved soon.
And we are still far away from reaching the first goal . There are lots of pieces missing before we can remotely hope to achieve it, but we are on the way.
Steve has long harped about adding more addresses to OSM, matter of fact I have too. From a pure usability point of view if you want to produce a competitive device or app with door to door navigation support you are going to need address data. To be more precise this boils down to adding hose/building numbers with their associated streets (intentional pun) to our dataset in some form.
Note on the side: there are lots of regions that don't have a conventional western way of describing locations, we don't have good support for that yet in OSM, solving the “address problem” tends to centred around 1st world countries.
The OSM community has lots of experience with adding addresses by both on the ground surveying and imports of suitable open data, matter of fact we've had a full country “complete” for half a decade now. What is however undeniable is that it is slow progress, even importing a couple of million addresses (we don't have good numbers, but it is likely that there is something between half and one billion address conventional house addresses out there) takes a lot of time if you want to provide some minimum quality and the preferable on the ground surveying tends to be even slower.
But we are making progress and looking at one of the larger countries, Germany, with 81 million population, we are now at roughly 1/3 coverage with a combination of on the ground survey supported by open data and smaller imports, completion likely in 2 years at the current rate.
There are some places where we don't have a good handle on the issue, for example in the “original OSM country”, the UK, due to the addressing system revolving mainly around non-surveyable proprietary schemes and house numbers playing a secondary role. But we are not the only group feeling the pain there and I'm optimistic that we will find a way out, and if it is simply by replacing such proprietary systems.
To summarize: addresses are important and yes it is something that the community is working on with dedication. There is really nothing visionary about it at all at this point in time, not more than “lets map all roads in Germany”.
Steve has a legitimate commercial itch that he wants scratched and he wants that fast. This is not unique, readers following the “imports” mailing list have seen similar requests from HOT, matter of fact one of the HOT requests had that other problematic attribute “non-editable”.
Luckily HOT hasn't tried to stage a coup d’état to get their way. But what it does tell us is that the OSM data format, its tool chains, its editors and other applications have become extremely popular and people prefer our tools over others. It is a great testimonial to OSM.
OSM was built around the notion of mappers collecting or curating data and then adding it to our central repository, iteratively improving the quality and completeness over time. “fast” and “non-editable” are the antithesis to what normal OSM is about.
Now I'm sure ESRI doesn't get many complaints about them not providing the ultimate multi-100 GB all free geo-data of the world shape file that includes everything, so why do we? I can only guess that it is because people are feeling left out and think that if their favourite dataset is not included that they are not part of this great community.
But, really, it is just a superficial marketing and packaging issue.
If Steve or better his employer want to create the
containing a quickly thrown together conversion of all the open data address sets in the US, great, the OSMF might even be convinced to distribute it in the same place as the planet dumps. And we have more than enough id space to support conflict free merging.
There is simply no need to compromise the normal OSM community process for a short term gain.
We don't only have hammers, please stop just seeing nails.
Rewind two years, September 2012. I'm rather fuzzy on why I stood for the OSMF board in the first place, but I remember at least being annoyed by the treatment of the working groups by the board, experiencing first hand a dress down of one our most respected community members just because he didn't jump when the then chairperson said jump. And yes something didn't feel quite right about the organisation.
To make a long story short, I wasn't expecting to end up being elected the chairperson and further was expecting the fault lines inside the board to be in other places than they turned out to be. Naive me assumed that we would be having heated discussions about the scope of services the OSMF provides and how we delineate what the OSMF does from commercial and other services providers, how to grow the OSMF membership base and how we could best support the wider OSM community. Alas things were not quite ready for that.
My approach was let the past be the past, fix the issues and then move on. The net result was an uphill battle every foot of the way over the last two years and for the moving on part, well that might have happened now.
Why did I step down?
No, I didn't step down because of some flak on the mailing lists. I'm a big boy (well I am kind of small actually) and can easily take the heat. A consequence of exposing yourself and engaging with the community is that you tend to function as a focal point for everything real or just perceived wrong about the organisation you are representing, it simply comes with the territory. But I still think it is necessary out of respect to the OSMF members to be open about what I think and where I believe the sticky issues are, instead of spewing safe political correct canned sound bites or hiding away somewhere.
Steve Coast has decided to stage a come back after two years as chairman emeritus, as board member and chairperson that put me between a rock and a hard place. On the one hand I have to respect our democratic processes and not interfere with the ongoing election, on the other hand I do hold strong opinions on where we should be heading and in what fashion.
To make things clear: I consider it a very silly thing for Steve to do, bad for him, for the OSMF and for the whole project.
As the board member that has driven most of the change over the last two years, I see the danger of old alliances undoing the baby steps towards a more mature organisation we have achieved. An organisation that could actually engage in all the things the current board candidates envision. An organisation that holds a stable course and can function as a trustworthy intermediary between our fantastic community and the rest of the world.
I stepped down to make clear this is not about power, or my personal position, but about what is best for OSM.
Somebody who has been to a recent talk of mine on OSM might remember a slide representing OSMspace, showing some of our fairly complicated interactions and overlaps with the outside world, the OSMF a tiny speck somewhere off centre. Yes the OSMF is just a tiny piece of the puzzle with limited influence on the project as a whole. Most of our contributors will likely never know that it exists, no different than the Wikimedia Foundation. What is different is that OSM is a significant player in a multi-billion dollar market and the outside forces are of a very different nature.
While the OSMF is just that tiny speck in the OSM ocean it does hold the keys to our core product and with control over the foundation it is completely possible to misuse that. Yes, even handed, community control of the OSMF is important for everybody in OSMspace, and it scares me when somebody seriously proposes handing those keys to a Junta, even if it is “only for a year”.
I would like to end for now with thanking everybody that was kind enough to show me support over the last few days it is much appreciated.
More to come .....
A bit over 10 months ago the SOSM board decided that we would try and systematically send a welcome note to every new contributor in Switzerland http://sosm.ch/how-many-mappers-are-there-in-switzerland/. Today I just sent off the 1000th such message, not a big deal, takes a couple of seconds per mail and I typically send them every 2nd day or so. The mail (in 4 languages, really should be in five), tries to be as low key as possible and simply welcomes the new mapper and points out our local and international resources.
- there hasn't been a noticeable increase in SOSM membership signups, on the one hand 10 months is too short term to expect that to happen, and on the other hand we don't actively solicit that in the mail anyway.
- just as above direct feedback is quite rare.
- the "one edit user" seems to be on the way out, most of the users already have more than one changeset by the time we send the mail (nailing that down in hard numbers is a project for another day).
- given that Switzerland is rather small (8 million population), well mapped (as all DACH countries) you wouldn't be surprised to see some indications of saturation after 9 years, but there are no such effects noticeable at all.
If you think this is a good idea and are considering doing something similar yourself, please note:
- in most countries you will want to split the work up and coordinate, swamping new users with mail is likely not a good idea.
- Pascal Neis provides a RSS with new users on a country by country base here: http://resultmaps.neis-one.org/newestosmlist
- users signing up to OSM don't have to agree to receiving unsolicited mail, matter of fact there is currently no way that they can opt-in at all. This implies that you should consider the legal implications in whatever jurisdiction you are and be careful how you word your mails. I haven't heard of any problems, but you never know.
As a final note: most old timers know about or have participated in the reoccurring discussions on why people sign up for an OSM account and then never edit (there is currently no way of getting zero edits sign ups per region, so you can't contact those). I'm personally fairly sure that we have quite a good conversion rate, there are simply no comparable numbers from other projects available to prove that, others disagree. But as this example shows http://www.openstreetmap.org/user/user_4682 in the end we get them all :-).
PS: I should point out that by no way this is particularly original, we have had users doing this for a long time, it should be seen more as a reminder that this can be done.