OpenStreetMap logo OpenStreetMap

SimonPoole's Diary

Recent diary entries

Searching (in) OSM on a PI2

Posted by SimonPoole on 28 April 2015 in English.

Two days back there was a longish discussion on the OSM IRC channel about supporting OSM on XSCE, aka on offline, potentially slow devices. Currently it seems if they (XSCE) distribute a pre-rendered set of tiles, and during the discussion (which was mainly about alternatives to distributing tiles) it was mentioned that it would be nice if they, besides a slippy map, could provide a search function.

Now given that disk space idoes not seem to be an issue in the project and keeping in sync with OSM central is not a requirement, it occured to me that Photon might be a viable way of providing a global search function.

Installing it on my PI2 (running Ubuntu 14.04) was surprisingly easy and the only painful part was downloading 30GB of search index over my not particularly fast Internet connection.

Photon on a PI2

Now perfomance is not great, something like 7 to 10 seconds for a query, ruling out using “search as you type”, but bearable for individual queries. Potentially single language search indices would be faster, but that would needed to be tested.

Naturally alternative approaches for example using mapsforge might be better and would get around the requirement for prerendered tiles, however it is not clear if a global map would actually be feasible.

long time - no edit

Posted by SimonPoole on 28 April 2015 in English.

Anybody who has any interest in the growth of OpenStreetMap has probably read at least one paper or blog that has moaned about “only” a couple of 100’000 users actually having contributed out of the 2 million + that have signed up for an account.

The over 500’000 contributors are a good 25% of the total registered accounts and I’m not sure if that really counts as “only” given that we don’t really have any comparative numbers, I would suspect it is actually very good.

In any case there have been calls to simply delete the “inactive” accounts as they inflate the numbers and in general do no good. I’m very much against that for two reasons: on the one hand we don’t know why inactive members have joined, maybe they simply wanted to show support, maybe they wanted an OSM account for autentication in uMap or any of many other possible reasons. On the other hand they are a reservoir of new mappers, and every day we likely have dozens of old accounts starting to map.

As an holdover from the licence change I’m still running a script that produces a daily list of old accounts that have newly accepted the contributor terms and typically there are a dozen or so each day. These accounts have not been active since at least April 2011, when we had roughly 350’000 accounts total. The majority tends to be accounts that hadn’t previously edited but there are always a couple that somehow didn’t get the message during the licence change and had actually edited more than 4 years ago.

In any case the tl;dr version: deleting the 1.5 million inactive accounts would deny us the pleasure and fun of welcoming new contributors like https://www.openstreetmap.org/user/Geocurioius to the active mappers.

Addresses Revisited

Posted by SimonPoole on 11 April 2015 in English.

Two years ago I produced some statistics on addresses in OSM. While I did regularly re-run the scripts, I never really revisted the topic to see how things are developing,

Global Address Counts as of 2015-04-09

                2013-03-27     2015-04-09    Increase since March 2013
Total           20'168'470     51'903'310            257%
Germany          3'659'043      8'470'716            232%
USA              2'090'893      5'229'243            250%

The complete set of numbers can be found [here] (http://qa.poole.ch/addresses/), According to official statistics there are 18.5 million residential buildings in Germany so I would expect that we are roughly 1/3 to 1/2 of the way to complete coverage there.

The large global increase in early 2014 is mainly due to the import of 8.7 million addresses in the Netherlands which due to peculiarities in building and numbering in NL led to a far larger increase in the counts than you would normally expect of a country with a similar population size.

When I first produced the address stats the openaddresses.io project didn’t exist and it it is now interesting to compare the two projects. It should be noted at this point that openaddresses.io does not make any representations wrt licences of their individual datasets and there is no guarantee that all or any of the data could actually be used in OSM.

OA currently claims to contain 115 million address, slightly more than twice the 52 million in OSM. However how large is the overlap? A very rough (and unfair) estimate based on the numbers for the US, ES, NL, PL, DK, NO and JP in OSM, would indicate that 24 million address likely turn up in both projects, but, more interesting that OSM contains 28 million addresses not in OA. Or put differently a combined dataset would contain 143 million addresses (see caveat above).

A mad dash to Haarlem and back

Posted by SimonPoole on 18 March 2015 in English.

SOSM operates a fair number of services mainly for the Swiss community that up to now have been hosted on a single small server leased from Hetzner. While this is a very cost worthy solution, it doesn’t have any redundancy built in and further requires cross border Internet traffic to access, which raises privacy concerns at least with parts of our community. SInce we started operating the server a bit over a year ago it was clear that a local solution would be preferable.

Due to the good relations between SOSM and Wikimedia Switzerland late last year we were pointed to the fact that Wikimedia Germany was throwing away their old hardware which used to host the “toolserver” and that while most of it was clearly not worth continuing to use, there was three couple of years old Dell R520 servers that might still be useful. While we needed to find an affordable hosting location to go with the machines (more on that soon), it was clear, if at all possible, we would like to secure the machines for SOSM.

Wikimedia operates a larger site in the Netherlands in a commercial hosting facility just outside of Amsterdam in Haarlem (yes Haarlem and nearby Breuklen gave the names to the Harlem and Brooklyn in NYC). They don’t have direct on site staff and it was clear that we would have to arrange pickup when somebody was there dismantling the hardware. This turned out to be bit of a challenge and in the end, after determining that we wouldn’t have to commercially import the hardware (no financial difference, just a procedural one), we decided that the easiest and most efficient, perhaps not most ecological, solution would be simply to drive up to Amsterdam short term when Wikimedia was ready and pick up the servers in person.

Last Friday was the day, Michael (user datendelphin) and myself started on our mad dash to Haarlem and back, a total of over 1’700km. To at least get some more out of it we decided to record as much as possible of the drive with Mapillary and naturally navigate with OSM only.

An old S2 for Mapillary with two empty 32 GB and OsmAnd for navigation.

I initially tried to run RTKGPS+ on the S2 with an external GPS device with the antenna beneath the sun roof (GPS signal reception is really lousy in that car), which however proved to be a bit much for the phone together with Mapillary and from Freiburg im Breisgau on we switched to using a normal bluetooth connected GPS device.

The set-up worked very well fillng a 32GB card for each direction, our longest continuous segment had something over 6200 images. Currently there are still roughly 8000 images from the drive back uploading (of a total of roughly 23’000), in any case we did cover a lot of new territory.

Routing with OsmAnd was quite painless too, and while it did take a couple of minutes to calculate the 850+ km routes that didn’t really cause an issue. Very noticeable was the quite new support of destination, lanes and turn:lanes, lots of the intersections and on/off ramps were already completely tagged and only a handful had some errors. Matter of fact the only serious routing error we had along the way was due to a typo

Here exactly the same issue in OSRM:

which was caused by a short stretch of road with a wrong speed limit. OsmAnd does have an annoying tendency to try and short cut over off/on-ramps now and then, this may however be indicative of simply not penalizing these enough compared to continuing straight on. In summary the actual driving was uneventful and quite relaxing.

In the end, as we have already blogged about on sosm.ch, we picked up the servers (at 23:00 on Friday!) and had a quick and uneventful trip back on a different route on Saturday.

Special thanks to Wikimedia Germany and Switzerland and to everybody that helped getting this done. Michael is now working on getting the servers ready for deployment which we expect in a couple of weeks time.

Please consider our SOSM 2015 donation drive which should cover the initial costs we incurred acquiring the servers and any replacement and additional hardware we will need to purchase to get everything up and running.

Contributor Statistics 2014

Posted by SimonPoole on 9 January 2015 in English. Last updated on 10 January 2015.

I’ve updated the statistics that I regularly produce from our changeset dump, see stats page on wiki.openstreetmap.org, to the end of 2014 numbers.

The well known trends continue with growth in every category with a total of 472’925 contributors since 2005.

Looking back on the 10th anniversary celebrations in 2014 it is important to note that was celebrating the birth of the idea, not the start of success.

Success slowly started in 2007 with the project taking off in 2008, for a Web 2.0 undertaking prior to that it was a flop. Perhaps even better than a graph Martijn van Exel’s “then and now” map comparision with 2007 illustrated just how much of nothing OSM was back then (unluckily the comparision is no longer available).

It is diffcult to nail down why the sudden change in course happened, likely it was more the pieces of a puzzle falling in place than just one cause. However if asked to point to a single person and event, I would clearly point to Richard Fairhurst and the Potlatch editor being the pivotal person and thing that changed the course of history for OSM.

Dear OSMF members

You have likely already seen the announcement for the upcoming general meeting on the 7th of December. I would ask you to make the small effort and either participate directly in the meeting on Sunday or, better, vote via proxy now.

As I pointed out in the original proposal for this general meeting, actually achieving the 75% votes to carry the special resolutions is a very high hurdle and unlikely to be achieved without support by the board. But even if we do not pass the 75% mark, a high as possible support for the term limits will send a clear signal to the board that we want the issues to be addressed.

It has been suggested that the main problem with the proposed term limits is that they actually affect some of the board members ability to continue to stand for re-election. Without us sending that clear signal to the board, it is very likely that the board will implement placebo limits that only serve to pacify their electorate and have no real effect.

Please make the small effort to participate in this vote and make your voice heard!

Simon

Money!

Posted by SimonPoole on 27 October 2014 in English. Last updated on 29 October 2014.

One of Steves favourite sound bites is “we have the money, lets employ somebody” in a couple of variants, used again in his current manifesto to convince people in to electing him back to the OSMF board.

During my time I simply ignored them, not wanting to derail things more than they typically were, but the obvious response would have been “how on earth would you know?” (well that is actually the polite version).

The OSMF has limited financial reporting (it has improved over the last two years) and essentially no planning, also known as a budget. It is currently simply not possible to know what the financial status of the OSMF is or should be at point in time in the future. And while the OSMF has roughly £80’000 of cash available we shouldn’t be eating too much in to that given that substantial amounts of it would be needed in case of larger hardware problems (imagine one of our hosting locations catching fire) or other shortfalls and unexpected costs. Not even mentioning any kind of planning for short term cash needs.

It is one of the things that I consider a personal fail that we didn’t manage to get something resembling a conventional budget in place over the last two years. It is not difficult, but it would have required cooperation from multiple board members to be actually meaningful and that was not forthcoming and so I concentrated on other pressing issues.

One of the larger planning issues is the largest line item, SOTM. Right now a couple of days before this years event, the OSMF board has no idea what has been signed in its name (if anything has been signed that is), what the potential risks are, nothing. And this was the same the previous year, the year before, the year before that and so on ….

Last year the board faced a potential GAU (German expression for the largest conceivable accident) when the local organizers for Birmingham threatened to walk out, we patched that up, but till this day we don’t quite know how much we would have been in the red had the worst case happened. [I would like to point out that the local team was doing the right thing. I was simply showing that, yes, even “safe” things go wrong, and without planning you can’t even do “what if”s in such a situation]

Now there is quite a lot of potential sources of funding available for OSM, but we are kidding ourselves if we think it is going to rain manna if we don’t get our act together, get our financials under control and actually have projects worth funding (a separate topic).

To be clear, from a legal point of view there is nothing wrong with the OSMF finances and the way we have financed hardware purchases in the past has been low risk, leaving SOTM and our G&A as the larger variables. But it has trapped us in a vicious circle.

Rebooting (with an emphasis on boot) the OSMF board might be a way of breaking out, installing a Junta with the person mainly responsible for the current state of affairs in charge, not.

Vision?

Posted by SimonPoole on 27 October 2014 in English.

In 2011 the then current board had a face to face meeting in Seattle and produced a set of “audacious goals” for OSM that were

  • The World’s Most Used Map
  • More Than Just Streets
  • Cultivating Leadership of Mappers
  • Easier Contribution for Non-Geeks

Now some of these would have been directly actionable, but the first two are really “visions” under a different name. And as visions goes they are quite good, far out goal posts that nobody expects to be achieved soon.

And we are still far away from reaching the first goal . There are lots of pieces missing before we can remotely hope to achieve it, but we are on the way.

Steve has long harped about adding more addresses to OSM, matter of fact I have too. From a pure usability point of view if you want to produce a competitive device or app with door to door navigation support you are going to need address data. To be more precise this boils down to adding hose/building numbers with their associated streets (intentional pun) to our dataset in some form.

Note on the side: there are lots of regions that don’t have a conventional western way of describing locations, we don’t have good support for that yet in OSM, solving the “address problem” tends to centred around 1st world countries.

The OSM community has lots of experience with adding addresses by both on the ground surveying and imports of suitable open data, matter of fact we’ve had a full country “complete” for half a decade now. What is however undeniable is that it is slow progress, even importing a couple of million addresses (we don’t have good numbers, but it is likely that there is something between half and one billion address conventional house addresses out there) takes a lot of time if you want to provide some minimum quality and the preferable on the ground surveying tends to be even slower.

But we are making progress and looking at one of the larger countries, Germany, with 81 million population, we are now at roughly 1/3 coverage with a combination of on the ground survey supported by open data and smaller imports, completion likely in 2 years at the current rate.

There are some places where we don’t have a good handle on the issue, for example in the “original OSM country”, the UK, due to the addressing system revolving mainly around non-surveyable proprietary schemes and house numbers playing a secondary role. But we are not the only group feeling the pain there and I’m optimistic that we will find a way out, and if it is simply by replacing such proprietary systems.

To summarize: addresses are important and yes it is something that the community is working on with dedication. There is really nothing visionary about it at all at this point in time, not more than “lets map all roads in Germany”.

Hammers

Steve has a legitimate commercial itch that he wants scratched and he wants that fast. This is not unique, readers following the “imports” mailing list have seen similar requests from HOT, matter of fact one of the HOT requests had that other problematic attribute “non-editable”.

Luckily HOT hasn’t tried to stage a coup d’état to get their way. But what it does tell us is that the OSM data format, its tool chains, its editors and other applications have become extremely popular and people prefer our tools over others. It is a great testimonial to OSM.

OSM was built around the notion of mappers collecting or curating data and then adding it to our central repository, iteratively improving the quality and completeness over time. “fast” and “non-editable” are the antithesis to what normal OSM is about.

Now I’m sure ESRI doesn’t get many complaints about them not providing the ultimate multi-100 GB all free geo-data of the world shape file that includes everything, so why do we? I can only guess that it is because people are feeling left out and think that if their favourite dataset is not included that they are not part of this great community.

But, really, it is just a superficial marketing and packaging issue.

If Steve or better his employer want to create the

ultimate_free_address_superpack_US_edition.osm

containing a quickly thrown together conversion of all the open data address sets in the US, great, the OSMF might even be convinced to distribute it in the same place as the planet dumps. And we have more than enough id space to support conflict free merging.

There is simply no need to compromise the normal OSM community process for a short term gain.

We don’t only have hammers, please stop just seeing nails.

Thank you for your trust!

Posted by SimonPoole on 26 October 2014 in English.

Rewind two years, September 2012. I’m rather fuzzy on why I stood for the OSMF board in the first place, but I remember at least being annoyed by the treatment of the working groups by the board, experiencing first hand a dress down of one our most respected community members just because he didn’t jump when the then chairperson said jump. And yes something didn’t feel quite right about the organisation.

To make a long story short, I wasn’t expecting to end up being elected the chairperson and further was expecting the fault lines inside the board to be in other places than they turned out to be. Naive me assumed that we would be having heated discussions about the scope of services the OSMF provides and how we delineate what the OSMF does from commercial and other services providers, how to grow the OSMF membership base and how we could best support the wider OSM community. Alas things were not quite ready for that.

My approach was let the past be the past, fix the issues and then move on. The net result was an uphill battle every foot of the way over the last two years and for the moving on part, well that might have happened now.

Why did I step down?

No, I didn’t step down because of some flak on the mailing lists. I’m a big boy (well I am kind of small actually) and can easily take the heat. A consequence of exposing yourself and engaging with the community is that you tend to function as a focal point for everything real or just perceived wrong about the organisation you are representing, it simply comes with the territory. But I still think it is necessary out of respect to the OSMF members to be open about what I think and where I believe the sticky issues are, instead of spewing safe political correct canned sound bites or hiding away somewhere.

Steve Coast has decided to stage a come back after two years as chairman emeritus, as board member and chairperson that put me between a rock and a hard place. On the one hand I have to respect our democratic processes and not interfere with the ongoing election, on the other hand I do hold strong opinions on where we should be heading and in what fashion.

To make things clear: I consider it a very silly thing for Steve to do, bad for him, for the OSMF and for the whole project.

As the board member that has driven most of the change over the last two years, I see the danger of old alliances undoing the baby steps towards a more mature organisation we have achieved. An organisation that could actually engage in all the things the current board candidates envision. An organisation that holds a stable course and can function as a trustworthy intermediary between our fantastic community and the rest of the world.

I stepped down to make clear this is not about power, or my personal position, but about what is best for OSM.

Somebody who has been to a recent talk of mine on OSM might remember a slide representing OSMspace, showing some of our fairly complicated interactions and overlaps with the outside world, the OSMF a tiny speck somewhere off centre. Yes the OSMF is just a tiny piece of the puzzle with limited influence on the project as a whole. Most of our contributors will likely never know that it exists, no different than the Wikimedia Foundation. What is different is that OSM is a significant player in a multi-billion dollar market and the outside forces are of a very different nature.

While the OSMF is just that tiny speck in the OSM ocean it does hold the keys to our core product and with control over the foundation it is completely possible to misuse that. Yes, even handed, community control of the OSMF is important for everybody in OSMspace, and it scares me when somebody seriously proposes handing those keys to a Junta, even if it is “only for a year”.

I would like to end for now with thanking everybody that was kind enough to show me support over the last few days it is much appreciated.

More to come …..

1000

Posted by SimonPoole on 25 October 2014 in English.

A bit over 10 months ago the SOSM board decided that we would try and systematically send a welcome note to every new contributor in Switzerland http://sosm.ch/how-many-mappers-are-there-in-switzerland/. Today I just sent off the 1000th such message, not a big deal, takes a couple of seconds per mail and I typically send them every 2nd day or so. The mail (in 4 languages, really should be in five), tries to be as low key as possible and simply welcomes the new mapper and points out our local and international resources.

Observations

  • there hasn’t been a noticeable increase in SOSM membership signups, on the one hand 10 months is too short term to expect that to happen, and on the other hand we don’t actively solicit that in the mail anyway.
  • just as above direct feedback is quite rare.
  • the “one edit user” seems to be on the way out, most of the users already have more than one changeset by the time we send the mail (nailing that down in hard numbers is a project for another day).
  • given that Switzerland is rather small (8 million population), well mapped (as all DACH countries) you wouldn’t be surprised to see some indications of saturation after 9 years, but there are no such effects noticeable at all.

If you think this is a good idea and are considering doing something similar yourself, please note:

  • in most countries you will want to split the work up and coordinate, swamping new users with mail is likely not a good idea.
  • Pascal Neis provides a RSS with new users on a country by country base here: http://resultmaps.neis-one.org/newestosmlist
  • users signing up to OSM don’t have to agree to receiving unsolicited mail, matter of fact there is currently no way that they can opt-in at all. This implies that you should consider the legal implications in whatever jurisdiction you are and be careful how you word your mails. I haven’t heard of any problems, but you never know.

As a final note: most old timers know about or have participated in the reoccurring discussions on why people sign up for an OSM account and then never edit (there is currently no way of getting zero edits sign ups per region, so you can’t contact those). I’m personally fairly sure that we have quite a good conversion rate, there are simply no comparable numbers from other projects available to prove that, others disagree. But as this example shows http://www.openstreetmap.org/user/user_4682 in the end we get them all :-).

PS: I should point out that by no way this is particularly original, we have had users doing this for a long time, it should be seen more as a reminder that this can be done.

Contributor numbers Q3 2014

Posted by SimonPoole on 12 October 2014 in English. Last updated on 13 October 2014.

I’ve updated the statistics that I regularly produce from our changeset dump, see stats page on wiki.openstreetmap.org, to the end of Q3 2014 numbers.

The well known trends continue with continuous growth in every category with a total of 447’883 contributors. Of special note is

showing that at the end of the third quarter we have already had slightly more active contributors than in the full year 2013.

August 2014 showed the 2nd highest increase in contributors in the history of the project with over 10’700 new mappers joining OpenStreetMap. It is not unwarranted speculation that this was due to the tenth anniversary celebrations.

Upcoming Vespucci 0.9.5 release

Posted by SimonPoole on 1 August 2014 in English. Last updated on 2 August 2014.

Vespucci receiving GPS data from RTKLIB

Vespucci 0.9.5 is now very near the feature freeze for the 0.9.5 release, numerous tasks that have either been been requested by users for a long time or have been on my personal to do list have been completed and there are only one or two left to do.

Some of the more interesting new functionality

  • On device help and some usability improvements
  • JOSM compatible OSM file reading and saving
  • Auto download
  • Fast address tags adding with house number prediction
  • Import and upload of GPS tracks
  • Function to add node at current GPS position
  • Support for external GPS sources (for example RTKLIB)
  • Basic conflict resolution
  • and numerous “under the hood” changes

If you want to give the beta version a spin, it can be downloaded from

https://drive.google.com/folderview?id=0B9pKLmh8s1h8bFI5bGd4VnhYWkk&usp=sharing

Some of you may have seen my blog posts on contributor numbers in the past (May 2013 , December 2013) on our contributor growth based on analysing the regular changeset dumps.

While I was fairly sure that the numbers were correct, I did know that I hadn’t taken one potential systematic problem in to account: users that had signed up and produced only empty changesets. Now it could be argued that such users at least tried to contribute and should be counted, but opinions may differ on that. It was clear that any effect on current trends would be minimal, all modern editors will typically only allow you to save if you have actually changed something, implying that an empty changeset is something that can only happen in an error situation (for example an editing conflict, or a crash of the application).

Here are the corrected graphs:

The effect of removing the empty changesets lowers the overall number by roughly 25’000 users, however as expected does not effect the general trend. End of June 2014 we had an accumulated total of 421’701 contributors, increasing at a rate of around 8’000 per month.

The following graphs compared to the previous ones show that most of the effect of the change is in 2009, which as we know is the year changesets were introduced in, so some issues there are not quite unexpected.

In mid-2014 we have already had more than 90’000 active contributors, we can expect a substantial increase in the year total if the trend continues.

Back to the empty changesets, when and how much of an effect did removing them from the numbers have:

and the same relative to the new contributors per month:

(note: the above is the difference between the old numbers with zero edit changesets and the new ones, in some cases this simply caused users to be counted one month later, which explains the single negative month)

As already mentioned all changesets before May 2009 were generated when changests were introduced. I haven’t been able to determine what caused the large numbers from May to November 2009, however the changesets in question do not have a created_by tag and I suspect that they were created after the fact, by some kind of mechanical process. Nobody that I asked can remember, maybe a reader can shed some light on this.

After November 2009 most of the empty changesets were created by Potlatch 1 and a well founded suspicion is that they were caused by P1 live mode. This continues up to April 2011 when Potlatch 2 was made the default editor and the absolute numbers have remained stable in the 100-200 users per month range since then.

Now it is important to remind ourselves what we are looking at: the numbers are new users that signed up, tried to edit, failed and never had a successful edit after that.

In other words, users that wanted to participate that we lost (the total number of empty changesets is far higher, however for now I’ll assume that regular contributors are more tolerant of things going wrong than first-timers).

Done deeds are done deeds and there is nothing we can do about users that we have already lost, however what we can do is try and improve the situation going forward.

I’ve produced some numbers on which editors the users were using and nearly all (yes including JOSM) turn up, however the majority of the users effected are using iD (which is not a surprise given that it is the default editor) and, big surprise, “Go Map!” at nearly the same level. Given that “Go Map!” has a far lower user base than iD (editor stats), this indicates that there may be a real problem with the app this has in the mean time been resolved. Naturally given that iD is what is usually used by potential new contributors, further investigation is warranted too.

Crazy TIGER

Posted by SimonPoole on 5 May 2014 in English.

Anybody that has spent any amount of time fixing TIGER data in the US has seen all the artifacts that we have all come to love, over- and underruns in corners, lost fixes after being inside, wiggles where the surveyors stopped to read road signs, crosscountry non-existing residenials and so on. Today I saw one thing I hadn’t before:

(blue is the corrected version)

[As always this is just my own, personal, opinion, and is no way an official statement by anybody]

Yesterday I had a short exchange of tweets with somebody that was surprised that http://opendata-hackday.de/ was using google maps instead of OSM. Given that it is rather a convoluted subject, explaining why this in fact is not surprising was a bit difficult in 140 letters and is what prompted me to create this post.

It is probably just natural that outsiders, even members of both the OSM community and the Open Data movement, simply assume that these are essentially the same and have a large overlap in motives and goals. Numerous OSM contributors are active in the Open Data movement and undoubtedly we are a very large consumer of open data in various forms.

However this apparent overlap shouldn’t hide the fact that both our goals and motives are in large parts completely different. The Open Data movement is about liberating, accessing and exploiting data that is already there, and, please don’t take this negative, about improving the bottom line of the companies involved. One of the major arguments used in prying data out of the hands of government is that it will have a beneficial net effect for our economies and the involved companies and I don’t have an argument with that. On top of that, the “we have already paid for it” justification is surely, at least in some ways, correct.

The OpenStreetMap project is very different, it is all about producing free and open geo-data and while we do utilize open sources, we are clearly at our best when the data has been surveyed and curated by mappers on the ground. Our goal is, in the end, to produce the best “map” of the world that is at the same time freely usable and re-reusable. Yes, some of the economic arguments apply just as well to the OSM ecosystem as they do to the same in the Open Data movement. I think we should be all be proud of the enlightened view the OSM community has had on commercial re-use of our data from the beginning.

But it has to be very clear: the OSM community created and owns the OSM data, and we control the terms on which it can be used, we have not been “already paid for it”.

Hopping off the soap box, I believe it is now understandable why complete alignment of goals cannot be expected. For “joe open data” google is a just as valid member of the open data community as OSM. google may even fit the open data model better: consumes and produces non-open products from it. We are just the crazies that spend our own time producing something free.

FIX THE MAP

Posted by SimonPoole on 1 May 2014 in English.

It seems, even though it has been available for quite a while, that the fixthemap page on openstreetmap.org is still not particularly well known. If you have an OSM based project and you don’t want to create your own page for this purpose (as for example Mapbox has done now too) please provide a link to the page with a suitable text. You can provide coordinates with the URL (best probably the center of the map the user is viewing) and if the user chooses to add a note it will be positioned correctly.

An example of how to do this is for example the SOSM run osm.ch site: http://www.osm.ch/#13/47.2095/8.5237

Vespucci Release 0.9.4 Highlights

This release contains a lot of “under the hood” improvements and some work on making the UI more consistent and easier to use. In particular the following changes have been made

  • selectable overlay layer.
  • support for multiple simultaneous presets.
  • added find action to lookup location with nominatim.
  • add per zoom level imagery offsets with support for querying and saving to the imagery offset database or manual entry.
  • added support for name suggestions and auto preset setting.
  • added goto current GPS location.
  • added action to arrange nodes of a closed way in a circle.
  • limited support for geo: URIs and JOSM style remote control.
  • add action to directly set position of node by entering coordinates.
  • major rework of imagery provider configuration, now based on https://github.com/osmlab/editor-imagery-index .
  • make https API default.
  • major refactoring of projection code.
  • lots of bug fixes and stability improvements.

The full change log is available here http://code.google.com/p/osmeditor4android/source/list

We will be updating the documentation to include the new features as soon as possible.

Upgrading from previous Versions

There are a few points that you may want to consider when upgrading from previous versions of Vespucci:

  1. some of the new features may cause degraded performance on older phones, see http://code.google.com/p/osmeditor4android/wiki/FAQ#Running_Vespucci_on_%22old_and_small%22_devices
  2. 0.9.4 uses a new configuration file format for imagery as mentioned above. As a consequence the internal identifiers for the background tile providers have changed with the exception of the “standard” mapnik tiles and Bing imagery. If you have extensively used any other sources you will be left with unused tiles using space on your device. You can delete unneeded directories and tiles by navigating to the andnav2/tiles directory on your system (where exactly the directory is located depends on the device and Android version, but it will be in the same place as the vespucci directory) and simply deleting the directories with the exception of BING and MAPNIK.
  3. some of the defaults for preferences have changed in 0.9.4, your old values will continue to remain the same, however the new defaults seem to make much more sense. See http://code.google.com/p/osmeditor4android/wiki/Tutorial

When does share alike kick in?

Posted by SimonPoole on 13 March 2014 in English.

Last weekend I had a short discussion with a well-respected OSM community member on some aspects of the ODbL and it ended more or less on a question, “then when does share alike kick in?” Given that it was 2am my answer wasn’t particularly good and so I thought I should expand it a bit in writing. Particularly because I may have given the impression that it is a fairly complex matter, when in reality it is fairly simple.

Disclaimer: this is the personal opinion of a non-lawyer and it is neither an official policy statement by the LWG nor the OSMF. There are a handful of grey areas that I will not touch on, on some of them the LWG is preparing clarifications for discussion that will be available soon, in other words I am staying on safe ground.

Further it is well known that I’m not particularly in love with the ODbL, but on the other hand I do think it is a lot better than it is made out to be.

The ODbL has 3 concepts that are relevant to triggering share alike (verbatim quotes from the ODbL text):

  • “Derivative Database” - – Means a database based upon the Database, and includes any translation, adaptation, arrangement, modification, or any other alteration of the Database or of a Substantial part of the Contents. This includes, but is not limited to, Extracting or Re-utilising the whole or a Substantial part of the Contents in a new Database.

  • “Collective Database” - Means this Database in unmodified form as part of a collection of independent databases in themselves that together are assembled into a collective whole. A work that constitutes a Collective Database will not be considered a Derivative Database.

  • ““Publicly” – means to Persons other than You or under Your control by either more than 50% ownership or by the power to direct their activities (such as contracting with an independent consultant).

Starting with the last concept, share alike only kicks in when you “Publicly Use” a derivative database see (ODbL 1.0: 4.4(a) and 4.5(c)) , in house use, use by a contractor on your behalf and similar all do not trigger share alike and are not of interest. For the rest of this discussion please assume that whatever we are discussing, we are discussing it in the context of publicly using whatever you have created.

You are now probably already jumping up and down and shouting “And what about Produced Works?”. Produced Works are only relevant to share alike in that if you “Publicly Use” a Produced Work (ODbL 1.0: 4.4(c)) any derivative database that was used in producing the Produce Work is considered “Publicly Used”. Given that we already are assuming that, we do not need to consider Produced Works at all for the purpose of this discussion. Seems as if we have already considerably simplified the matter at hand.

If you read the ODbL *Derivative Databases” is what in the end share alike is attached to, original OSM data, extracts and modifications to such are all datasets that are, no surprise, subject to mandatory ODbL licensing. But what happens if you are using other data together with OSM derived datasets? Going back to the definitions, we see that such use creates a Collective Database.

How does share alike apply to a Collective Database? Well according to 4.5(a) “For the avoidance of doubt, You are not required to license Collective Databases under this License if You incorporate this Database or a Derivative Database in the collection, but this License still applies to this Database or a Derivative Database as a part of the Collective Database;”.

In other words if you simply lump together one or more datasets with data derived from OSM, you are only required to licence the OSM part of the Collective Database under the ODbL or a compatible licence.

Example: assume that you have a proprietary global database of waste bins and want to use that data together with OSM data. No problem, you can use your data together with OSM without any issue and there is no need to publish your proprietary dataset on ODbL terms.

Grey area alert: while the example is clear, there are some kinds of “lumping together” that need clarification.

Now given that OSM has a lot of waste bins already, the result might contain a lot of duplicates that you would like to remove. Again no problem, you can simply remove all waste bins from the OSM dataset. Now the resulting OSM data is clearly a Derivative Database and is subject to the share alike terms in the ODbL (as it was before), but it does not change the status of the collective whole which can still have different licences for its individual parts and the whole.

Grey area alert: this kind of Derivative Database (reduced and extracted unmodified OSM data) triggers a number of obligations that essentially nobody is adhering to.

This is the point I was in discussion at 2am and when the question “then when does share alike kick in? “ was posed.

Well the answer is: “when you modify OSM data”. The simplest example: you improve the position of a POI by changing the coordinates or you add further information to the POI, then you have to make the resulting dataset available on ODbL terms. Don’t forget we are always assuming that you are Publicly Using the data.

A more interesting example: assume you have a proprietary database containing road geometry and associated with that geometry, road surface information and further that you have permission to integrate the surface information into OSM. You add surface tags to the OSM roads in your copy of the OSM data: yes you have to publish the improved OSM data on ODbL terms.

The important thing to note is that it does not effect your original proprietary database, there is no infection or tainting of that dataset, you simply cannot keep the changes to the OSM data to yourself.

And what about the other way around? Assume you notice that OSM has some surface data that is better than that in your proprietary database and you replace the original information with that? Then the resulting dataset is subject to share alike and you need to make it available on ODbL terms.

To sum it up: When does share alike kick in? When you modify OSM data or apply modifications from OSM to third party data and use the results publicly.

That’s it really.

Name suggestion support in editors

Posted by SimonPoole on 7 March 2014 in English.

The iD editor has used the name suggestion index for quite a while and it was something that I wanted to support in vespucci in the upcoming release. The one thing you do want to reduce in a mobile app is typing.

Basically the idea is to suggest correct spelling (and tagging) for some of the more prominent chains of restaurants and shops to the mapper. The way it works in vespucci now is slightly different from the current implementation in iD and at least some aspects might be worthwhile supporting there too.

Using the name as a short cut for tagging

If you are creating a new object, or adding tags to an object that previously didn’t have relevant tags you can use the name as an “Ersatz”-preset (note vespucci supports normal JOSM-style presets too, and I may expand this functionality to automatically add further tags from the presets):

Enter the name tag

### Start typing and get the auto-complete list ### Further typing refines the list ### Selecting the correct item set the name and the corresponding tags

Adding the name to a “pre-tagged” object

In the example the building has already been pre-tagged with amenity=restaurant, which while not really wrong is not the best tagging for a fast food restaurant.

Enter the name tag

Start typing and get the auto-complete list

This includes suggestions not just for amenity=restaurant but for potenially further refined tagging.

Select the correct name after further refinement

Vespucci now asks for confirmation if the old tags should be replaced.

The new functionality is available in the vespucci test builds.