Recent diary entries
Last weekend I had a short discussion with a well-respected OSM community member on some aspects of the ODbL and it ended more or less on a question, "then when does share alike kick in?" Given that it was 2am my answer wasn't particularly good and so I thought I should expand it a bit in writing. Particularly because I may have given the impression that it is a fairly complex matter, when in reality it is fairly simple.
Disclaimer: this is the personal opinion of a non-lawyer and it is neither an official policy statement by the LWG nor the OSMF. There are a handful of grey areas that I will not touch on, on some of them the LWG is preparing clarifications for discussion that will be available soon, in other words I am staying on safe ground.
Further it is well known that I'm not particularly in love with the ODbL, but on the other hand I do think it is a lot better than it is made out to be.
The ODbL has 3 concepts that are relevant to triggering share alike (verbatim quotes from the ODbL text):
"Derivative Database" - – Means a database based upon the Database, and includes any translation, adaptation, arrangement, modification, or any other alteration of the Database or of a Substantial part of the Contents. This includes, but is not limited to, Extracting or Re-utilising the whole or a Substantial part of the Contents in a new Database.
"Collective Database" - Means this Database in unmodified form as part of a collection of independent databases in themselves that together are assembled into a collective whole. A work that constitutes a Collective Database will not be considered a Derivative Database.
"“Publicly” – means to Persons other than You or under Your control by either more than 50% ownership or by the power to direct their activities (such as contracting with an independent consultant).
Starting with the last concept, share alike only kicks in when you "Publicly Use" a derivative database see (ODbL 1.0: 4.4(a) and 4.5(c)) , in house use, use by a contractor on your behalf and similar all do not trigger share alike and are not of interest. For the rest of this discussion please assume that whatever we are discussing, we are discussing it in the context of publicly using whatever you have created.
You are now probably already jumping up and down and shouting "And what about Produced Works?". Produced Works are only relevant to share alike in that if you "Publicly Use" a Produced Work (ODbL 1.0: 4.4(c)) any derivative database that was used in producing the Produce Work is considered "Publicly Used". Given that we already are assuming that, we do not need to consider Produced Works at all for the purpose of this discussion. Seems as if we have already considerably simplified the matter at hand.
If you read the ODbL *Derivative Databases" is what in the end share alike is attached to, original OSM data, extracts and modifications to such are all datasets that are, no surprise, subject to mandatory ODbL licensing. But what happens if you are using other data together with OSM derived datasets? Going back to the definitions, we see that such use creates a Collective Database.
How does share alike apply to a Collective Database? Well according to 4.5(a) "For the avoidance of doubt, You are not required to license Collective Databases under this License if You incorporate this Database or a Derivative Database in the collection, but this License still applies to this Database or a Derivative Database as a part of the Collective Database;".
In other words if you simply lump together one or more datasets with data derived from OSM, you are only required to licence the OSM part of the Collective Database under the ODbL or a compatible licence.
Example: assume that you have a proprietary global database of waste bins and want to use that data together with OSM data. No problem, you can use your data together with OSM without any issue and there is no need to publish your proprietary dataset on ODbL terms.
Grey area alert: while the example is clear, there are some kinds of "lumping together" that need clarification.
Now given that OSM has a lot of waste bins already, the result might contain a lot of duplicates that you would like to remove. Again no problem, you can simply remove all waste bins from the OSM dataset. Now the resulting OSM data is clearly a Derivative Database and is subject to the share alike terms in the ODbL (as it was before), but it does not change the status of the collective whole which can still have different licences for its individual parts and the whole.
Grey area alert: this kind of Derivative Database (reduced and extracted unmodified OSM data) triggers a number of obligations that essentially nobody is adhering to.
This is the point I was in discussion at 2am and when the question "then when does share alike kick in? " was posed.
Well the answer is: "when you modify OSM data". The simplest example: you improve the position of a POI by changing the coordinates or you add further information to the POI, then you have to make the resulting dataset available on ODbL terms. Don't forget we are always assuming that you are Publicly Using the data.
A more interesting example: assume you have a proprietary database containing road geometry and associated with that geometry, road surface information and further that you have permission to integrate the surface information into OSM. You add surface tags to the OSM roads in your copy of the OSM data: yes you have to publish the improved OSM data on ODbL terms.
The important thing to note is that it does not effect your original proprietary database, there is no infection or tainting of that dataset, you simply cannot keep the changes to the OSM data to yourself.
And what about the other way around? Assume you notice that OSM has some surface data that is better than that in your proprietary database and you replace the original information with that? Then the resulting dataset is subject to share alike and you need to make it available on ODbL terms.
To sum it up: When does share alike kick in? When you modify OSM data or apply modifications from OSM to third party data and use the results publicly.
That's it really.
We will be meeting in Zürich tomorrow for the fifthiest Zürich OSM meetup, if you have time please feel free to join us.
The iD editor has used the name suggestion index for quite a while and it was something that I wanted to support in vespucci in the upcoming release. The one thing you do want to reduce in a mobile app is typing.
Basically the idea is to suggest correct spelling (and tagging) for some of the more prominent chains of restaurants and shops to the mapper. The way it works in vespucci now is slightly different from the current implementation in iD and at least some aspects might be worthwhile supporting there too.
Using the name as a short cut for tagging
If you are creating a new object, or adding tags to an object that previously didn't have relevant tags you can use the name as an "Ersatz"-preset (note vespucci supports normal JOSM-style presets too, and I may expand this functionality to automatically add further tags from the presets):
Enter the name tag
Start typing and get the auto-complete list
Further typing refines the list
Selecting the correct item set the name and the corresponding tags
Adding the name to a "pre-tagged" object
In the example the building has already been pre-tagged with amenity=restaurant, which while not really wrong is not the best tagging for a fast food restaurant.
Enter the name tag
Start typing and get the auto-complete list
This includes suggestions not just for amenity=restaurant but for potenially further refined tagging.
Select the correct name after further refinement
Vespucci now asks for confirmation if the old tags should be replaced.
The new functionality is available in the vespucci test builds.
Over the last couple of weeks I've done a substantial amount of work on vespucci the OSM editor for Android http://code.google.com/p/osmeditor4android/, with the goal of a new release 0.9.4 some time in March. While a lot of the work is under the hood, there are a couple of new features that will make editing with vespucci a lot easier
- support for overlays (for example the OSM GPS tracks)
- imagery configuration from OSM imagery index hopefully making it easier to keep the available imagery updated
- support for imagery offsets
- multiple simultaneously active presets (vespucci uses JOSM presets)
- support for accessing the API with https and OAuth
and not to forget, you can now zoom out completly without getting sea sick :-).
One of the issues the OSM site has had, is that we have been missing a trivial "Report a problem" / "Fix the map" landing page on openstreetmap.org. Not that it was in any way difficult to do, we just had never added one. I suspect that the absence of such a page is one of the reasons why the OSMF board now and then gets complaints about things missing on the map to its e-mail addresses and even more times the DCMA take down form is misused for such purposes.
Now, with the redesign live and google making the concept popular the time was ripe to add one at last, nothing fancy, but linking to OSM in the fashion below is a lot better than simply dumping people on our main page:
(a zoom parameter will work too).
Why is it that getting users of our data to attribute OpenStreetMap correctly seems to be such an uphill battle? This is not something new, browsing the wiki, mailing lists and forums show that this has been an issue from day one.
Why is it so difficult for us to get the only compensation we ask for, when at the same time nobody has issues attributing google? Matter of fact it is not rare to see google attribution on OpenStreetMap derived content, to add insult to injury.
Why do OSS projects that would go ballistic if somebody violated their licence take such a cavalier attitude to our requirements?
Why do third party services based on our data go out of their way with smartass attribution links and buttons, just designed to hide the fact that their maps are OpenStreetMap derived?
The OSMF board at their last meeting laid down the law (again) http://wiki.osmfoundation.org/wiki/Board_Meeting_Minutes_2013-12-10
Googles says on the same topic "Without exception, we require attribution when Content is shown. Please do not ask to negotiate this requirement." .
Please don't treat OpenStreetMap worse than mega-corporations!
Updated for the full year 2013
In May (2013) I produced a set of graphs and numbers on our contributor growth based on analysing the regular changeset dumps. Given that this was half a year ago I believe it is time for an update.
Our total contributor number has continued to grow at something around 7'000 per month giving a total of 391'367 at the end of December:
The number of active contributors per month continues to show a slight upward trend: The jump in April 2009 coincides with anonymous edits no longer being allowed.
Which reflects itself in the number of active contributors per year too (the year end number for 2013 is 129'000).
The above graph also shows that the number of contributors that contributed in a previous year is showing growth too. Note the lack of "old" editors going from 2008 to 2009 is likely due to anonymous edits going away in 2009. This is likely also the reason for other low counts in the early years.
And while it is a bit too speculative to state reasons, we do see, compared to active contributor growth, over proportional growth in the number of changesets. Going over 500'000 per month for the first time in September of this year:
You may have noticed that the 0.9 Vespucci release (see http://code.google.com/p/osmeditor4android/downloads/list and on the google Play Store rsn) has an experimental geo-referenced photograph overlay:
this is already very helpful, you can walk around surveying then sit down in peace and quite, enter the information that you have collected and upload right there. Vespucci does not support taking photographs directly (yet?) so I'm doing this with osmtracker. Now and then it isn't really clear in which direction the photograph was taken and it would be helpful to have that information available too.
Most mobile phones today can measure magnetic fileds and have what used to be called a compass :-), however there don't seem to be any (or at least none that are well known) apps that add this information to the picture EXIF information (there are a couple of Android "features" that need to worked around to be able to do this). In any case I hacked together a modified version of osmtracker that actually does this today and made the necessary changes to be able to read the information in Vespucci and this is the result: the rotated camera icon is pointing due east in the direction I took the photograph (of some grass so I'm not showing -that- here :-)).
There is still some work to do to make the actual photographing more robust, for example rotating the mobile phone into landscape mode will mess the compass reading up in the current modified osmtracker version, but the concept seems to have some merit.
We've moved the translation mangement to transifex (the same platform used for iD): https://www.transifex.com/projects/p/vespucci/ .... the sooner we hve at least the major languages complete the faster we can release.
We are nearly ready for a release of Vespucci 0.9, the translations have to be updated and, more important, some more testing needs to be done. Head over to Vespucci Home for the current beta build.
Here's a short video showing the new features.
Maybe some can remember that I produced some statistics on per country address counts end of March this year. I've rerun the scripts twice since then see http://qa.poole.ch/addresses/ .
What is naturally interesting is not so much the absolute number of addresses we have in the database (we know that we are just at the beginning of collecting addresses), but how fast are we adding them.
The full numbers can be seen on the website linked above, but for a quick comparison here are the overall numbers, Germany and the USA (which has had some address imports lately).
2013-03-27 2013-05-03 2013-07-13 Increase since March Total 20'168'470 21'072'447 22'939'124 13.74% Germany 3'659'043 3'836'410 4'144'198 13.26% USA 2'090'893 2'122'662 2'277'269 8.91%
While obviously the numbers are still quite small (the number for Germany is roughly 10% of all addresses there) I believe it is encouraging that we are adding substantial amounts even without large scale imports and do not depend solely on addresses becoming available as open data (which will not happen everywhere).
I suspect that nearly everybody has heard that google has acquired waze for a substantial amount of money.
While waze has historically made noise about generating their own map and have occupied a substantial part of the mind share in the crowd source data space, the effort seems to have never really beared much fruit and inspection of the maps has mainly uncovered third party sources. Still, in the heads of potential consumers and contributors to OSM, waze has continued to be present as a OSM competitor.
The acquisition by google will remove waze from the equation leaving the well known players and OSM as only visible and viable players at a global level.
Very often, if not always, in such high visibility corporate acquisitions, the resulting construct is less than the sum of the individual undertakings and it is not unheard of the net result in the long run ending up smaller than the size of the larger player in the beginning. Now that is unlikely in this case, but on the other hand, just from a numbers point of view, waze is just a pimple on googles user base. Naturally the constructors of such deals are not stupid and it is more likely, regardless of what google say in public, that this was not about acquiring additional market share and technology but more denying that a competitor.
Waze has had in the past had an enthusiastic and loyal community that in the end is mainly responsible for its success, I believe it will be very difficult for google to maintain that community in its context and given the complete integration of waze that will happen regardless of any statements now, it is likely to fall apart in a short time.
Any way I look at it, the google-waze deal opens up opportunities for companies and organisations in OSM-space to provide similar crowd based traffic reporting and avoidance services and to fill the void left by waze.
This is really good news for OSM.
I've added support for adding turn restrictions in the branch of vespucci I've been working on, nothing fancy and rather straight forward.
Select the "from" way
Select "Add restriction" from the drop down menu
Add the "via" and the "to" elements:
Then set the restriction type and you are finished:
We all know that while we have over a million, actually over 1.2 million now, user accounts, the number of actual contributors is quite a bit lower. Often you will see a number of 200'000 quoted, however that only considers the last editor of an object and not all the editors.
Analysing the mid May 2013 changeset dump gives a total of 335'000 unique user ids that have created a changeset. We can see this number increasing more or less linearly in recent years:
Over the last 12 months we have averaged roughly 7'000 new contributors per month, in total a good 80'000 per year:
The numbers only take half of May in to account so it is too early to see if the announcement of the iD editor will already have a noticeable effect. In the long run we hope for a clear increase in the number of account holders actually contributing.
May 2013 numbers now for the whole month, based on the early June changeset dump. Total number of contributors 340'311, new in May: 9357 While this isn't a new record it is a good 2nd place.
With Samsungs recent drive to increase screen size to enormous levels :-) a lot of OSM Android apps have run out of screen real estate that they can reasonably use. Luckily it is trivial to add support for Samsungs multi window facility (IMHO currently available on the Galaxy SIII, SIV, Note, Note II and Note 10.1) without impacting compatibility with other devices.
Adding support is so easy that it doesn't make a lot of sense to provide patches see http://www.modaco.com/page/news/_/android/developers-add-support-for-samsung-multi-window-to-your-apps-r823 and http://developer.samsung.com/s-pen-sdk/technical-docs-09.
OSMAnd has just added support, and I have patched OSMtracker and vespucci versions available.
Note: I have no affiliation with Samsung other than owning a couple of their devices.
The weather in Europe is currently less than great and I took the opportunity to spend some time this weekend adding support for relations to vespucci, the Android OSM editor.
This is currently only a minimal internal support implementation to stop vespucci from breaking relations when editing nodes and ways, however could easily be extended to allow creation and editing of relations in vespucci itself.
If you want an alpha level version for testing again the dev API please send mail to my OSM account.
The "no address" layer on qa.poole.ch now highlights landuse=residential layers that don't contain addresses. I originally intended to do this based on address density, but that doesn't seem to make a lot of sense at the current point in time.
Here an extreme example of how the new layer looks near Atlanta:
You can see bits of green on the above map from the new "has address" layer that highlight buildings and nodes with addresses. With the B&W backgroud you can get a good overview of where we actually have addresses:
More information can be found on the wiki.
One property that differentiates the noaddress layer on qa.poole.ch from other similar layers is that it checks for address information on nodes inside and on the building polygons. This is trivial, ST_Covers for example should return true if a node is in or on the polygon, but, a tiny bit suprising, didn't work with a standard (srid 900913) osm2psql imported database.
select St_Covers(p.way,n.way) from planet_osm_polygon p, planet_osm_point n where p.osm_id=26681434 and n.osm_id=1946037917;
returns false even though node 1946037917 is one of the nodes used to construct the building polygon 26681434.
On inspection of the data in the database the issue was glaringly obvious: the nodes in the planet_osm_point table had a lot more decimal places than the corresponding coordinates in the polygons. This is simply due to the polygons and ways being constructed from the node cache that scales the coordnate values so that they fit in a 32bit integer. In the process the decimal places get truncated (note that there is no loss of information since the main database schema stores the coordinates scaled as 32bit integers anyway, the extra decimal places are likely a result of reprojection for the import).
The fix is easy: simply scale the node coordinates the same way (and back) before inserting them into the database.
If you have a database imported in the standard spherical mercartor projection you can truncate the existing coordinates in planet_osm_point with the following SQL and don't need to reimport (you'll need to get the patched osm2pgsql version from the OSM svn too naturally):
update planet_osm_point set way = ST_SetSRID(ST_MakePoint(trunc(cast(St_X(way) as numeric),2),trunc(cast(St_Y(way) as numeric),2)),900913) ;
I've added a 2nd layer to qa.poole.ch that highlights building outlines for buildings that have neither addr:housenumber or addr:housename tagged and don't have a node with such tags inside or on the outline.
Minor buildings (hut, garage, garages, rood, terrace, greenhouse and shed) are currently rendered in orange, however I may suppress them completely in the future.
The check for a node in or on the building outline is computationally rather expensive leading to the layer not being as fast as the noname one. Zoom 0-12 are regularly re-rendered in batch mode.
Naturally in some areas it is common to not have building specific address information, we are currently missing tagging to suppress such false positives. I'm planning on adding support for addr:interpolation and some further addr tags that are not so common. If you have any susgestions please leave a comment.
Added support for addr:conscriptionnumber, addr:full and addr:interpolation. Note that this currently makes the layer substantially slower at lower zoom levels.
I've set up an experimental noname layer at http://qa.poole.ch/ .
It differs a bit from previous attempts in that it respects noname and unsigned tags (that have never become popular) and is generated directly from a synchronized database. Currently changes can be expected to show up in roughly 5 minutes.
- red: major and minor roads, service with service=alley without name or ref tag
- orange: unspecified service, unnamed roundabouts, motorway_link, trunk_link
- dashed red: unsigned=yes
- dashed orange: unsigned=yes and/or noname=yes
update 28th October 2012
- render motorway_link and trunk_link in orange. There is currently no clear way to tag such roads in the absence of explicit sign posting.
- reduce line widths at lower zooms
update 16th November 2012
- render *_link in orange.
- support validate:no_name=yes as synonym for noname=yes and validate:no_name=no_sign as synonym for unsigned=yes
- note: only zoom levels 0-9 will be automatically re-rendered
update 26th November 2012
- support name:left amd name:right as valid name tags
- note: only zoom levels 0-9 will be automatically re-rendered