OpenStreetMap

aharvey's Diary

Recent diary entries

I’m no expect in the tagging proposal process but I do think on a whole it’s helpful and a good thing to have in the OpenStreetMap community.

It provides the necessary documentation to both help mappers (to know what tag they should use to be consistent with everyone else) and data consumers (to know what each tag is so they can decide how to interpret it).

The process of putting together this documentation and asking for feedback is great because it ensures that a wide range of people from different backgrounds and perspectives can raise suggestions or issues.

However one aspect of the proposal process I’m convinced should be changed, as it stands the proposal goes to a vote and needs over 74% majority to pass. While on the surface it sounds good that a significant opposition can block it, in practice it can leave the mappers and data consumers with no way to map something. If the proposal is changed to make the 26% opposition happy, then the 74% of supporters don’t swap over in enough numbers to the other side then nothing would ever pass.

The proposal process instead should be changed to something with the high level like “We need a way to map X”, anyone can propose an option, eg.

“X is mapped like A.” “X is mapped like B.” “X is mapped like C” etc.

Anyone could propose how they feel is best to map X, and when it goes to a vote you have to choose one of the options, and the one with the majority wins.

We shouldn’t be in a situation where there is no approved way to map a mapable feature just because the community doesn’t agree completely on how to map it.

TfNSW Speed Cameras Conflation

Posted by aharvey on 21 September 2020 in English.

I recently wrote about using the JOSM Conflation plugin to identify traffic lights in OSM missing from the TfNSW’s open data.

Now TfNSW have released some data on enforcement cameras, so I’ll try again using JOSM.

  1. Download the CSV data
  2. Rename the Lat (1) and Long (1) headers to Latitude and Longitude. 3 Open the CSVs in JOSM.
  3. Select all the imported nodes from the Fixed Speed Cameras and replace all the tags with highway=speed_camera
  4. In JOSM, download existing speed cameras as a new layer from overpass with ((type:node and highway=speed_camera) or (type:relation and enforcement=maxspeed)) in "NSW, AU"
  5. Conflate

  6. Select all the nodes from Red Light Speed Cameras, and replace all the tags with highway=speed_camera
  7. Conflate again.

Conflating TfNSW Traffic Lights dataset with OSM in JOSM

Posted by aharvey on 18 July 2020 in English. Last updated on 19 July 2020.

Conflation

  1. Download the XLSX from https://opendata.transport.nsw.gov.au/dataset/traffic-lights-location
  2. Open the XLSX in LibreOffice and add an extra column start_date with formula =IF(C2, CONCAT(YEAR(C2), "-", TEXT(MONTH(C2), "00"), "-", TEXT(DAY(C2), "00")), "").
  3. Convert the XLSX to CSV
  4. Open the CSV in JOSM (may need open data plugin)
  5. Select all the CSV nodes and then update Equiptment_ID to ref, delete Date_install, Suburb.
  6. Edit > Search, with search string Asset_Type=VEH with results set to select. Add the tag highway=traffic_signals and delete the tag Asset_Type.
  7. Edit > Search, with search string Asset_Type=PED with results set to select. Add the tags highway=crossing + crossing=traffic_signals and delete the tag Asset_Type.
  8. Add a new layer and download from OSM via Overpass using the wizard text (highway=traffic_signals or (highway=crossing and crossing=traffic_signals)) in "NSW, AU"
  9. Active the conflation plugin, and configure. Select all the TfNSW nodes and then freeze that as the reference, then select all the OSM nodes and freeze that as the subject.
  10. Change the distance to 50 (in meters)
  11. Set merge tags and list highway,crossing,ref,start_date and then except STREET_1,STREET_2.
  12. The other options should be Simple, Disambiguating method.
  13. Generate matches

Now in the conflation tab there is “Reference only” which are ones in TfNSW but not OSM. Subject only are ones that are in OSM but not TfNSW, and matches shows ones that were found in both but have have a different location and/or tags.

At the time of writing there were 768 found in TfNSW not found in OSM and 5,306 found in OSM not found in TfNSW. But take this with a gain of salt as there are differences in how they are modelled in OSM.

Building a Maproulette Challenge

  1. From JOSM I exported the nodes missing in OSM as a .osm file.
  2. Following the example at https://github.com/maproulette/mr-cli#example-1-adding-new-unrelated-osm-nodes using mr-cli run mr-cli cooperative change --out challenge.json josm-export.osm
  3. Create a new challenge on Maproulette using the challenge.json file you just created, eg https://maproulette.org/challenge/13925
Location: Chinatown, Haymarket, Sydney, Council of the City of Sydney, New South Wales, 2000, Australia

I try to capture Mapillary imagery where I can, mostly from my bike, sometimes 360 imagery sometimes forward facing only.

I started in 2017 basically just to give it a go, I bought a 360 camera, the LG 360 CAM. Even that was a struggle, they didn’t sell them locally here so I needed to buy from random sellers on ebay. LG don’t make these anymore, but I’ve stockpiled 3 which is good if one breaks or for swapping them when the battery runs out. Overall I’m happy with the camera, it’s not top of the line, but for the price I paid about $100USD each they have satisfactory image quality, battery life and capture rate.

While I can’t usually use my 360 imagery to read off the smaller road signs (only larger destination signs) I find the it’s much easier to do mapping later from 360 images.

After a capture, I’ll usually go through and use it to add in as much detail to OSM as I can from my imagery, and then upload it to Mapillary.

While I’d get value even without uploading to Mapillary I feel this is important to be able to provide some kind of verification to my edits. It also proves very useful later on when reviewing others edits, I can take a quick look of what was on the ground before from Mapillary and just check that new edits look right.

I’ve been very supprised how much I’ve ended up re-checking my captures in Mapillary. Many times reviewing edits in OSMCha I’d be surprised that I actually had coverage for that area.

At the start, mapping out a whole city might seem like too big a task, but just like OSM, chipping away one bit at a time, progress slowly is revealed year after year.

Anyway, back to the point of this diary entry. I recently upgraded to a new phone and found that the LG360 app no longer worked. It would connect to the camera (to see the camera stats) but wouldn’t let me start the capture.

So I did a bit of digging and re-discovered an old post I’d come across before https://forum.xda-developers.com/lg-g5/accessories/lg-360-cam-live-streaming-t3590564

This reminded me that the camera supports the Open Spherical Camera API from Google https://developers.google.com/streetview/open-spherical-camera. Good on Google for making an open standard API and good on LG for building upon this.

So basically I now needed a way to invoke that OSC API from my phone to the connected camera.

Termux to the rescue.

Termux is pretty cool. I worked out I could write a bash script to trigger the camera to start a 2s capture, and then add this as a home screen launcher. Sweet. You can see the scripts at https://gist.github.com/andrewharvey/680e370e8ad66b745c1e9dd02c0b4d0f

This means you can even capture Mapillary imagery from the LG360 without the LG app (since it seems LG is no longer maintaining it).

Community is hard

Posted by aharvey on 17 February 2020 in English.

I like to stay up to date with discussions about OSM but I think I’m at a tipping point.

I connect to IRC through Riot.im (DWG channel), Discord (OSM World), two Slack channels (Maptime Oceania and Maptime US), a few mailing lists (talk-au, tagging, talk), Twitter (@josmeditor, @openstreetmap), a weekly newsletter (weeklyOSM), a couple of RSS feeds (help.openstreetmap.org, new notes, diary entries), OSMCha (monitor changesets), OSM wiki changes on some pages I watch and changeset comments topped of by a once a year in-person community event, FOSS4G SotM Oceania.

My Bicycle setup for Mapillary

Posted by aharvey on 25 August 2017 in English.

I’ve started getting into capturing on Mapillary and have found it to be incredibly helpful for mapping on OSM. I wanted to go through my setup in case it’s interesting for others.

bike setup

  • I prefer to capture 360 images
  • I use the LG360 (R105)
  • I bought two LG360 cameras off ebay, $125USD and $90USD. The LG360 is on the cheaper end of the scale compared to some of the others on the market.
  • Though there is a flaw in one of the cameras I bought which is a shame as you can see the left half of the image (front camera) is blurry.

LG360 lens issue

  • I’m happy with the image resolution and quality. I wouldn’t want it any worse though. Most of the time street signs and parking signs aren’t readable, but speed limits and destination signs are.

Destination sign

  • The LG360 is unreliable, sometimes it stops capturing, connecting to your phone can take several minutes and attempts before finally working, there can be gaps in the GPS tagging where it will use the last location for maybe 30 seconds (even though the GPS log might have data; in this case I end up using gpscorrelate, however I’ve found you have to play around with the time offset and even then you might get it right for the start of the trip but it will be out at the end).
  • I think the battery should last for around 1hr 45min at a 2 second capture interval.
  • I’m happy with the 2 second capture interval, but wouldn’t want anything not supporting at least 2 seconds.
  • I’m using a 64GB Micro SD card XC I U1. Photos are about 5MB so you should get around 10,000 photos or 6 hours give or take.

  • I capture on my bicycle. I have a rack mounted which I attached an old broom stick I found on the side of the road to, which I then attach the Monopod to. It’s all held together with Duct tape. The broom stick I’m using is light weight but does give some flex (if the camera had a built in gyro it could correct for it in software, I mean it’s a 360 camera after all!).

  • I use this phone mount. It’s great. Very easy to attach/detach your phone, has a Silicone band to hold you phone in place (I’ve had my phone come out of the claw once going over a pothole, and the band ensured it didn’t come out). There is this ball joint which is stiff enough that you can just change the angle of your phone and it usually stays in place. I got mine for $2.10 USD.

bike phone mount

bike phone mount

pole setup

This pole is longer than the previous one. All the 360 photos on my Mapillary profile to date have been with the shorter pole. I’m still working out the best height to use. The higher the camera then the less I obstruct the view, but the more tree branches, signs etc I run into.

The main problem at the moment is it’s a real pain uploading to Mapillary when you have something like 10GB a day and can only do around 90KB/second. I end up using rsync to a remote server as rsync doesn’t mind being interrupted a hundred times and it’s easy to limit the bandwidth. I also use lepton to compress the JPEGs further. WebP would be even better! The scripts I use for this are at https://github.com/andrewharvey/lg360-mapillary-helpers

If you get lucky on ebay parts are around $105 USD + duck tape, a bike, a phone and a Micro SD card.

Location: Quay Quarter, Sydney, Council of the City of Sydney, New South Wales, 2000, Australia

In reply to the comment from rw__ (http://www.openstreetmap.org/user/aharvey/diary/14408#comment16624):

I simply read through the CTs (http://git.openstreetmap.org/rails.git/blob/696fbb4e16c13f8f86134ad5c029644a5277a373:/config/legales/GB.yml), and by the end I had all the parts checked, I couldn't find any reason not to press agree, so I did.

At no time did I understand it to mean I must ensure my contributions are compatible with ODbL. I cannot ensure my contributions are compatible with ODbL, and I believe that I have contributions which are not compatible. But that isn't my problem.

>intro: "Thank you for your interest in contributing data and/or any other content (collectively, 'Contents') to the geo-database of the OpenStreetMap project (the 'Project'). This contributor agreement (the 'Agreement') is made between you ('You') and The OpenStreetMap Foundation ('OSMF') and clarifies the intellectual property rights in any Contents that You choose to submit to the Project in this user account."
> 2 next_with_decline: "Please read the following terms and conditions carefully and click either the 'Accept' or 'Decline' button at the bottom to continue."
> 3 introduction: "Introduction"
> 4 section_1: "We respect the intellectual property rights of others and we need to be able to respond to any objections by intellectual property owners. This means that:"

okay... just statements so far.

> 5 section_1a: "Your contribution of data should not infringe the intellectual property rights of anyone else. If you contribute Contents, You are indicating that, as far as You know, You have the right to authorize OSMF to use and distribute those Contents under our current licence terms. If You do not have that right, You risk having Your contribution deleted (see below)."

Check, I indicate that, as far as I know, I have the right to authorise OSMF to use and distribute the work I upload under the current license terms (i.e. the CC-BY-SA license).

Check, latter part doesn't matter as I have the rights. (i.e. I've only uploaded works compatible with the CC-BY-SA license)

> 6 section_1b: "Please note that OSMF does not have to include Contents You contribute in the Project, and may remove Your contributions from the Project at any time. For example, if we suspect that any contributed data is incompatible, (in the sense that we could not continue to lawfully distribute it), with whichever licence or licences we are then using (see sections 3 and 4), then we may delete that data."

again just a meaningless statement hear, of course the OSMF doesn't have to use works I upload..

> 7 rights_granted: "Rights Granted"
> 8 section_2: "Subject to Section 3 and 4 below, You hereby grant to OSMF a worldwide, royalty-free, non-exclusive, perpetual, irrevocable licence to do any act that is restricted by copyright, database right or any related right over anything within the Contents, whether in the original medium or any other. These rights explicitly include commercial use, and do not exclude any field of endeavour. These rights include, without limitation, the right to sub-license the work through multiple tiers of sub-licensees and to sue for any copyright violation directly connected with OSMF's rights under these terms. To the extent allowable under applicable local laws and copyright conventions, You also waive and/or agree not to assert against OSMF or its licensees any moral rights that You may have in the Contents."

So far as I have these rights to grant, I grant these rights. I do not guarantee I have these rights to grant, but if I do, then I grant these rights.

If the OSMF switched to the ODbL from CC-BY-SA they would be violating the rights of someone else (i.e. the person who owns the rights or parts of the rights to some of the content I've uploaded to OSM), but that is the OSMF who is violating the rights...

> 9 section_3: "OSMF agrees that it may only use or sub-license Your Contents as part of a database and only under the terms of one or more of the following licences: ODbL 1.0 for the database and DbCL 1.0 for the individual contents of the database; CC-BY-SA 2.0; or such other free and open licence (for example, http://www.opendefinition.org/okd/) as may from time to time be chosen by a vote of the OSMF membership and approved by at least a 2/3 majority vote of active contributors."

Again I'm not guaranteeing that I am able to grant these rights, but merely insofar as I can, then I agree. In fact I know that I can't grant these rights for a number of my contributions, the ones I can I do, the ones I can't well I cant.

> 10 active_defn_1: "An 'active contributor' is defined as:"
> 11 active_defn_2: "a natural person (whether using a single or multiple accounts) who has edited the Project in any three calendar months from the last 12 months (i.e. there is a demonstrated interest over time); and has maintained a valid email address in their registration profile and responds to a request to vote within 3 weeks."
> 12 section_4: "At Your or the copyright owner’s option, OSMF agrees to attribute You or the copyright owner. A mechanism will be provided, currently a web page http://wiki.openstreetmap.org/wiki/Attribution."

I don't care... for the rights I do have, I disclaim them, I wish the works to be CC0, OSMF or anyone else can do as they wish. But I'm not the sole owner of the rights to the works I've uploaded, and if you don't license this CC-BY-SA you are violating their rights, not mine.

> 13 section_5: "Except as set forth herein, You reserve all right, title, and interest in and to Your Contents."
> 14 limitation_of_liability: "Limitation of Liability"
> 15 section_6: "To the extent permitted by applicable law, You provide the Contents 'as is' without warranty of any kind, either express or implied, including without limitation any warranties or conditions of merchantability, fitness for a particular purpose, or otherwise."
> 16 section_7: "Subject to any liability that may not be excluded or limited by law, neither You nor OSMF shall be liable for any special, indirect, incidental, consequential, punitive, or exemplary damages under this Agreement, however caused and under any theory of liability. This exclusion applies even if either party has been advised of the possibility of such damages."

That last part, I'm not liable for anything under this agreement...

> 17 miscellaneous: "Miscellaneous"
> 18 section_8: "This Agreement shall be governed by English law without regard to principles of conflict of law. You agree that the United Nations Convention on Contracts for the International Sale of Goods (1980) is hereby excluded in its entirety from application to this Agreement. In the event of invalidity of any provision of this Agreement, the parties agree that such invalidity shall not affect the validity of the remaining portions of this Agreement. This is the entire agreement between You and OSMF which supersedes any prior agreement, whether written, oral or other, relating to the subject matter of this agreement."

okay...

After some convincing from some folks on IRC and a re-read of the CTs I've ticked the accept button.

My head is in knots at the moment trying to comprehend all this legalese with logic, but I'll try to explain my line of thought.

Basically by uploading to the OSM API, I am publishing works. Sometimes I am the copyright holder of such work, and sometimes I am uploading work that someone else is the copyright holder to. I ensure that if I am not the copyright holder to the work, that the work is indeed compatible with the current OSM license -- CC-BY-SA. If I am the copyright holder, then I grant everyone a license to do anything that copyright would forbid, i.e. I license it CC0. That is my intent. But I can't guarantee this. A lot of my work needs to be CC-BY-SA because it is derived from nearmap, and the other bunch is derivative works of existing OSM data, which also must be CC-BY-SA.

If I interpret the CTs as I give you the rights to do anything with the data I upload, so much as I can, but don't guarantee such, rather than I guarantee that you have the rights to do anything with the work I upload, then I believe I can tick the agree to the CTs button.

anyway... I'll have to think about this some more later.

Crunch Time

Posted by aharvey on 24 June 2011 in English.

So push finally turned to shove, and I decide to abandon the OSMF/OSM database/API and instead rely on the FOSM database/API. It's the only real alternative for me at the moment.

It all came down to mostly two things,
1. nearmap. I depend heavily on nearmap imagery for most of my on the ground/surveyed edits (and wholly depend on it for armchair mapping places I've never been to). I do this because
a) I though it's more accurate than a consumer grade GPS, or bing (which I never found had a solid enough legal footing than I would prefer)
b) It is much more convenient for me to trace imagery than carry around a GPS everywhere and taking down tracks and waypoints.

Of this nearmap derived data, I must use the CC-BY-SA license. I can't change that, so I can't agree to the CTs and continue mapping to the OSM database.

nearmap is important to me, because most of the data in my area and most of my contributions are derived from nearmap, and I don't want to see that lost.

2. Back when I tried to read the CTs it was my understanding that you either needed to own full rights over the data you contribute, or the data must be essentially public domain for you to upload it. I would rather a project which distributes its data under the foo license to be able to accept foo licensed data. (ie. I think a CC-BY-SA or ODbL OSM should accept CC-BY-SA or ODbL data. it seems the CT's break this)

The rest of my data is mostly wholly owned by me (eg. completely survey or derived from PD data sources), in which case I license this data CC0.

This decision has nothing to do with the ODbL or any other license. Since nearmap isn't compatible with the CT's, and I want to keep my/others nearmap data, the actual future license is irrelevant at this point in time.

As 80n said, fosm is a fork of the data, not the community. I want to share the same wiki and tagging discussions, not fork the whole of *.osm.org.

Sure the fosm API seems slower than the OSM one, but at least it seems to be functioning properly, and is publishing the minutely diffs. It could do with this rails port thing for browsing the data via the web/tracking changes via rss... Hopefully some kind soul will set up a tile server based on fosm data. Unfortunately I can't as I lack the network connection and hardware (but if the NBN gives me a cheap unlimited data quota 100Mbps uplink speed I would probably have good reason to invest in some hardware).

osm is turning me into a botanist

Posted by aharvey on 24 May 2011 in English.

... a year ago I would have never guessed OSM would turn me into a botanist: http://www.openstreetmap.org/browse/changeset/8234306

This particular area was overwhelmingly comprised of the homogeneous genus and species of plant. Not sure how to area areas which have a mix of plant types.

Location: Cronulla, Sutherland Shire, Sydney, Sutherland Shire Council, New South Wales, 2230, Australia

Just saw some JOSM news

>2011-05-16 (4081, 4085, 4086)
>New Follow Line feature to ease drawing of adjacent areas. (similar to Potlatch 2's "F" shortcut)
>Search (and Filter) can query closed ways now, and those enclosing a certain minimum and maximum area. (Keywords: closed; areaSize:MIN-MAX or areaSize:MAX, in m²)
>The download Object... feature now accepts a list of ids.

Can't wait for the next stable release into debian/unstable. If these features work as advertised they will be big time savers for common tasks of mine. Particularly making a new way which shares the nodes as another and downloading a bunch of osm objects given a list of ids.

Thanks to those who contributed these features into JOSM!

Yesterday I committed a patch to the main OSM mapnik rendering style to render slipways (the icon had been published in the SJJB set, but wasn't in the OSM stylesheet), seemed like an easy fix, so without knowing if there is any protocol to follow I made the change after observing similar commits, and already tile.openstreetmap.org is serving tiles with this slipway icon! Excellent.

There's only one way for progress to occur and that is to get to it and start making those changes happen.

I support this proposal http://wiki.openstreetmap.org/wiki/Relations/Proposed/Collected_Ways_Simple

It makes a lot of sense to me, often you need to split up a way to assign different individual tags, whilst keeping some tags common. eg. you have a road with a name, but it has varying maxspeed. You don't want to have to duplicate the name across each section.

There seems to be a bunch of proposals for specific cases, it makes sense to me to just use a relation with type=collection and no roles for the ways. Of course you would need to be allowed to have relations as members of other relations too.

I see uses for type=collectedStreet and multipolygon (where you need inner and outer roles), but for this kind of thing, type=collection seems like the best solution.

To make this change happen I think we need to make sure renders and editors can handle and are aware of this.

This would replace,
http://wiki.openstreetmap.org/wiki/Relations/Proposed/Collected_Ways
http://wiki.openstreetmap.org/wiki/Relations/Proposed/Segmented_Tag
http://wiki.openstreetmap.org/wiki/Relations/Proposed/Street (using associateStreet to bind houses to a street)
http://wiki.openstreetmap.org/wiki/Relations/Proposed/Waterway

changeset comment

Posted by aharvey on 9 November 2010 in English.

Putting my changeset comment here (for changeset http://www.openstreetmap.org/browse/changeset/6325940) as it won't fit in the comment tag.

Add land parcels (cadastral boundaries) from best guess from nearmap imagery, and add street numbers and apartment block names from survey to these parcels. boundary=cadastre is something I just made up, will need to discuss the use of this further. Unable to assign the role street to the associatedStreet relations because you need to make a relation containing all the segments of the street, and how to do that is not well agreed upon (multipolygon, collected way, route?).

Location: Kensington, Eastern Suburbs, Sydney, Randwick City Council, New South Wales, 2033, Australia