Recent diary entries
High intensity OSM data capture just driving about..Posted by bob3bob3 on 15 February 2023 in English (English).
I have just spent the last 2 months “intensely” making observations around lesser populated areas in WA’s southwest almost solely from the vehicle drivers seat. I then enter that data into OSM ID most afternoons/nights.
- Forward looking 8.3MP/4K dashcam 2FPS
- Left facing (270 degrees) old Samsung 8MP phone 0.3FPS
- Left rear facing (225 degs) 1.3MP webcam 1FPS
- The dashcam records audio (my voice) in one minute chunks
- All photos and audio are geo-referenced to their current location
- The Garmin navigation GPS has a current OSM based map, a POI set of all WA stopping places from MRWA and any OSM FixMe’s.
Most of the data is by voice prompts that are reviewed afterwards, but a small proportion come from only viewing (mainly) the 270 camera images of shop etc fronts and signs. It is surpising how useful the multiple camera views are.
- Every road surface change paved vs unpaved on route. (incl concrete bridges)
- Any road calming objects (humps and chicanes) and gates on existing roads
- Any one-way road situations and associated signage.
- Road surface left and right of track (surprising how many are not currently valid)
- Verify road names signs vs Garmin GPS and voice differences. (including missings)
- Verify waterway names signs vs Garmin GPS and voice differences. (including missings)
- Voice any obvious missing public cross etc roads or major private ones (eg sealed mine roads)
- Voice all bridge names (if they exist)
- Respond to FixMe’s (many an unknown water crossing!)
- Respond to any WA stopping places. (The Garmin will show “Parking” if it is already on OSM, so could be ignored)
- Voice any obvious informal parking areas (like stockpile/gravel pit areas) plus any formal not in the MRWA POI’s
- Voice any farm/property names/type
- Voice as many town/other objects as possible, priority to Fire, Police, Ambulance, Post Office & Telstra but try to get all shops, parks, recreation grounds, playgrounds, toilets, camp areas, dump points, water points/tanks, fuel stops, rest areas, touristy things, information boards, welcome signs, comms towers, windpumps, large dams etc as possible.
- The photos (with certain filtering) are also uploaded to Mapillary, but I don’t actually use the current set with ID.
- How I enter/edit is covered in previous diary posts. The basic flow though is to scroll each 1 minute segment of the forward dashcam ONLY at the active audio point. Then use all camera views, the overhead imagery and what I have voiced to edit ID.
I now need a rest and throat/larynx recovery!
Integrating audio information with Mapillary image handling and OSM inputPosted by bob3bob3 on 4 January 2023 in English (English).
I run a Mapillary supplied BlackVue dashcam uploading my take via the mapillary_tools in two “stabs”. The first is usually same day for road speeds segments below 55kph, the second is up to 2-3 months later. This gives town/village streets and remote road intersections/signs coverage first, for other mappers. I am also taking 1FPS rear left (webcam) and right angle left images (Samsung phone) at less than 40kph for immediate geotag, process/upload. (after manually culling privates)
- Waiting for images to appear on the Mapillary site for my own OSM work degrades and reduces the result. For some time I had processed and geotagged locally, so I could scroll through (1FPS) images at the end of every day. Unfortunately the camera doesn’t always capture sufficient detail and scrolling through thousands of images takes time!
- I would use Geeqie (image viewer) and a “side” text file (gpscorrelate -m) of image filenames and lat/lon, then copy/paste these direct to the OSM ID editor search box.
Some months ago I started talking into the camera whilst driving. Eg street addresses, business names, features in rest areas etc. The effort is about rough cut removing “no data” and being able to find feature add/changes quickly/accurately. My action is now;
- Process/geotag images as before, but also extract a (1 minute) wav file (ffmpeg) from the BlackVue mp4. This has the same file naming standard and thus indirectly geotag references against the associated image(s).
- Move all images and mp4’s that have no movement (speed=0) for an entire 1 minute block out of future processing.
- Run the wav files through a sox bandpass (sinc) filter to reduce road noise.
- Run each wav through sox vad and remove those that have no voice on it. I have also modified certain timing constraints to reduce false detects. (Currently vad -t 12 -l 1000 -L 1000 -h 300 -H 300 -T 1)
- Move the already tagged images in with their one minute block matching wavs.
- The directory now only contains 1 minute blocks of audio/wav and images/jpg of voice only and vehicle moving.
- All of the above steps are launched by a single bash script on the laptop near the BlackVue. I only need connect to the BlackVue WiFi and run it. It starts downloading the camera (curl/wget) and 1-2 hours later the action directory, gpx data and moving mp4’s are ready. This data is rsync’d to another laptop for the OSM processing.
Whilst the above is running I often do some OSM/ID work on the previous day’s data (2nd laptop) plus action the side camera culling and uploading.
Have setup 3 plugins (toolbar icons) in Geeqie; Launch the 1 minute wav file in Audacity associated with that image, copy the lat/lon to the clipboard, and move the entire 1 minute band of images and wav out of the active/processing directory.
- Playing/viewing the 60 sec wav in Audacity (via the Geeqie icon) has 5 second tick marks that correlate with the image filename. Eg 20220504_142256_044.jpg will be at the 44 second mark. If there is only one voice peak on the Audacity display I can quickly scroll/roll to it, not having to view intervening images. If the audio track is very dense, like driving past a long row of shops I can pause audio and scroll/roll as needed. Passing vehicles and UHF FM chatter looks very different on the Audacity display, so these can be ignored.
The side camera files are similarly named so can be viewed for editor input. The raw >40kph take is also available, but is often too wide time spaced or blurred. They do however work very well on shopfront signs. These image files have also been geotagged, so the Geeqie copy lat/lon button also works.
- Clicking on the lat/lon icon puts the current image position onto the clipboard, that then gets pasted to the ID editor search box.
- I would suspect that I could save a further few seconds by URL launching “remote” on the browser, but not yet!
- When I have completed the feature in ID I click on the Geeqie “move” icon and the entire 1 minute image and wav file set vanishes!
And of additional interest.
Voice detection isn’t perfect, but sox vad can be adjusted. Currently I select on sound peaks and get some false detects from the two way radio and passing vehicle slipstreams.
I have a GPS unit in the top left of the windscreen and the (centred) BlackVue creates *.gps files that can be converted to nmea. I can then therefore do a credible street alignment check on OSM by driving up and down, cutting the associated 4 tracks with Viking then combining (gpsbabel) for a OSM private trace upload. It is easy to see any accuracy issues by having so many detailed tracks superimposed.
It appears that gpx files created (gpsbabel) from the BlackVue *gps (nmea) file don’t have complete speed information. I therefore use the nmea data to tag (exiftool) with. As an interesting extra the BlackVue can be GPS unlocked for the first 1-3 minutes, so the second GPS unit nmea data can be used to fill the gap.
I also do a rough cut of the BlackVue mp4’s to remove those where the vehicle was not moving. Handy if I stop for lunch and keep the camera running. I only want to eventually upload mp4’s that have movement.
Regional parking and rest areas - my methodPosted by bob3bob3 on 4 January 2023 in English (English).
Thought it worth posting this.
After much frustration I have come up with what I do regarding amenity=parking and highway=rest_area (plus details) and have been running this way for 6-12 months. This is live data input (my eyeball/camera/recording etc) as I drive through. I think there have been maybe 1200-1400 objects so far!
I have generally adopted a formal vs informal data set. Formal is almost based wholly on site signs and informal on a mix of triggers and interpretations.
This generally only applies to regional places, not so such towns and high population areas.
There are always contentions! I have come across many (other mapper) highway=rest_area objects that are not “my formal”, as I make no allowance for locations that “could be”. Indeed any parking area could be an informal rest area, so are informal ones needed at all? In the end I sometimes use informal=yes or informal=no to clarify.
So looking at the SA and WA govt databases and my limited travels, SA calls everything “rest areas” and their naming system starts “RA”. This is also seen on roads where virtually every stop has a rest area sign at 1km before, followed by a “P” or two. WA calls everything “stopping places” with a plain number designation. It’s almost as if “P” locations in WA are not allowed to be rest areas! There is also a distinct lack of HGV defined “P” in WA where I wonder if trucks actually get stuck in small ones! Of course Qld is quite unfriendly in defining large areas with useful facilities as HGV only, but that’s another story!
The “informals” are basically NHVR Green Dot, Stockpiles/gravel pits and any place that looks like a truck has been parking for a few years! I also map gravel pits that “could be” used for parking.
So onto the process.
- A formal sign saying rest area or the usual graphic tree and table; highway=rest_area also implies additional amenity:parking object and sub tags
- A formal blue “P” parking; amenity=parking access=yes (public) hgv=yes if self judged that an HGV will fit and no other HGV signs exist
- A formal blue HGV “P” parking; amenity=parking hgv=designated
- Any formal “no truck signage” hgv=no (ie hgv are prohibited.) access=yes (public)
- Any formal “no car/LV signage” in a dedicated HGV location hgv=designated access=no
Rest area objects are most often nodes only with just the name. No other tags. (except a possible informal as mentioned above) Parking objects contain maximum tag details. A rest/parking area that has multiple signs only usually has two objects, the parking one containing both access=yes and hgv=designated etc A rest area sign implies parking exists within it, but a parking sign does not imply a rest area exist
- Any informal site hgv=yes if that is the current usage or will fit hgv=no if the entry/exit is too tight Then above are also in precedence order. Example - some formal parking areas were formerly NHVR green dot type, so the formal tagging is used with description including prior NHVR status. (unless the green dots are removed…)
INFORMAL vs not tag
A formal sign, obvious high expenditure and/or listing in a formal (waivered) government database informal=no or no informal tag Road type that routes to is always highway=service, service=parking_aisle and goes through a parking=surface. If parking objects are nodes they are placed on the service road. Alternatively parking=street_side adjacent the existing road. capacity:hgv is NOT used bin=yes or bin=no lit=yes otherwise not tagged name=Formal or anecdoctal name Description=Anything special. Examples may be “Informal turnout, low speed vehicles”, “HGV LV Stopping Bay”, “HGV only Stopping Bay”, “LV only Stopping Bay” - preceded by relevent blue sign
The NHVR green dot areas informal=nhvr Road type is always highway=track and goes through parking=surface (area or node) In most cases surface=unpaved of both the parking and through road. It is however possible to see asphalt, fine_gravel and so on. Alternatively parking=street_side adjacent the existing road. capacity:hgv=1 (or more) as a judgement of how many moving 25x4m shapes can independently arrive/leave. bin=yes or bin=no description=NHVR informal parking (Green Dot) area
Stockpile or gravel pits that are or are likely to be used for parking informal=stockpile Road type is always highway=track and goes through parking=surface (area or node) In most cases surface=fine_gravel of both the parking and through road. Alternatively parking=street_side adjacent the existing road. capacity:hgv=1 (or more) as a judgement of how many moving 25x4m shapes can independently arrive/leave. Allow for the piles of dumped gravel reducing that number, the rule of thumb being just one lane width going through bin=yes or bin=no description=Stockpile site with possible use as truck parking area OR Stockpile site with evidence of use as truck parking area OR Stockpile site, possibly too tight for HGV with hgv=no.
Any other informal places that are or are likely to be used for parking informal=yes In most cases parking=street_side adjacent the existing road is used, else any through road type is always highway=track and goes through parking=surface (area or node) In most cases surface=unpaved of the parking object. capacity:hgv=1 (or more) as a judgement of how many moving 25x4m shapes can independently arrive/leave. description=Anything useful but most often left blank
Other objects at parking areas
- Picnic shelter and if provided bin and lighting
- Picnic tables
- RV dump points
- Toilets with normal iD full details including lit=yes if so. I also enter the rest area etc as name if known.
- When creating parking objects directly on both sides of a road offset them slightly so the left side “comes first” (for navigator routing engines)
Fixme's in the workflowPosted by bob3bob3 on 14 March 2022 in English (English).
14th March 2022
I am now starting to tackle (mainly my own) fixme’s.
Something like a farm gate and station name is not always readable on my imagery, but even after creating a feature and merging in other sources (like overheads), there are information gaps. A fixme is handy, but I need to action them as well!
I ran a simple overpass query for all node and way fixme’s in a boundary box, then exported it to gpx. I initially thought a Viking moving (GPS) map would work well, but the laptop is on the passenger seat away from my driving vision. I ended up (default) converting the gpx to a Garmin poi file with gpsbabel. set to warn at 800m distance . The “name” is displayed if that feature field is populated, else the node/way number. Additionally if one calls up the POI information on the GPS navigating unit the fixme text is displayed.
Basically I just navigate to the nearest fixme POI using the Garmin GPS display/selection tools. (Note that most of my work is in rural/remote areas, not urban)
I can then drive up close to the “problem” with the dash camera and capture for later editing. I also use the audio from the dashcam mp4 which is me talking into it! Each movie and frame are time referenced to the audio, so it is easy to correlate. I can for example read a fine print sign at a distance that the camera doesn’t capture as well.
My workflow method...Posted by bob3bob3 on 4 April 2021 in English (English).
5th April 2021
As I tour around Australia I both capture street images for Mapillary, plus do online OSM editing. I don’t have a fixed address, so all my work is done mobile.
Mobile Internet access is quite expensive here, so the daily take of between 8 and 20 Gbytes is saved to a portable USB drive and another mapper kindly uploads 500-1500 GBytes for me at a time. This of course can delay online imagery by 3-4 months as I have to mail or take the drive to the upload location.
The same imagery (downrated to 1FPS) I take on a daily basis is locally geo-encoded and viewed at perhaps 10FPS - 8,000-15,000 per day. Those of interest (100-200) I save to a FIFO directory and periodically run a gpscorrelate job to dump lat/lon to a text file. This text file is then used to cut/paste into the OSM ID editor search/location fiel. Each completed image is moved to a “done” directory and the co-ordinate cut from the text file makes it easy to know which is next.
I correlate the overhead Bing/DCS/Mapbox etc view with ground level ones to make map changes.
I tried launching OSM/ID from Viking or the image directly but it always created a new browser instance.