OpenStreetMap

Cleaning Up GPS Traces for OpenStreetMap and Visualizations

Posted by skorasaurus on 18 September 2012 in English (English)

While going through the rest of my GPS traces from the HOT exploratory mission in Senegal more on that in a series of blog posts, I had found one GPS trace that has 2 problems that I've occasionally experienced while collecting gps traces for OpenStreetMapL

  • The GPS signal is temporarily lost and the distance between 2 points of your trace is abnormally lengthy, and cuts across roads and features.

An example: Did I time-travel across that area ? Nope.

  • And what I call the 'scribble effect' (is there another name for it?), where I accidentally forget to turn off the gps after I arrive at my destination and end up with multiple points near each other, looking like a child scribbled in a coloring book.
    An example:

If you're planning to make any visualizations with your traces, these two symptoms allow viewers to misinterpret where you've been and make your visualizations inaccurate.

The scribble effect also negatively affects OpenStreetMap because GPS traces that are uploaded in the future in the same area as your scribbled GPS trace, other users will be able to not be discern the newer gps trace from your scribbled gps trace. There's also still a few places on earth (including some regions of Senegal) that don't have any satellite imagery to (legally) use for drawing ways for OSM.

So, after you complete a trace, how do you eliminate these symptoms described above?

I've tried a combination of different filters with gpsbabel - explained more here

but have yet to find a set of filters that would solve both issues.

I've tried a few and here's the results:

gpsbabel -t -i gpx -f gpx_louga_03062012-withmay.gpx -x track,pack,sdistance=0.6k -o gpx -F test7.gpx - this had fixed the temporary signal loss ... but it didn't fix the scribble effect.

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,fixnone,fixunknown -o gpx -F test.gpx (resulted in a blank GPX File).

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,fixnone -o gpx -F test.gpx (no effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x position,distance=3m -o gpx -F test3.gpx (this removed the long points between the 2 segments, but didn't solve the scribble effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x discard,hdop=5,hdop -o gpx -F test6.gpx (no effect)

gpsbabel -t -i gpx -f gpx_louga_03062012-withway.gpx -x position,distance=10m -o gpx -F test8.gpx - removed 1100 of the trace's 2400 points and the , but still left me with the scribble effect.

I'll continue to explore cleaning up the GPS traces with GPSbabel filters and hope to find a way - and I'll update this - but if you know of a specific filter in gpsbabel or any other way to do this, besides manually deleting the offending points in gpsprune, I'd appreciate it.

Comment from BCNorwich on 19 September 2012 at 06:23

Hi,

You can edit the file in GpsPrune which you are using, don't ask me how as I don't use it, (GpsPrune HowTo says you can edit). You can edit the file in JOSM before uploading it to OpenStreetMap. I have a Garmin GPS and edit my tracks (gpx files) in Garmin MapSource (I remove track section directly to my house and trashy sections recorded when the GPS is first getting a signal.

I'm now going to look at GpsPrune, it seems to be a very neat piece of software.

Regards Bernard

Hide this comment

Comment from schnitzelinmypocket on 19 September 2012 at 06:39

I have similar problems and I'm assuming since a lot of people use the Garmin Vista HCX or similar models that do not store number of satellites,hdop or another quality indicator there should be others that have to deal with the same issue.

As you described, getting rid of the temporary signal loss can be accomplished wuth gpsbabel, while the scribble thing is harder to come by. I've been removing these manually (with GPSprune - but there are quite a few tools out there).

Something automatic would be nice too. Besides the scribble effect, from time to time the signal is picked up partially and the location wanders off in a straight line sometimes several hundred meters (can be seen in the lower image, once towards 11 o clock, once towards 4 o clock.

Hide this comment

Comment from Jean-Marc Liotier on 19 September 2012 at 11:05

No amount of automated processing will get rid of the scribble effect, but a simple manual edit will solve the problem : just cut the trace at the arrival point and delete the whole heap of spaghetti. Same with all the places where the GPS bearer has halted and produced a small heap of spaghetti. I usually do that sort of processing with Viking, but JOSM is fine too.

As for the segment with lost signal : delete it and split the way in two parts.

Hide this comment

Comment from bri g on 20 September 2012 at 21:08

Yep Standard problem - leaving the Garmin on after a hard day, a thousand data points of your desk with a Gaussian distribution of noise roughly resembling a drunken walk - easy fix.

I load all my traces into JOSM with the GPSEdit plugin.

Remove the 'blobs' of data points where I have stopped, or entered a building or whatever, all the obvious noise.

Then merge all the layers down into one and upload as a single GPX

Takes minutes, 'intelligent' filters are unlikely to work so well or so efficiently

Cheers bri Garmin Legend HCX Set to log once / second

Hide this comment

Comment from skorasaurus on 1 December 2012 at 17:17

After a bit of experimenting, I found the best workflow was to use the test3 filter (mentioned above) and manually removing the points in gpsprune (from the scribble effect) is the easiest way to clean up your GPS points.

Unfortunately, there doesn't appear to be an existing automated way of removing the scribble effect.

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment