OpenStreetMap

Ugg I screwed up

Posted by seattle-buildings on 19 May 2012 in English (English)

Off to a horrible start (http://www.openstreetmap.org/browse/changeset/11638818) and I'm not sure if it's my fault or not (user error or technical issue?). Uploading set 1 just stopped at chunk 46 of 50 (chunk size 1000, too low?). It was going slow but ticking along until #46, which I let it try to upload until the changeset idle closed. Now all the nodes were uploaded but only a few thousand of the ways. I was trying to revert the changeset or at least download the objects and salvage some of the import but no changeset downloads are working (http://www.openstreetmap.org/api/0.6/changeset/11638818/download). Finally I was able to grab everything from Overpass (http://overpass-api.de/api/xapi?*[@user=seattle-buildings][@meta]) but now the challenge is filtering out everything that has already been imported from what I have left of this set, about 1800 ways and trying to merge them. So I got everything back into QGis and deleted all polygons that intersect the successful upload. This leaves me with unique polygons from my set that were not uploaded. Now the problem is getting the existing nodes that did upload connected to these polygons. This sounds simple enough but I can't find anything that preserves that data, the osm_id and whatever tags would be associated with it, everything seems to treat one node like any other nodes, discarding any possible data. JOSM isn't any help here, I tried it out and it just wants to upload new nodes on top of the olds ones.

I think what I'm left doing is deleting all unclosed nodes (19300) and using my resulting qgis selection of unique polygons (that didn't get completed in the upload) to try again. The original set was ~49000 nodes or 4700 buildings so it wasn't a complete wash (2988 got up fine) but not something I want to repeat. I think I'm going to do smaller imports next time, I'll probably just split my existing sets in half to 25000 objects.

I'm going to sleep on it to try and come up with a better solution and get cracking on this in the morning.

Location: 47.599, -122.330

Comment from AndrewBuck on 19 May 2012 at 16:02

Josm has the validator tool built in which can fix duplicate nodes. If you load the existing osm data that got uploaded and then your data on top of it it should detect and be able to fix the duplicates.

I am not sure how it determines whether to keep the existing nodes that are already in the DB or if it just picks one at random to delete.

Also in case you haven't already done so, you should send an e-mail to the imports mailing list. This is supposed to be done before you do any importing at all so I would do this before you do anything else. They can also probably give you better advice on how to recover from this.

-Buck

Hide this comment

Comment from seattle-buildings on 19 May 2012 at 17:23

Hey Andrew thanks for commenting, I completely forgot about the validator. It's running through everything now but looks like it's going to take an hour or two. It seems like it has preference for existing objects so that's good. Now I'm wondering if it's going to preserve the ways if all the nodes that connect the ways are removed as duplicates, we'll see.

I did a quick test of 100 buildings to see how fast I could manually connect the nodes and it only took about 15 minutes, so worst case I'll connect the other ~1600 by hand if the validator doesn't work.

I'll let the imports mailing list know after I finish this up, I don't want to alarm anyone with this problem which seems to be under control.

Hide this comment

Comment from pnorman on 19 May 2012 at 20:26

Could you provide a link to the imports@ and talk-us@ mailing list discussion that happened before you started this upload? I can't seem to find it.

Hide this comment

Comment from Glassman on 20 May 2012 at 00:07

Let me know if you need help manually fixing the nodes. Can I suggest a smaller test next time?

Also, I have a list of businesses for the downtown area. Would be nice if we could work together to add know business to the buildings.

Clifford aka Glassman

Hide this comment

Comment from compdude on 22 May 2012 at 16:19

Hello, as a fellow Seattle mapper, I welcome you as a new OSM user! Thanks for your help with uploading buildings in Seattle, saves a lot of work for the rest of us! Of course, there's already some buildings there, so make sure to avoid uploading buildings which already exist on the map in order to avoid duplicates. :)

Thanks for your help!

-Compdude

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment