OpenStreetMap

San Francisco data imports, anyone?

Posted by Alan on 7 July 2015 in English (English).

Thanks so much to the Mapbox Data Team who traced all the building footprints in San Francisco, California last year!

However, I think it’s time to start giving our buildings the next bit of love: addresses. I checked and found a stagnant import proposal from 2010. Maybe it’s time to reboot that? If you’re interested in importing addresses for San Francisco, join in the conversation on the San Francisco Address Import wiki page

The other thing that our buildings lack is height data. Perhaps we could import that too? I started a page for that discussion here: San Francisco Building Height Import. Feel free to let me know if that’s a terrible idea.

Location: Mission District, San Francisco, California, 94110, United States

Comment from nvk on 7 July 2015 at 05:05

Great idea!

Comment from lxbarth on 7 July 2015 at 15:12

I love this. The building dataset for SF in OSM is one of the best out there, it would be great to add building heights right into it.

Comment from woodpeck on 7 July 2015 at 22:56

Please don’t use the phrase “to give love” when you’re talking about writing a script to import government data into OSM. There are cities the size of San Francisco where mappers have surveyed (a sizable portion of) the addresses in their (unpaid) spare time. These cities have received love. What you’re planning to give to San Francisco is just an import of someone else’s data which is going to sit orphaned in OSM, devoid of love and care. Do it if you must, but don’t call it love.

Comment from Alan on 7 July 2015 at 23:21

Good point, Frederik. I’m keenly aware of the different points-of-view within the OSM community regarding imports.

I meant the phrase “to give love” to be taken in a playful and colloquial manner, but I can see that’s its unnecessarily incendiary given the tensions surrounding imports in OSM. I’m sorry about that.

Comment from pnorman on 8 July 2015 at 07:25

California needs a lot of import attention. But the attention it needs is not more imports, it’s cleanup of existing imports, particularly landuse imports.

Comment from SOSM on 8 July 2015 at 15:45

And to not forget that love could mean fixing glaring problems before somebody multiple 1000 of miles away has to do it …. http://www.openstreetmap.org/changeset/29687035

Comment from bbq on 8 July 2015 at 18:59

Greetings from right here at City and County of San Francisco. The past week I’ve been extracting just buildings, flattening building polygons, and have a summary of ground and first-return elevations for each building.

City and County would like to leverage OSM excellence, yet OSM licensing restrictions make it not feasible for us to use these OSM shapes as our own public reference set. We’re keenly interested in having buildings that contain their addresses along with elevation and height; we don’t have those just yet.

We have reference surfaces for ground and for LiDAR first return prepared at 50cm gridding, but the ortho-derived alignment of many nicely shaped OSM buildings is presenting challenges for their height assignment.

I’ll add more in the individual effort wiki pages.

Comment from AndrewSnow on 9 July 2015 at 19:04

I would like to ask that this be very carefully considered before proceeding. I have done a lot of work to micromap business districts and homes in SF, particularly in the West Portal - Inner Sunset areas and SOMA around 8th and Folsom using Vespucci in the field and the City’s website for visual reference.

I foresee a number of SF specific issues with an import:

  • While the Mapbox drawn building outlines are a big improvement over those available from CCSF, they are not particularly well aligned to Bing. In general, I often need to move entire blocks 10 to 15 feet in order align them to the image reference we are generally expected to use.

  • The Mapbox team frequently combined adjacent flat roofed buildings into a single building. When I am adding addresses, I first find I need to sub-divide the buildings in order to represent the condition on the ground.

  • San Francisco buildings are narrow by California standards (typically 25-30’ widths) but will often have 3 or more addresses within that width. Small spatial errors either the address database or in the Mapbox building outline are going apply the wrong address to many, many buildings.

  • The SF Planning address registry has addresses that have no physical presence on the building. Likely these were assigned to suites that have now been combined into larger spaces.

  • San Francisco OSM already has a lot of addresses. It’s in patches and pieces, but aside from the neighborhoods I have worked on, there are large sections of the Castro, Noe, along Church and in strips of downtown, and along Geary that are done to a high level of detail. You can see this at OSM Inspector or using the QA tool.

  • SF does suffer from a lot of drive-by mapping where folks enter a housenumber, but nothing else. OSM Inspector shows these too - I cleared several dozen of these a few nights back. Several have multiple driveby entries a few feet apart for the same address. If we import, we are just going to add another layer that needs to be reconciled with the existing problems.

Instead, what I have been doing on and off for the last couple of months is to enter house numbers from the SF planning website and then hit the streets with Vespucci for the business districts and where the website records are ambiguous. I started at West Portal Boulevard and am working a strip northward toward GGP. It’s the most tedious thing I’ve done on OSM, but I have managed to get ~1400 addresses entered this way, and I have high confidence in the results. You can see the small dent I am making at either OSM Inspector or qa.poole.ch.

What I would love to see done with the SF address shape file is to produce an OSM/GPX layer we could open in JOSM to view while manually editing. The cursor in the CCSF is too large - you frequently get the result for the adjacent building - having this same information available in the JOSM edit window might speed things up tremendously.

By contrast, I have thought that many parts of Marin County would be ripe for import. The Marin County GIS building outlines are excellent, the degree of tree cover in the neighborhoods makes digitizing them from Bing mostly impossible, and there are large areas where there have been no local addresses or buildings entered (last I checked).

Comment from pnorman on 10 July 2015 at 19:16

City and County would like to leverage OSM excellence, yet OSM licensing restrictions make it not feasible for us to use these OSM shapes as our own public reference set

Just to clarify, it is city and county licensing policy and would likely impact any foreseeable license.

Comment from lxbarth on 20 July 2015 at 14:39

While the Mapbox drawn building outlines are a big improvement over those available from CCSF, they are not particularly well aligned to Bing. In general, I often need to move entire blocks 10 to 15 feet in order align them to the image reference we are generally expected to use.

Andrew - note that Bing imagery is not the authoritative image reference for OpenStreetMap as it can be off in bad cases by meters. In the San Francisco tracing effort we have used existing GPS data and the LIDAR derived SF public domain building dataset as location reference.

http://wiki.openstreetmap.org/wiki/Good_practice#Align_Aerial_Imagery_before_Tracing

Comment from Yury Yatsynovich on 25 June 2020 at 14:56

Greetings!

Please, find a link to an archive with my code, inputs and resulting “open_in_JOSM_and_upload.osm”-file below: https://drive.google.com/file/d/14ECmeyQN8HMrITfcsa3Nw3BfAv8Fi687/view?usp=sharing

Despite the name of the resulting file – “open_in_JOSM_and_upload.osm” – this file has attribute [upload=’never’] which should prevent accidental upload of it. But, just in case, let me make it explicit: THE FILE “open_in_JOSM_and_upload.osm” IS PRODUCED ONLY FOR REVIEW BY OSM COMMUNITY, IT MAY CONTAIN SERIOUS ERRORS, PLEASE, DO NOT UPLOAD IT!

The archive also includes a README .pdf-file that describes in simple language my approach to matching address points with OSM buildings.

The script file, “import_sf_addresses.py”, should run in a standalone mode (like “python3 import_sf_addresses.py”) if you have all required libraries installed, but I ran it step-by-step.

Should you have any questions, I’d be happy to reply. Looking forward to receiving your feedback!

With kind regards,

Login to leave a comment