Hello everyone I’m starting a OSM themed podcast called Nodes and Ways. I’ve got episode one out and episodes 2 and 3 are in the works now. It’s on YouTube and LBRY and you can get the RSS feed at nodesandways.com . It will be in more podcasty places like Apple podcast and Spotify in the future.

Ill be doing interviews news around mostly OSM. If you have any ideas leave them in the comments.

The audio on episode 1 is kinda rough but future episodes should be better.

Today I would like to share what I have learned about using external data in OSM but not in the classic import fashion.

I have gone over what I have learned about data importing in OSM after being involved in a few of them. But today I want to talk about how we’re using data from our local government as a reference but not using the usual import method and why.

So what is the definition of an import in regards to OSM? The Wiki defines it as “the process of uploading external data to OSM.“.

But uploading is not always the most effective way to use external data to improve OSM. What we are doing is showing external data as a layer over OSM and using that to edit OSM by hand.

But Why?..Wouldn’t it be better to direct upload the data?

Not always. We have found that using other people’s data for OSM comes with a lot of hurdles and direct importing can do more damage than good, but that’s no reason to give up on using the data. We just change methods to make it work. As I said in my last post, it’s important to get governments and organizations contributing to OSM, so we want to find a way to make use of data that is not suitable for direct import into OSM. As you can see in the picture above there is a layer that we can use to trace over. But you also can see that the subdivision is split into 3 blocks. This is probably due to the fact that it was built in that order. In this data set we see this concept referred to as sections, blocks, phases, and numbers like 1,2,3 smaller sections inside of subdivisions like “commons” and “place”. Naming can be somewhat unpredictable. There are also other things in the data that aren’t great for us, like plots of land that have the names of the development company or name of the owner, things that are not ground truthable so they need to be left out. It’s important to keep in mind that the people collecting this data were not doing it for OSM, and it’s the responsibly of the people bringing in the data to bring it up to local OSM standards.

But couldn’t we figure out those variables, account for all that, then do a normal import?

Well going through all the trouble of merging and fixes errors in the data takes longer than just fixing it on the fly. Plus after we have finished doing it by hand, everything is to our local OSM standards, and we have a good place to measure against and track further progress as we automate the process. Also we find many places where we use different tags and draw the lines differently than the county GIS department, so doing it by hand lets us make those judgment calls on a per area basis.

So is it an import or not?

Strictly according to the definition on the Wiki, No I don’t think so. However were still using external data to improve OSM so it is important to get permission, document what you’re doing, track the work, and give attribution. Even though the actual edits are coming from our mouse clicks and not automated software. I do think that there should be direction on the wiki for this kind of “reference” work where you get many of the same prerequisites and plan of action to input the data as an import. Both to make sure the data going into OSM is legally allowed and to provide guidance on what tools and methods to use. I’ve seen other structured tasks like this called Map-A-Thons and campaigns on various sites around OSM. I think the term “Campaign” fits well, and having documentation and tooling and best practices around this concept would be a good thing for mappers who want to make use of data that’s not in good shape for direct importing.

You can look at the code and follow along with the residential area import that were working on now at the links below.

https://github.com/OpenStreetMapSC/SpartanburgSubdivisions

https://tasks.openstreetmap.us/project/143

https://wiki.openstreetmap.org/wiki/Spartanburg_County_Subdivision_Import

https://wiki.openstreetmap.org/wiki/Import

So as you know from my first two posts I have Microsoft building foot prints and county address data, two good sources of data. Both worth a import on there own. However we want to make the most efficient use of our mapping time so Mike came up with a way to conflate the data sources. And save a lot of time! Importing data into OSM has a good bit of overhead (still worth it). As I’ve talked about before you have to go get the data, figure out how to process the data with a procedure for Tracking, coordinating, and QA. Then once you figured out all of that you have to have local community buy in and go through the mailing list where your proposal will be vetted in a public way. So where it makes sense, it is worth thinking about using your available data sources in conjunction with each other to minimize going through that over and over. Now it does come with some additional risk, your proposal will face more scrutiny the more complex it is. And simpler single imports might lend them self’s to more mappers helping you with it. That’s a judgment call you will have to make. Im going to stay fairly high level in this post if you want to know the nuts and bolts of everything there is great documentation that Mike wrote on the GitHub https://github.com/OpenStreetMapSC/MergeAddressesAndBuildings and https://wiki.openstreetmap.org/wiki/Spartanburg_County_Address_And_Building_Import

And here is a screenshot of where we are at right now with the work to give you a idea of how it looks while in progress.

Here is a section of map before the building import

And after

As you can see adding in this data instantly makes the map more useful to most people. When I first show someone OSM they always want to see where they live first, and now we can show them there address. The second thing people always want to see is where they work but ill cover adding POI’s in a future post. Another benefit of covering the map in this way is that we are constantly running validation on the area we are working on for each import. So even if im not looking for a certain issue it will be seen during the QA process so you end up fixing a lot of existing issues with the map that are non import related. All and all we are making the map more useful to more people by adding data and improving the quality of existing data. By defining a area to work in, and finding all the available and applicable data sources we are organizing to Make OSM to be as useful to as many people as possible with the time available to us as local mappers, and we are making measurable progress :-)

This might be handy to anyone who gets a waynes camera as a mounting idea.

https://archive.org/details/VID20180523172816

Here is a post I made about making a 360 rig for use with Mapillary

https://archive.org/details/LG360CAMMountGuide

Ok! I’m on board, let’s use some data for OSM!! Wooo Yeah!!! Where’s the data?…

I spent a little over a year trying to get access to data, once I decided that was what I wanted to do. I now have permission from the local county where I live, some other county’s and some state data. And WoW! We now have more data to work with than we have time and mappers to processes it. It’s a good problem to have. So how did this happen?

This was my struggle and why it was worth it.

First a little back story. I’m relatively new to OSM, I started mapping in 2016 after a series of events led me to get fed up with Google Maps and I said to myself “I’m just going to figure out how to make OSM work for me”. I’ve been proponent of open source for much longer than that, but maps was one of my last hold outs where I said it’s just too hard to switch. After Mapping on my own in my home town for a while I met Mike who had commented on a few of my change sets and I had been talking to via the OSM pm system. When we first met, I asked “Can we use government data in OSM? That seems like a logical thing” and he told me in his nine years of working with OSM he has tried over and over to get permission to use the data but has mostly been ignored. Well me being a new mapper I said “Ok I’ll just let it go”, but it was like an itch that would not go away. Why can’t we use that data? It’s up on the county website and I can download it right now, if they go through the trouble of making it public facing why ignore us when we ask to use it? So I said “Mike I’m going to try and get the permission to use county data” Mike said OK go for it, and showed me some template letters that he had sent to county officials. So I took the same template letters and customized them to my liking, and sent them in!! aaaannnnndddd nothing… Not one response. I waited and said the wheels of bureaucracy turn slow so I will have to be patient. Just FYI I’m not super patient, it’s part of my personality I’m working to improve. After all, I’m trying to make OSM the best it can be so I can use it in my everyday life, I got no time for standing around! So I called my county GIS office and after talking to a bunch of people who did not know what to do with my request I finally got a hold of an official who said “I would like to help you but that has to go through the attorneys”. Great! That was the direction I was looking for. I took my letters and sent them into the attorney’s office aaaannnnddddd!!!, nothing, nada, more getting ignored. I called in, left messages. Once I got a secretary who happened to be standing beside the attorney when she picked up the phone, and asked him while on the phone with me then told me that “He has seen my request and will reply soon” That got me pretty excited, finally some communication from someone who was in charge, Great! Thhheennnn nothing, more weeks and months of waiting for nothing. I called back in to GIS a few times and asked for advice or if they could ask the attorney in my behalf. They were starting to get to know me at this point since I kept calling in and they liked that someone thought their data was really important and wanted to use it, a lot of people are apathetic to this kind of “boring” government data collection. Then after over a year of this I got a break. The GIS head was retiring and said “I’m going to write a letter giving you permission because I think you should have it”. Now you might think this is just coincidence (Ok, great but my GIS head is not about to retire) and you would be right. However! I had been building that relationship for over a year. Walking the line between being consistent and being a pest. And after that the next county over with whom I had been going through the same ordeal (but getting even more silence from) with gave me permission to use there data as well. It took over a year of emails, calls, civic websites, Facebook messages, learning more about legal processes than I ever thought I would. But I only needed to get the permission once. Also during this process, Microsoft open sourced their first batch of building data and I will never forgot adding 40,000 buildings in Greenville SC to OSM in a night. After hand tracing my entire town I was like “WOW that was easy!” They even had height tags!

Microsoft has since released building foot prints for a lot of other places and if you are waiting to get other data I highly recommend starting with this data as is available for many places and a lot of people are also doing it so there is lots of information on how to do it. (shout out to Microsoft, they started MapRoulette challenges and found some data we could use so we did not have to)

So, was a year of spending time beating my head against the wall, trying to get a response and not be ignored, time that I could have just gone out and collected data myself, worth it? Oh Yeah! We went through and edited every road in the first county in less than two months and I can tell you that would not have happened otherwise. There are people who keep up with the interstates and major highways in the US but small roads in our area have changed a lot since the Tiger import. I can tell you in our area right now we have the best roads map. That took us in Spartanburg County roads from worst to best in less than two months. And we are working on a rolling update and review process to keep it that way.

Other nice things about this approach

1. we look at our area as a whole not just the places we go a lot or like, so we map more evenly. Places where we live and go are mapped really well but that town “over there”, you know the one no mapper lives and I’ve never been to. Those kinds of places have not received much attention. But if it’s in decent shape, it might pique someone’s interest and, Boom! We now have a mapper in that town. Now I am not saying we need to map every little town to get new mappers? No. I started Mapping my town on my own with very little to start with. Like I’m sure many of us mappers have. But, what I will reiterate from earlier is that there are a lot more people who are willing to improve a map that basically meets their needs than people who are willing to make a map meet their needs.

2. We can define a road map(pun intended), measure progress, and iterate to improve how we do things.

3. When places donate data and see that it is used, they are more likely to contribute more.

In this blog I will talk about what data we are using, how we are processing the data, then importing it into OSM. Some things are not considered imports at all since they’re more of a stare and compare, but I will cover them as well since we are using data others collected and have made available to us for use in OSM. Even analyzing data that is already in OSM to improve data quality. All that and more! :-)

First off, I understand many people do not like using others’ data for OSM for a couple of different reasons.

1. The data could be low quality. I once talked to a mapper when I saw that they removed an imported move theater from the middle of a corn field, due to someone blind importing. …Ouch!

2. Imports require a lot of technical knowledge of OSM and that excludes a lot of mappers.

3. Making sure the data is not copyrighted in a way that prevents it from being used in OSM. (I’ll touch on that in another post) There might be more, but these are the ones that I have heard so far.

In this blog I hope to address these concerns by demonstrating how we can set things up in a community-oriented way that streamlines the process and also keeps quality high. We have so many tools to at our disposal that I keep stumbling across them and finding new ways to use them for different projects! I hope to gain working knowledge and demonstrate how they can all work.

Also, I will be talking about how I got access to data for our local area and tips on how you might be able to do the same. That can be half the battle!

So far using data from other sources has enabled our small local community to begin to get the map to the point in our area where it is becoming a more preferable map to more and more people, which we have seen is starting to drive interest from prospective new mappers. As I have seen there are a lot more people who are willing to add to a map that already basically meets their needs than are willing to get the map to the point where it would be useful to them. I have even seen this among people who understand the value of open source and want a map that they can use without being tracked, but just don’t have the time or inclination to become mappers themselves. If I cannot get these kinds of people on board that makes me nervous!

So, all that being said, I think getting and working with data is the answer. I do not think we will, or should ever stop in-person mapping or armchair mapping. But in a place where there is a small mapping community getting organized and using, not repeating work that has already been done by others will make OSM the best it can be.

Also by aggregating data from more and more places we will ensure OSM is the standard for Geo-Spatial data. I would like to see companies and governments in our area be invested into OSM as a platform.

I am in upstate South Carolina, USA so everything I do will be with our area, and I doubt that will change as I understand that other places do things differently and I really am only an expert on where I live. But I hope you can use, adapt, improve and get inspired on how to improve the map in your area.

Because hey that’s what open source is all about: sharing ideas.