Recent diary entries
This week, I created a new MapRoulette challenge, called "Massachusetts Schools".
Thank you Martijn van Exel, and Serge Wroclawski for making it possible (even easy) to add in your own challenges to MapRoulette. I used the loader python script, which requires a postgis database. It was just a couple of hours of fiddling with the SQL statement to make it work. I still hope to make small adjustments to the help and instructions text.
In Massachusetts, most of the schools were imported twice. Once from the national GNIS database import, and a second time from a statewide MassGIS data set. Both imported data sets are old and are getting stale. A surprising number of schools have moved, closed, or changed names since the data was imported. The schools need some attention.
Hopefully over the next 2 years, we will get them cleaned up.
We also plan on using this challenge in our local OSM OpenStreetMap-Boston meetup
What is OSM?
A crowd sourced, free map of the wold, containing information that can be verifiable by ground surveying. What about administrative borders? Probably don't want them removed ....
A crowd sourced, free map of the world. What about imports? We are up to our ears in imported data. That ship has sailed.
Lets not make things more complicated than they need to be.
The best free map of the world.
This is a bridleway (horse race track) that I traced out last year. The bing map is the stage/MassGIS data. I remember the same road line in OSM.
You can see that Google has traced out the same path I used through the parking lot, and incorrectly traced out the bridleway as a two lane road. Google map maker says it was made by "Google Automated Data Quality Improver 1" on July 24th. The nodes are not in the exact same places, I bet they are monitoring OSM, and setting up tasks for the 7,000 strong mapping army they employ to trace new segments by hand when something new pops into OSM.
Hey google, you should probably not copy highway=bridleway from OSM as roads.
Some software to pick out road signs from video.
The company is out of Russia. Any Russia OSM'ers have a contact with itseez? It would be cool to use this software for OSM.
There has been a lively thread on the talk mailing list about adding "consumer" features to the main openstreetmap.org page. This is really about adding routing, clickable points of interest, and geolocating support to the top level home page.
Some people, including our Chairman, Simon Poole, worry about several issues
- Diluting our energy on a non-core activity
- Diluting our web design, making the main page less functional for core mapping activities.
- Competing with our partners and downstream data users, making them less likely to work with us.
- Trying and failing, hurting our brand. It is a hard task.
- Trying and succeeding, driving zillions of people to our front page. The increase in scope will require a professional organization to manage. This is also known, as let’s not look like Wikimedia Foundation, with 176 employees.
Start with first principles, our mission statement: provide free geographic data, such as street maps, to anyone, for the entire world. This is a big goal, just a tiny fraction of humanity lives in places with a good OSM map. We need more people, a lot more people. If we want the entire map to looks like our gold standard, Germany or London, we need around 35 million mappers. By that metric, we are only about 2.5% complete. Another metric is Wikimedia, which currently has 20 million named user accounts. The mission statement is not going to be fulfilled, if we don’t have enough people editing the map, period. The best way of getting those mappers, registered, and mapping is for us to provide a service to allow people to use our data for their day to day needs on our site, with the big fat “edit” button on top. Our third party data consumers will not put that “edit” button on their interfaces. There is no benefit to them to dilute their interface to help us. Don’t get me wrong, I am very happy that people are using our data on other services such as apple, motion gps, craigslist, 4 square, etc, however, they are not going to carry our water for us and hand over our next 34 million mapper. We have to go get them ourselves. I don’t see any way of getting 34 more million registered mappers unless our site is can be used by everybody for normal mapping activities. So sorry, the current mappers are just the first 2.5%, getting the next 97.5% onboard is more important than serving the first 2.5%. To say otherwise, is giving up on our mission statement.
Our second principle is that we are a “do-ocracy”. When somebody shows up on the lists asking for routing, clickable POI, location service, mobile support, etc the only acceptable answer should be to ask them to help do it!
- If they can write code, do that,
- If they are system administrators help with that.
- Volunteer to test out new code.
- If they are good at raising money, get some money.
- If they can get some big servers donated, do that, hosting, donated bandwidth, do it, do it, do it!
- or the simplest way to help is to go work on the map and be as supportable as possible of the people that are working on these features.
The reality is that getting routing, or good POI support, mobile, etc are all very tough jobs. When they launch, we should expect them to not be competitive with the commercial providers. To be blunt, we are going to suck at it for years after we launch. The probability of everybody crushing our website the day, week, or the year after we turn on routing is exactly zero. However, this is how software is developed; without shipping something crude first, it is impossible to make something great later.
Please no more “no’s” on this topic.
There was an import of lakes/ponds over half of Massachusetts in 2010 that had no conflation logic. It ignored the lakes/ponds that were already mapped. Since then, OSM has had several hundred overlapping bodies of water all over Massachusetts. Last night, the last one was cleaned out.
I am very happy that the job is completed now!
I have been playing with osmosis/postGIS this week. I was finally was able to import the MA OSM extract today using osmosis.
It took a lot longer than using osm2pgsql and I was very surprised at how large it got when it was imported.
massachusetts-latest.osm.pbf - 205 Megabytes massachusetts-latest.osm - 5 Gigabytes - 25 time (it is an XML file) PostGIS snapshot db - 10 Gigabytes - ???
It looks like PostGIS/osmosis is not noticing that we repeat the same tags, over and over, and over again in the data.
The Massachusetts building import was completed this week.
Massachusetts has virtually all of its structures mapped in OSM now. 2.1 million structures were imported.
There is an address data layer being released this fall from MassGIS.
Hopefully we can get addresses on all of those building this year.
When you log onto the OSM web site, you get this really nice map of all of the users hometowns, click on the bubble, you get the user name. This is a great! Disappointingly, I have discovered at least 80% of the "users" nearby are in fact people that made an account but never managed to make a single change on the map. Also, many of the accounts are several years old.
I propose that OSM should delete any account that is over a year old and does not have any actual map edits. The account is dead.
This will make the "find nearby users" feature signal to noise go way up and will keep everybody honest about the actual size of the OSM community, which is probably 20% of the registered user count.
Last week the Massachusetts mapping department (MassGIS) released data for all of the buildings in the state.
Since it is likely that next year we will be importing this data into OSM to support addresses, I figured it would be useful to convert the files to OSM format and share them.
The data is here.
Each town in MA has its own ZIP file. In each zip file is two OSM files. One OSM file has all of the buildings in the MassGIS data set. The second file has just the buildings that are missing from OSM.
If you are mapping in MA, please take a look.