Recent diary entries
This is an update on the ongoing import of New York City buildings and addresses. For background read up on New York City and OpenStreetMap cooperating through Open Data
We have taken the time to take a close review of existing uploads. Here are some issues we've found that are worth highlighting as we restart the import.
- Make sure every upload to OpenStreetMap completely validates and all critical warnings are resolved before you update.
- Critical warnings are at least any warnings or errors that stem from
- Buildings overlapping with buildings
- Buildings overlapping with other features they cannot overlap with such as roads
- Resolve not only buildings duplicate with existing buildings but also addresses duplicate with existing addresses
- Merge point of interest information from existing nodes to new buildings when they clearly building-level such as schools, fire houses, super markets, etc.
To get started head over to the tasking manager carefully (re) read instructions and grab a task.
Make your life easier and get these JOSM styles for buildings and addresses by emacsen. They'll allow you to see issues with the data better. Learn how to install them in JOSM docs.
If you have any questions, fire away here on the comment thread.
On the imports list I recently raised the question on whether to tag addresses on buildings ways or not. Specifically, if there is only one address for a given building polygon, should the address tags sit on the building's ways or should the address tags sit on a separate node within the building? Obviously, if there is more than one address per building, there is no other way but mapping them as nodes separate from the building way.
Eric Fischer just ran an analysis to figure out what is actually the current convention in OpenStreetMap. Here's the short answer: addresses are tagged on building ways where possible. By a wide margin.
Read on for the numbers.
Address tagged on building ways (left) is the more common approach in OpenStreetMap versus address tagged on a separate node (right).
The rough numbers break down like this:
- 10 million buildings carry addresses on the way.
- 3 million buildings contain one or more address nodes.
- 4 million address nodes sit within a building.
So the maximum theoretical number of buildings with a single address node is 3 million minus 1. Contrast this with 10 million buildings with the address information on the way. This still assumes one crazy building containing one million address nodes and it does not discount redundantly tagged addresses in the case of POI nodes that duplicate the address of the building they sit in.
Here are the full numbers (OSM planet, September 25 2013):
Buildings 91,917,857 Buildings with address on way 9,386,811 Buildings that contain one or more address nodes 2,960,363 Address nodes within a building 3,858,096 Address nodes that are not on or within a building 10,135,036 Addresses on a node of the building way 673,975
(An address node is defined here as a node that contains an
addr:housenumber tag and is not part of a building way.)
Good or bad?
For now, I'll personally stick to this convention as it's established. For the same reason I also want to stick to it for the ongoing New York City building and address import.
In principle though, I question tagging addresses on building polygons. It's a special case with no benefits while separate address nodes would work in both, the case where there are multiple addresses per building polygon and where there is only one address per building polygon.
Last Saturday we officially kicked off the NYC building and address import with a community session hosted by OSM-NYC and Public Labs at the Pfizer building in Brooklyn. The goal was to get the local NYC OSM community involved in this large data undertaking and at the same time harden our import process.
Over 20 people attended, and we knocked out 158 of the over 5000+ sub-tasks total. Both turn out and tasks accomplished were great and exceeded what I expected for a casual Saturday afternoon event.
Working through this import we're learning very interesting lessons:
- OSM data structure is significantly different from traditional GIS, nailing down conceptual differences when translating to OSM takes time.
- Importing is a high inertia problem, partly due to sheer volume but also due to the lack of a solid tool chain like safe roll back tools or established conversion tools.
- Expect interesting quality issues in your source data. NYC data for instance has inconsistent address formatting in the source.
- Doing a fully automated import is non-trivial. For example, in NYC, buildings often intersect with misaligned TIGER roads. That's one big reason this import is not fully automated.
- Once all data is uploaded, we'll need a QA check on inconsistent data to catch any errors introduced by humans during the upload.
- This all feels a little like heart surgery.
Here are a couple of pictures and screenshots from the Saturday event. If you'd like to get involved drop me a line. Again, the import is on hold until a couple of issues are sorted out, but you're welcome to join.
Because I love it.
Open data is changing the world. OpenStreetMap, as a true open data project of the commons, is proving its viability with continuously growing contributor numbers and expanding adoption. We're well under way to replace what has been historically the realm of governments and proprietary-data companies with amazing open data that is not better because it's cheaper, but actually provides fresher and more detailed data because it's open and community driven. It's exciting to be a part of this in my job as data lead at MapBox, as an individual contributor and as a board member of the OpenStreetMap US chapter.
This last year on the board of OpenStreetMap US has been amazing. Working together with Jim, John, Randy and Martijn on the board and with the great support of community members like Kathleen Danielson, Bonnie Bogle and Ian Dees, we've brought the organization to a new level. We're honing in on our goal to not only promote OpenStreetMap in the United States, but to make it bigger, stronger and more diverse. Here are some of the things we've accomplished:
- Ran the biggest OpenStreetMap conference to date, bringing together almost 400 members of the US and international community in San Francisco.
- Ran quarterly editathons promoting OpenStreetMap on a local level. Twelve cities are participating in the October editathon, and more than 1,000 edits came out of the last one.
- Relaunched OpenStreetmap US to do a better job introducing OpenStreetMap to the world.
- Connected with government agencies to promote the use of OpenStreetMap and explore areas of cooperation.
For the next year I want to stay 100% focused on continuing to grow OpenStreetMap in the US and beyond. I am convinced we can do this only by uniting the many voices of our project and by being as open and inviting as possible to newcomers. This is why it will be so important to nail our conference again. The importance of State of the Map US for the growth of OpenStreetMap cannot be overstated. It is the main tool we have to convene our community, pull in new individuals and institutions and discuss the future of the project. OpenStreetMap brings together interests from very different backgrounds: it's being used and improved by individual mappers, businesses, governments and nonprofits. Individuals work with it as data consumers, data producers, software developers, designers and researchers. This diversity is exactly our strength and is exactly what we need to continue to grow OpenStreetMap.
I would love your vote for pushing on this vision on the board of OpenStreetMap US.
OpenStreetMap US elections are running from October 5th to October 12th, to vote, you simply need to be a member of OpenStreetMap US. Signing up only takes a minute, find out all details about the election over on the OpenStreetMap.us blog.
OpenStreetMap colored by contributor id
There has been lots of talk about groups on OpenStreetMap.org lately. In early 2013 Mikel called for better social tools, including groups on OpenStreetMap, and lately more often groups have been mentioned as a replacement for our ailing mailing lists. Saman had a version of groups in his blue sky mockups for OpenStreetMap.org. Tom's posted a sketch for groups as pull request.
I'd like to add a dose of skepticism in this discussion: I don't think we should implement groups on OpenStreetMap.org right now, there are better alternatives to get started with if our goal is to make OpenStreetMap more social and let mappers connect better.
- Most conversations ideally don't require groups.
- It's hard to do social software right, groups in particular.
- Social media platforms are distributing.
(1) Most conversations ideally don't require groups
When you stop to think about it, groups are a crutch. They require you to set up a space with a topic and name (even if it's just a couple of clicks), then people need to find it, subscribe to it and sustain interest in the group. If the group doesn't go well, it bleeds members and lives on as a distracting zombie. Ideally, you'd be able to have conversaions ad-hoc around a certain topic or locality. That's one reason why you don't find groups at all or in a dominant role on some of the most successful social networks today.
(2) It's hard to do social software right, groups in particular
What was the last forum or groups software you used that didn't suck? Right. It's hard to do groups right on an interaction design level. I personally haven't seen general group discussion software ever done right, but what I do know is this: whatever we embark on means significant investment - or falling short on expectations. The risk to wind up with another level of noise in our already brittle social space is real.
(3) Social media platforms are distributing
Today OpenStreetMap enthusiasts gather in spaces on mailing lists, Meetup.com, Twitter, Facebook, forums, and Google Groups. Whatever we build competes in this space. Right now, we shouldn't attempt to build the better replacement for all of this, but think of OpenStreetMap.org as a compatible layer, allowing mappers to bring OpenStreetMap into their respective social online environments with ease.
Instead of introducing groups as a large new feature on OpenStreetMap.org I suggest we fix current social functionality on OpenStreetMap.org. This would vastly improve how mappers connect on a local and global level and would allow us to take an iterative approach, giving us real returns at each step, building on firm, well known ground. Here's a first back log:
- Great opt-out email notifications for edits, diary posts, comments of who you're following and posts you've commented on.
- Make it much easier to see who's mapping in an area
- Introduce public wall-style messaging, allowing conversations in the open.
- Ideally shut down private messaging to avoid abuse (which is happening according to administrators).
- This is small: Rename 'friend' to 'follow' - because that's what it is, no one confirms a friend request on OpenStreetMap.
- Kill the home location feature including the map on the profile
- Replace the useless friend listing and 'in your area' listing on your profile with a list of latest edits by who you're following
- Encourage users to link to local groups from their profile (Facebook links, meetup.com links, mailing lists links, wiki links, etc.)
- Possibly: vote up (down?) comments on diary.
Each of the above steps is small compared to implementing groups - still, each one will require dedicated work. Together they are designed to move us forward in a solid fashion from where we are right now. And note: some of the features like notifications could come in handy if we still wanted to introduce groups at some later point.
So what about our mailing lists?
Done right, the above improvements will already take important weight off of our mailing lists, we should iterate from there. With improved notifications and commenting on diaries we'll have much better spaces for meaningful discussions. I assume that much of our outcome oriented work will continue to move to GitHub. It's also going to be interesting to watch how Map Club will move in this realm. In addition, I suggest evaluating Discourse.org, a promising new discussion forum by Jeff Atwood the maker of Stack exchange.
What do you think?
Disclaimer: I am offering this simply as food of thought for those who're interested in pushing on social features right now. From a MapBox team perspective, we're not queueing up any immediate work on the social features mentioned here.
Finding out fast who's modified the map is hugely valuable to review changes in areas you care about, to connect to new mappers or to just show how fresh the map is.
Unfortunately, the part of OpenStreetMap.org that's supposed to provide this functionality - the history tab - is functionally broken. I'd like to suggest here a straightforward way to fix it, punting on some of the hard engineering problems that fast browsing of historic changes bring with them.
To recap if you're new to the issue, here's why the history tab doesn't work today: virtually anywhere in the world you'd like to see the latest changes of a particular area on OpenStreetMap, what you'll actually get is large-scale changes whose bounding box happens to intersect with with the area on the map you're viewing while not actually impacting any data in the area you're viewing.
The underlying engineering problem is Hard: changes to OpenStreetMap are organized in changesets, each one of which can contain up to 50,000 edits and whose modifications can be geographically huge. Querying all changesets that actually modified data in an arbitrary bounding box of the world and displaying them in reverse chronological order is computationally expensive while at the same time it should happen in milliseconds to satisfy a web request and allow for fast browsing.
Fixing the history tab
At the Chicago hack weekend Tom and I created a prototype that completely punts on the expensive problem of fast browsing for the entire changeset history. The approach we've taken is essentially to present you with a map and a list of the latest changes on visible elements first, then only reveal the history of an element when you click on it.
Conceptually, this is very straightforward and supported by existing APIs, computationally this is dirt cheap. This is not actually a novel approach in OpenStreetMap (most editors do something like that) but it is viable as an alternative to today's history tab.
The prototype doesn't do any data processing itself and is actually just a simple HTTP and JS application hosted on Github pages. It uses the OpenStreetMap API's map call. The latter means it is querying OpenStreetMap in an unefficient way, but this is a comparatively simple problem to fix. It could just as well query a very simple tiled data source.
Check out the result for yourself. I think it is already a very useful browser for exploring changes on OpenStreetMap. With few iterations this could be much faster and a viable fix to OpenStreetMap's history tab.
- Prototype: http://osmlab.github.io/latest-changes
- Codebase: https://github.com/osmlab/latest-changes
Related conversation on [dev] list can be found here: http://lists.openstreetmap.org/pipermail/dev/2013-May/026891.html
We just relaunched LearnOSM - the step by step resource for learning OpenStreetMap. LearnOSM was launched in 2011 by the Humanitarian OpenStreetMap team for the workshops they are giving world wide. It has grown into a resource used by OpenStreetMap newcomers and trainers well beyond its roots in humanitarian aid and disaster risk management.
Read up on the new LearnOSM and learn how to contribute to translations and other improvements by Jue Yang, the designer and developer behind the new face of LearnOSM:
Andy Allan recently ported the OpenStreetMap standard style from pure Mapnik XML to CartoCSS and TileMill. This is exciting as it's a huge step towards making contributing to the style more accessible. The port is nearly perfect and kinks are being worked out right now. I took a minute to spot check and pull together a couple of screenshots showing just how close this awesome piece of work is. I hope to see this go up soon on OpenStreetMap.org. Andy's port wouldn't change anything about how tiles are being rendered on OpenStreetMap, all that changes is how the style for those styles would be created: In the future, we'd use TileMill and generate the Mapnik XML from user friendly Carto CSS.
Here is San Francisco in the new OSM-carto port, just looks like the existing map:
For spot checking I use the comparison app that Ian and Tom cobbled together using OpenStreetMap US servers and bl.ocks.org. It shows the existing OSM Standard style to the left (I'll just call this style 'OSM' for the remaining post) and the new port to Carto CSS on the right (I'll call it 'OSM-carto' for the remaining post). Note: right now it's slow / down as performance problems are being figured out.
What follows here is a quick log from my review, others have been busy spot checking, too, head over to the issue queue to find out more.
School labels are different
School labels aren't bold in the new OSM-carto style. This has been fixed in the meantime.
Complex junctions seem to be rendering great with roads correctly rendered on top of each other and labels placed well.
OSM-carto has halo on secondary
OSM has no halo on secondary, while OSM-carto does have one in orange (see Castro street label) #25.
OSM-carto has no halo on tertiary
To the contrary, OSM has a white halo on tertiary highways where OSM-carto has none #24.
Label placement seems to be slightly different
This might be due to slightly stale data and generally doesn't seem to make a difference in terms of cartographic quality. In this difference rendition between the OSM and the OSM-carto style you can see how all labels seem to be offset by a certain factor and some street labels are placed at different positions along the way.
Low zoom levels
Low zoom levels look almost perfect at quick inspection. I do not know where differences in labels (see Sapporo for instance) come from.
Mid zoom levels are almost 100% the same
Differences in landcover order
There are some known differences in the order of land cover. This is being worked out right now in #15. Here is an example where the difference in landcover order surfaces in the visual result.
Stoked about test driving the great work of Tom, Richard, John, Saman and others on the new iD editor. Here's a quick screenshot. You can get started yourself with iD fast, just clone the repo, point your browser to the index.html file in it and get started editing. Note that if you'd like to upload your changes to the test server, you'll need an account on it. Report any bugs you're finding on the issue queue.
We had good turnout for the OpenStreetMap and MapBox workshops in preparation for Desarrollando America Latina.
The videos of the webcasts are up now, all in Spanish:
We're on the bus now heading back from New York City to Washington DC after an eventful week (read about the $575k Knight awarded Development Seed for OpenStreetMap work on the MapBox blog).
Here are some pictures of the OpenStreetMap intro workshop Ian and I held yesterday at foursquare's headquarter's. Thanks to the NYC OSM community leads Liz, Serge and Eric for the initiative and promotion.
Thanks to David Blackman of foursquare for kindly offering foursquare's great digs in Manhattan for this workshop. Join the OSM NYC Meetup Group if you'd like to stay in the loop on all things mapping NYC.
A great crowd of about 30 people turned out for the workshop.
Kicking off with examples of the many ways of how OpenStreetMap data can be used.
We quickly went hands-on and started editing and improving data on OpenStreetMap using JOSM.
Ian showing how to build quick OpenStreetMap based maps using Migurski's shapefile extracts and TileMill
Liz Barry explains how DIY aerial imagery can be used in OpenStreetMap
And yeah, the view from foursquare's office is killer
We'll arrive on Wednesday afternoon and head out on Monday afternoon. Hotel TBD. On Saturday, I'll give a talk about how we're improving OpenStreetMap with Foursquare.
I'm looking forward to meeting / seeing many great people. Drop me a line if you'd like to get in touch ahead of time.
Come out to the OSM introductory workshop in New York City on September 22nd 2012.
We'll start form square one, with an intro to OSM and its data, learning JOSM, using OSM resources and more. This clearly targets new comers to OSM mapping, but if you're a seasoned mapper, you should come out too. The NYC community is just reviving and there are lots of priorities to discuss and new people to meet. Of course, we should have drinks after (where TBD).
Thanks to Liz Barry for organizing the event.
I am looking forward to see you there!
For documentation's sake: a couple of pictures from OSM workshops in Rio de Janeiro's Cidade de Deus and Complexo do Alemão with Ônibus Hacker, around the Rio+20 Conference for Sustainable Development in June.
Cidade de Deus - street signs
Cidade de Deus - jotting down location and name of points of interest
Cidade de Deus - splitting up our survey areas with Walking Papers
Cidade de Deus - bike path
Cidade de Deus - not quite a point of interest
Cidade de Deus - the hacker bus
Cidade de Deus - the survey team
Complexo do Alemão - dense footpaths
Complexo do Alemão - cable car
Complexo do Alemão - shot from the cable car
Complexo do Alemão - the survey team
The American Red Cross and Humanitarian OpenStreetMap Team called on Volunteers to Map the Ugandan Cities Gulu and Lira on World Humanitarian Day
Mapping at the American Red Cross Headquarters yesterday. On the screen a video message from the team in Uganda.
GeoEye has provided improved satellite imagery that State Department helped process and host for the event. About two dozen volunteers helped trace roads, tracks, buildings and huts from imagery. As a next step the American Red Cross will send a team of trainers to Uganda to initiate necessary ground surveys verifying tracing work and to teach Quantum GIS skills for using OSM data.
You can still join the effort by picking up tasks from one of the two HOT OSM Tasking manager jobs:
For any questions around this effort, get in touch with Robert Banick from the American Red Cross.
- HOT OSM: Preventative Mapping in Uganda with the Red Cross
- Mikel has captured some of the buzz around yesterday's event on storify.
(English further down)
Gracias a todos por atender al taller de OpenStreetMap Mexico DF el viernes y el sabado pasado. Huracán Ernesto hizo que nos quedamos adentro, pero de hecho eso fue excelente, como así pudimos enfocarnos de manera muy profunda en las herramientas de OpenStreetMap.
Ahora, mantengamos el impulso! Como resultado del evento Vitor George abrió una lista de correo para México: talk-mx, suscríbete aquí para mantenerte en contacto! Planificamos unos talleres parecidos alrededor de Desarrollando Latinoamerica que tendrá lugar en varios ciudades Latinoamericanos el 1 y 2 de Diciembre. Mandame un mensaje si estás interesado en ayudar a organizar, impartir, programar o diseñar.
Aquí algunas fotos, una lista de enlaces y temas de los cuales hablamos y algunos usuarios que atendieron. También veanse estas fotos de fotografía aerea del domingo en el Chapultepec.
Hasta muy pronto!
Thanks everyone for coming out to the Mexico City OpenStreetMap workshops on Friday and Saturday. The fact that Hurricane Ernesto kept us inside I think was a good thing as it allowed us to do a great deep dive into doing OpenStreetMap work.
Let's keep the momentum going now, as a result of the event, Vitor George has initiated the talk-mx list, I encourage you to sign up! We're planning on similar OpenStreetMap workshops around Desarrollando Latinoamerica on December 1st and 2nd, please get in touch if you are interested in organizing, teaching, promoting, designing or programming for this event.
Here are a couple of pictures from the workshop, a list of useful links we've talked about and a links of some of the OpenStreetMap users who were present. Also check out these balloon mapping pictures from Sunday.
Vitor explaining Relations
Saturday's training Photo: Karla Lopez
We did breakout sessions for introductory classes. Photo Karla Lopez
Temas y enlaces
Unos cuantos temas y enlaces de Viernes y Sábado. / These are a couple of links around topics we've talked about on Friday and Saturday.
- JOSM Editor: http://josm.openstreetmap.de/
- Aprender OSM: http://www.learnosm.org/
- Edificios en 3d: http://flyjs.com/buildings/
- Quality Assurance: http://keepright.ipax.at/
- Visualizar activitades: http://itoworld.com
- Ver daños por cambio de licensia / redaction http://tools.geofabrik.de/osmi/?view=redactionbot
- Coordinar trabajo alrededor de remapping http://rebuild.poole.ch/
- Herramientas para revisar errores: http://tools.geofabrik.de/osmi/ y http://openstreetbugs.schokokeks.org/
- Cómo saber qué usuarios de OSM están trabajando alguna zona: http://www.itoworld.com
- Para imprimir mapas: - http://fieldpapers.org y http://walking-papers.org Humanitarian Open Street Map Group: - http://hot.openstreetmap.org/
- Tagging documentation http://wiki.openstreetmap.org/wiki/Tagging
- surveying with field papers and GPS
- cambio de licensia y redactions
Listas de correo
- General discussion: http://lists.openstreetmap.org/listinfo/talk
- Mexico list http://lists.openstreetmap.org/listinfo/talk-mx
- Spanish speaking list http://lists.openstreetmap.org/listinfo/talk-es
- Tagging discussion http://lists.openstreetmap.org/listinfo/tagging
Enlaces de usuarios que dejaron su información. Iré agregando en cuanto sepa de los demás. / These are the links to attendees' user accounts who I've managed to gather, I'll add as I get word from the others.
Este domingo salimos al Bosque de Chapultepec para volar un globo del excelente Balloon Mapping Kit del Public Laboratory. Encuentra aquí una pequeña historia de fotos de este día. La idea era de sacar fotos aereas, no fue tan exitoso como nos hubiera gustado...
Haz click en las fotos para ver más información.
This Sunday we went to Chapultepec park to fly a balloon (using the excellent [Balloon Mapping Kit of Public Laboratory. See here a pictures of the day. The idea was to create areal imagery, it didn't quite work out that way...
Click on pictures for more info.
Eric and myself will be leading a MapBox training and OpenStreetMap mapping party in Mexico City tomorrow, Friday (10) and Saturday (11). On both days we will meet at the Telmex Hub in the center of Mexico City. You can find all the details on our blog.
In preparation, I've done some research in areas of Mexico DF we could be focussing on. The weather might not be cooperating, scroll to the bottom to see our plan for if we're stuck with rain.
Here are a lot of areas in Mexico DF that we could survey with Walking Papers and GPS. I've pulled a bunch of them together based on what I see missing on the DF map, not knowing these areas myself. Let's pick ones that are busy and safe enough. Please add more where you see fit.
We should also figure out whether we can/should get bikes or cars for the survey.
(Click to go to see map on openstreetmap.org)
Around La Raza
Around Camarones station
Area around Casco de Santo Tomás IPN
(Just south of the Camarones area described above)
North of Aragón
Peñón de los Baños
West of Iztacalco
The missing street names are due to the redaction process.
Around San Joaquín
This is just north of Polanco
There's a good chance of rain - if we can't do a ground survey (some rain shouldn't stop us) we have a backup program:
- Satellite tracing
- Using OSM data to render your own map with TileMill
- Visualizing progress in OpenStreetMap
- Research local geo sources
- Data density and quality research, create a backlog of next actions for Mexico DF or the entire country.
I'll be in Rio for Rio+20 and launching InfoAmazonia. I'd love to do some OSM work there, I'm thinking walking papers. Please get in touch if you're interested.
I'm working with Rub21 right now on improving Lima. We're finding a lot of disconnected streets there. What's the story behind these? I'm seeing that at least some of the remaining streets are by user Telecom IP - I'll be sending him a message. Not sure whether this is license change related, I remember seen these disconnected roads in Lima well before the bots kicked in.
Right now our plan of action is to repair the streets that match nicely with Bing satellite imagery and not touch those that don't. I'd like to eventually repair or remove the ones that appear to be garbage, too.