OpenStreetMap

mikelmaron's diary

Recent diary entries

Making Detroit the Best Bike Share Map in the World

Posted by mikelmaron on 16 July 2018 in English (English)

I took a tip from Dexter from the City of Detroit Office of Innovation, and started on the Detroit Mapping Challenge by browsing City of Detroit Open Data. What open data could help make OpenStreetMap Detroit the best map in the world? BikeShare locations looked like a useful and straightforward starting point. And now OSM Detroit has very accurate MoGo bike share docking stations. There turned out to be a few surprises getting there, and lessons to absorb for mapping all of Detroit.

The data looked decent on quick inspection, and is licensed public domain. Maybe a very small human supervised import is in order. I browsed OSM to see what was already there, and turned out this already all 43 docking stations were already added by mapper175, with the changeset comment Added nodes for MoGo Bike Share system stations (resurvey needed for most of them). Are they in the right place? Is there something a remote mapper could do here?

I picked the Second Ave & Prentis St docking station at random, and opened it up in iD.

I cycled through all the imagery options available to OpenStreetMap, and no evidence of a bike share dock station. Turns out MoGo launched just over a year ago and all of the aerial imagery is apparently older than that.

I then tried street level imagery. Bing Streetside is comprehensive but collected back in 2014. Fortunately in Detroit, Mapillary and OpenStreetCam have extensive and recent coverage. Clicking through the street level images in iD to find something with the correct alignment to capture the dock was sometimes tricky — depended on the precision of the capture position, the direction, the field of view, and distance. Confirming features seen in the street level imagery against aerial imagery was helpful to choose a well located shot.

For Second Ave and Prentis Street, the docking station moved just a few meters, from on street to inside a parking lot. While the accuracy of the first location in OSM was probably ok enough to find the docking station, the position of on the street vs sidewalk has substantial meaning for high definition mapping and analysis.

I next checked out the dock at Cass Ave & W Hancock St. The original position on West Hancock looked ok, but the OpenStreetCam imagery quickly confused me. One image showed the doc on West Hancock, and another from a few months later on Cass Avenue. I couldn’t trust what my eyes told me here — is the location or direction of the images incorrect? Am I looking at the same dock?

I searched and found that this dock had moved due to street construction. The reported move date didn’t quite match up with the first image but seemed reasonable to explain what I was seeing. And it also mentions the move of ****2nd Ave. and Prentis — somehow luckily the first two stations I picked had substantial changes.

The MoGo site also had a map, and browsing it, the locations for the above two dock stations were very accurate. It looked like the location in the MoGo map corresponded exactly with the docking station kiosk unit. Perhaps I could simply use these locations to correct the data in OSM. But what about the license? Asked Dexter and ..

.. he confirmed that I could us it in OSM and that they would now update the data set on the main Open Data site. Open Data is more than a data source, it’s a conversation.

But first, had to screen scrape. Viewing source, the dock locations were stored in two lists of coordinates. I wrote a quick script to scrape this and transform into GeoJSON and then load into iD as a local file.

I had confidence in this data from the first two stations, but decided to confirm each location from street level imagery before adjusting. I zoomed in on each station from the local file in turn, enabled street level imagery, clicked to find the best view and recency of imagery, until I had a confirmation of the MoGo location.

It became a bit monotonous, though still interesting to investigate the streetscape across Detroit. I began day dreaming about a process that would make it easier for me to confirm. Something that would show the current and new point, automatically determine the best source of imagery from all available — biased towards most recent, and including machine learning to filter images that probably have a dock in view. Step through each point, reposition if needed, and confirm.

For the most part, the locations were spot on. This one confirmed with Mapillary imagery contributed by, in fact, Dexter!

And in just a couple cases, very close but slightly off. Like the station above, where the MoGo map has the station on the street side of the sidewalk, but OpenStreetCam had the station closer to the building. I decided to go as close as I could tell from the imagery.

I learned a lot from the short exercise.

  • OSM is iterative. The first version of this data was pretty good, now it’s great. And we’ll need to update again as MoGo changes and grows.
  • Open data is a conversation, not just a download site. Connect with data holders and it will help unlock more data and possibilities.
  • Street level imagery is a superb source. Utilize as many sources as possible, research from multiple angles, and pay close attention to recency.
  • Human review is always key, but we need to make it very easy with processes that minimize drudgery and take best advantage of human intellect.
  • You can learn a lot about a city, even from far away. Takes patience but it’s rewarding to understanding the geography of a city.

There’s going to be a lot to learn about the city, and especially about the process of mapping, from making Detroit the best map in the world. Excited to see what happens next.

Holiday reflections on OpenStreetMap

Posted by mikelmaron on 21 December 2017 in English (English)

I’m starting to reflect on OpenStreetMap over the holiday. The last several months have personally been simultaneously trying and inspiring. Here’s a few thoughts…

We are all the community Do you contribute and participate in OpenStreetMap in any way? Map, organize, code, discuss, etc? Then you are in the OSM community.

We need to move away from talking about the "OSM community" as being either the people we agree with or the people we disagree with. It’s a pattern I see too much. There are plenty of people and groups that are 100% part of the community, but don't fully realize it.

Community looks different in different places The kind of people, background and settings hosting our community look very different in every city, every country.

This is one of the most amazing things about OpenStreetMap — we’re all working together! University students, open source coders, slum dwellers, professional teams, ambassador(s), geographers. Keeping this in mind is super challenging and necessary for a global project. Trying to understand where others are coming from is something everyone can learn to do, and do better.

We agree far more than we disagree The things we agree on our huge — mapping the entire world openly is still a radical idea.

But the things we argue about might seem like insurmountable gulfs. Yet even on the “polarizing” topics of the past months — organized editing, code of conduct, quality etc — from my seat there’s a huge amount of agreement. Lot of the gulf seems to be about particulars of language and how to get there, rather than essential meanings.

Most of us are quiet The overwhelming vast majority of people on mailing lists and in the OSM community as a whole are not saying anything.

So far in December, there were 411 posts by 94 unique posters. The top 20% of those posters by volume contributed 58% of the posts. I don’t know the exact number of subscribers, but there are about 700-800 OpenStreetMap Foundation Members. There are tens of thousands more active mappers. This is extreme long tail participation.

We don’t know if these people are enjoying or recoiling from these discussions, or totally ignoring them. Every time I post, I do try to keep in mind that my words are going out to hundreds of people.

There are very few barriers to action I have seen very few ideas which are not actionable in OpenStreetMap, there is extraordinary freedom.

That doesn’t make it easy, but it’s much easier than building the map alone. Winning arguments with work is more effective than with words only. You need to listen to and work with others. But there are no absolute blockers. Follow and understand our basic community practices, and big or small things can happen.

Local Chapter Congress Notes from SotM 2016

Posted by mikelmaron on 17 April 2017 in English (English)

The sun shined on the Local Chapter Congress at State of the Map 2016. It was fantastic to hear from so many people from so many communities.

Yes, sorry, it is long overdue to share -- shortly after SotM, I took leave after the birth of my son, and only finding space to pick this up with the upcoming Board Face to Face.

There were many solid ideas, and of course further discussion. Would love to find several avenues to explore these. I think one could be the Advisory Board, which will include representatives from official Local Chapters. For "incubating" local chapters, maybe we discuss ideas on the local chapters mailing list. For communication ideas, we should figure out the right place...

Dorothea took thorough notes from the session. Posting the summary below.

Organising local communities

How OSMF Could Help?

  • Add new "tier" to Local Chapters for semi-formal groups not ready to register as a formal organization or full Local Chapter status
  • Subsidize some costs of groups in poor communities, including support for accessing internet and computer equipment

For OSMF membership

  • 2-level tier for membership fee, to include mappers from poorer countries
  • Vetting of members at lower membership fee by local chapter

Other fundraising options for local groups:

  • Ask local companies
  • Mapathons at places (like bars / coffee shops) which donate percentage of the proceeds
  • On membership signup, option to donate additional amounts

More:

  • Restart equipment (GPS / phones / laser distance devices) lending program, either from OSMF, or between local communities
  • Legal questions on OSM activities in places like Pakistan
  • Groups as part of larger organisations
  • Organize national level donation collection through non profit associations for tax benefit, then donate to support OSMF

Improving communications

  • Central OSM-supplied platform for social aspects, there is fragmentation between communication tools (lists, some people find IRC/forum unappealing, FB, etc) ** The platform should also be helpful for organisation purposes (i.e easy past message retrieval)
  • Creation of map that has local groups with contact details on it
  • Tickbox on sign-up page to accept push-notifications
  • Short video (maybe localised) before the first tutorial in iD, that explains what OSM is all about
  • Identifying & contacting new mappers as they join -- Belgium and Switzerland have models
  • Build tool to contact new mappers. The welcoming message will, ideally ** be personalised ** push people to come into contact with local groups ** let them know options available ** mention local upcoming events ** also, it will be followed-up after a few weeks/month (people have limited time)

  • Possible revitalisation of Welcome WG

  • Friendly reminders to decrease duration of mapping inactivity

Additional suggestions

  • Gamification
  • Drop-down list with current news (national/regional) on osm.org

Running for HOT Chair in 2017

Posted by mikelmaron on 7 March 2017 in English (English)

I'm running again for Chair of Voting Members for Humanitarian OpenStreetMap Team.

My outlook on the role is similar to last year.

I've been busy with the Governance Working Group, and the Election Committee -- for both the new member nominations and Board / Chair elections. These seem to be going smoothly! Working on these has helped further refine our processes and documents. We now have a new member "Welcome Pack".

Happy to continue with this responsibility next year. Please get in touch if you have any questions or ideas for the role of the Chair.

100€ for a notifications listings page

Posted by mikelmaron on 13 October 2016 in English (English)

Working on the challenge laid out by @Zverik to add subscriptions to diary comments was fun! Now I want to suggest another -- an overview of all notifications across OSM.org. This would include diary entries, comments, and notes.

Ideally this page would list subscriptions, in order of most recently commented.

One complexity, Notes have a different subscription workflow than diary entries and comments. With Notes, the original poster and any commenter are notified of comments. There's no way to subscribe otherwise, or unsubscribe. We may want to, in the future, modify that to follow the same workflow.

As laid out by Zverik :"The offer is not indefinite: the PR must be submitted until the 15th of November and merged before the 15th of December. And yes, there might be a competition, in that case OWG will decide the winner by merging a pull request."

Let's Talk Local at the Global State of the Map

Posted by mikelmaron on 21 September 2016 in English (English)

State of the Map attendees are coming to Brussels from (at least) 52 countries! The global State of the Map is a unequaled time to come together in person to share experiences from every corner of the world, find common ground, and plan what's next for OpenStreetMap.

Many of us, among the over 400 attendees, are local community organizers. We hold mapping parties, organize local SotMs, even register organizations and sign up as official Local Chapters. I'm excited that we have dedicated time to talk as local communities on Sunday -- during the panel discussion of State of the Local Map and the open discussion of Local Chapters Congress. There's call to have a Local Birds of a Feather. The discussions we have in Brussels will continue with local communities gathering next week in Manila for State of the Map Asia.

Folks like Martijn, Joost, and myself have been talking with OpenStreetMap local community organizers over the past few months, to learn more about what they're doing, motivations, their challenges, and what they need from the global OpenStreetMap community.

What I've found is that local communities are seeking to get more organized to engage more officially with government agencies, universities and other institutions. They find they need financial administration beyond borrowing someone's bank account. While some have seen the value of becoming an official OSM Foundation Local Chapter, there is still lack a clarity to some about the necessity and benefits. Nevertheless, they see a lot of value to learn from others working on similar issues -- everything from legal and administrative issues of starting an organization, to sharing community engagement strategies that work, to amplifying the voices of their community in the global OSM conversation especially for non-English speakers. Regional connections are especially valuable, for working with mappers in similar languages, timezones, and to some extent culture.

I think there are straightforward things we could do here -- like better communication about and between Local Chapters, develop some simple benefits like schwag and templates of core organizational documents, and more support for regional conferences. Just some ideas.

Really looking forward to hearing more about what local communities are up to and what we think we can do together! See you at State of the Map.

Running for HOT Chair of Voting Members

Posted by mikelmaron on 20 April 2016 in English (English)

I'm running for Chair of Voting Members for Humanitarian OpenStreetMap Team.

I see the Chair as a straightforward role. The key responsibility is to communicate responsibilities and opportunities to HOT Voting Members, and organize the space for official convening and processes. This includes notification of Annual and Special Meetings, Elections and Ballots, as well as ensuring announcements of other meetings, like Working Groups. Expect to work closely with the Board Secretary and HOT's Operations Coordinator, and the Governance Working Group, in these tasks.

My work over the last year with the Governance Working Group has prepared me well for this role. I have been closely studying, revising and clarifying HOT's Bylaws and processes, with focus on making our governance work well for us.

This work be done with excellent clarity. HOT Voting Members cover nearly every time zone, many languages, and everyone's time is precious. Our governance responsibilities should be straightforward and understandable, so we can focus most of our efforts on the amazing core work of HOT.

HOT 2015 Year in Review

Posted by mikelmaron on 31 December 2015 in English (English)

That time of year again ... take stock of my year in HOT. Started off the year as a Presidential Innovation Fellow at the State Department working on MapGive, supporting HOT from the US government side. End the year working at Mapbox, still supporting HOT!

At State, got to help facilitate some truly remarkable collaborations. Nepal was a huge focus for all of us. I worked a lot on coordination, imagery, communications, especially within the USG. Worked with a great group of people to increase cooperation among institutions in OSM. We helped formed a Open Government Commitment to OpenStreetMap, with a great showing at the OGP Summit in Mexico City.

Was part of the team that put together an incredible, inaugural HOT Summit. What an incredible event. Got to tell a story of some of the early HOT history. Started off that week lending a hand with HOT Activation Curriculum Sprint.

Spent time on the Governance Working Group, putting together Bylaws updates. We now have 2 year terms for Board members! Lots more to do.

Sadly saw Kate depart as ED, but warmly welcome Tyler. There's been a super skilled group of folks volunteering and working with HOT over the year, and happily talked with them about various things. What an amazing year --- Tanzania, OpenAerialMap, Export Tool, and everything else I'm missing.

At Mapbox, we made a public commitment to HOT, which I hope is a model for other organizations supporting HOT. We matched the first 10k of the HOT fundraiser.

What about 2016? I'm going to keep volunteering on the Governance WG, we have work to do. Also interested to connect up more with HOT Training and education efforts. I'm on the State of the Map WG, and think we could pull off a great HOT Summit adjacent to it in Brussels. I'm very interested to invest time in local organization capacity, and hope efforts with Local Chapters in the OSMF (where I am now on the Board) can help with that.

Picturing Proposed Development at Josephite Seminary in DC

Posted by mikelmaron on 30 December 2015 in English (English)

Recently learned there is a new real estate development in the early stages of planning in my neighborhood. The Josephite's Seminary has over a block of undeveloped space, and they've entered into agreement with EYA to build townhomes on the property. The number of townhomes being discussed is 150, a higher density of development than the surrounding neighborhood.

screen shot 2015-12-30 at 8 48 51 am

imagery: © Mapbox, Digital Globe.

I had a hard time picturing how 150 townhomes could fit on the site. EYA hasn't yet come up with detailed plans, and has stated that they want to work with the community in the design phase. I am also interested in how maps could help the neighborhood envision ideas for what they want for the development.

So I have simulated what the footprint of the initially proposed development could look like. EYA had developed part of another religious property nearby into Chancellor's Row. I assumed the density and townhouse size would be similar in the new development, that some kind of circulation road would bisect the property, and that the front of the property would not be developed, to preserve the character of the historic facade of the building (similar to Chancellor's Row).

The result is below. This is based entirely off my assumptions, and not with any particular architectural finesse. But this turned out really helpful to see how it could be possible to build 150 properties on the site.

screen shot 2015-12-30 at 8 49 59 am

© Mapbox, OpenStreetMap. interactive version

screen shot 2015-12-30 at 9 11 51 am

imagery: © Mapbox, Digital Globe. interactive version

For comparison, here is Chancellor's Row.

screen shot 2015-12-30 at 8 49 22 am

© Mapbox, OpenStreetMap. interactive version

After making this map, did some searching and found this promotional site for the property, along with this presentation from the December 3, 2015 community meeting. The "Site Use Diagram" shows a similar layout to my assumptions, but doesn't show the layout of individual structures.

I'm interested to continue researching the Josephite Development, and work to see how open mapping can benefit the community process. Questions I have so far are what has been the historic development of the neighborhood since the 19th Century; the estimated cost of purchasing and developing the land, and potential profit and tax benefit to the city; what are the costs for upgrading infrastructure for the development, and what is the impact on transportation network; and what would various alternative schemes look like?

A quick technical note. I took the building footprints for Chancellor's Row from the DC Open Data site. Not all of the structures were in this data set, so I added missing structures by tracing satellite imagery in QGIS. Most of these structures are not in OpenStreetMap, and will add these later on. I copied a selection of building footprints, and pasted them over the Josephite Seminary property in a possible configuration. Saved these from QGIS as GeoJSON, uploaded to Mapbox Studio, and styled to make this map.

Location: 38.944, -76.989

Excited to put myself forward to serve on the OSM Foundation Board

Posted by mikelmaron on 13 November 2015 in English (English)

I am excited to put myself forward to serve on the OpenStreetMap Foundation Board. I'm a mapper, coder, communicator and organizer, obsessed with OpenStreetMap for over 10 years. The OSM community has grown phenomenally. The core governance of OSM, the OSM Foundation, has kept the core resources of OSM stable and strong, but has struggled to keep up with the community. OSMF needs to grow. Growth doesn't necessarily mean get bigger; I believe within our community we have everything we already need. What it does definitely mean is getting smarter and faster about how we engage and collaborate together beyond the map. That means creating proper space and structure in OSMF for a much broader diversity voices and activities of the OpenStreetMap community. I have a strong record of building alliances and networks in the OpenStreetMap community, and am ready to bring my efforts to OSMF.

OSM is a global project, and participation in OSMF should reflect that diversity. Local Chapters are a critical means to bring more voices and energy into OSMF. Local Chapters are national and local level groups of OSM mappers, some more formalized than others. We should engage Local Chapters (whether officially signed up with OSMF via an agreement, or more nascent) to broaden our discussions and deliberations, and recruit more help for the critical activities of the working groups. We can help Local Chapters do what they do better, with support for community management, events, and organizational capacity. Linking chapters together to share their knowledge benefits everyone. I'd help kickstart this, through targetted discussions through Local Chapters, on what they hope to see from OSMF and OSM.

Re-energizing working groups are key. I want to talk with as many OSMF Members as possible, hear about your interests, and help you find a way to contribute your time to OSMF. I want to promote Working Group activities loudly, and actively and directly recruit people to get involved. Working groups, whether Licensing, Data, Communications, Operations, etc, should be able to move quick and comprehensively as issues come up, as well as have a longer term view of where they're heading. When this involves funding, OSMF needs to have a greater (if still relatively small) financial capacity to raise and spend money.

Partner organizations, whether government, academic, corporate, or non-profit have a role to play as well. Providing means for folks working with OSM from their organizations to contribute as well to OSMF will provide a breadth of much needed support and skills. This means boosting our sponsorship program, with more definition of what partners should expect from OSMF and vice versa.

OpenStreetMap will have an amazing State of the Map in Brussels. I want to see many opportunities for broad participation there, with solid outreach to other communities and interest groups, space for our global community to represent through scholarships and a Local Chapter "Congress", and space for related communities to gather before and after, like the HOT Summit. And smaller events are critical too. Part of our support for Local Chapters translates into help to make even more great regional and national State of the Maps.

This will be my second stint on the OSMF Board. I've learned a lot about how organizations work since then. I've facilitated buy in to OSM everywhere from international organizations, to informal settlements, to activists, to educators. I served on the Humanitarian OpenStreetMap Team Board as President, and continue to lead the Map Kibera Trust Board. My most recent year was spent as a Presidential Innovation Fellow at the State Department, where I helped the USG publicly commit to Open Mapping in a big way. I've recently joined the Mapbox data team. I've helped initiate mapping efforts around the world, from the ground up.

OSM continues to fascinate me. Please get in touch with your questions and ideas. I bring many perspectives and experiences to the Board, and can't wait to work with you all.

HOT 2014 Year in Review

Posted by mikelmaron on 5 January 2015 in English (English)

I started my HOT year a worried president of the HOT Board, and ended as a confident regular member.

We all worked hard. Really hard. Most importantly on HOT activations and projects. HOT has had a stunning impact on humanitarian response. But we knew we could this. What challenged us more was ourselves. We worked hard on HOT's organization, the processes and relationships that make the space for amazing to happen.

Ok, let me just say personally, I had a lot to learn this year. I'm not proud of everything I did. But I'm immensely proud of where HOT is now.

The Board Face to Face was a real turning point. Sincere thanks to the Board for putting our all into this. And thanks to our guide Gunner.

Some other things I spent time on: helped coordinate to get V2 of the OSM Tasking Manager developed; formalized imagery coodrination; put together trademark applications for HOT; formally employed our Executive Director.

I joined the US government for a year, and really just getting started. My HOT 2015 orbits around this. Since I'm no longer on the Board, I'll have more time to put into working groups, community building, technology.

Excited to Say, I am a Presidential Innovation Fellow

Posted by mikelmaron on 15 September 2014 in English (English)

Today, it can finally be said, I am a Presidential Innovation Fellow at the State Department working on OpenStreetMap for Diplomacy.

This is very exciting, and honestly a bit boggling, how it's all turned out.

9 years ago, I was living in Brighton, UK, and travelled to Nottingham for several days of hacking with some very creative technical people. Invitation was from Ben Russell, "author" of the Headmap Manifesto (read this). Ben was a kind of hero to me, so that was great, and we spent a lot of time with Steve Coast, I built a slippy map for OpenStreetMap. We blew our own minds. Ben summed it up ... OpenStreetMap was going to totally succeed, or fail spectacularly.

The people I've met through pursuing this crazy dream of OpenStreetMap, the adventures, the real places opened up ... I won't even try to sum up how this project has taken over my own life, and changed the whole world for the better. Just this. When we first started talking about OpenStreetMap for Disasters, the response was sometimes polite, often condescending, and always bewildered. Today, Humanitarian OpenStreetMap Team (HOT) provides geographic data direct to the ebola response. We didn't ask permission, but believed, listened very carefully, kept working, and created something entirely new.

And this year, the White House and State Department created a Fellowship for OpenStreetMap. The Humanitarian Information Unit (HIU) has already moved mountains with Imagery to the Crowd, providing satellite imagery for digitization into OSM for humanitarian response. Makes whatever comes next seem easy! I'm really looking forward to working with HIU on MapGive, with eDiplomacy, and with others at State, USAID, and other agencies working with OSM. Our class of Round 3 fellows are an impressive bunch, working on hard problems. We've met many folks at our sister groups, 18F, US Digital Service, White House Office of Science and Technology. There is a vibrant spirit here, true believers in transformation of government, but rooted in smart, sobering, hard work and commitment.

The problems facing the world, humanitarian, climate, environmental, economic, are so extreme, everyone must find a substantial way to work together. No matter if you are a slum dweller in Chennai, or a bureaucrat at the World Bank, you contribute to the same database in OpenStreetMap, and that is powerful. In fact this is the role of HOT, to serve as the interface between the quite different worlds of the OpenStreetMap community and humanitarian response, and bring people together. And this is why I put myself forward to serve as a PIF. There are entirely new ways to cooperate and organize now open to us, and while the answers are certainly not obvious, I felt a calling to explore the landscape of potential.

The specifics of what I'll work on are still being created. But all our discussions so far focus on supporting growth of OSM, building as usual in the open, with reusable work for all. I think there will be three parts. First, contribute to curriculum and guides for teaching OSM, and organizing OSM projects and communities (the "softer" stuff, mostly in our collective heads). Second, build software tools to support community organizing, improve data quality, and flow of imagery. Finally, trial all of this somewhere out there, including connecting with other groups working for OSM in DC, through cross agency and organizational collaborations.

I needed to make a few changes to focus on the PIF. I'm stepping down from my role as President of the Board of HOT (but will remain a dedicated Board member). As well, have minimized other commitments at GroundTruth Initiative, Map Kibera, and Moabi. Yet, I'm not going far. I'm still editing right here, contributing on GitHub, publishing, probably more than ever. And especially listening. In a very fundamental way, I can't do this fellowship alone, and rely on the energy and ideas of our collective. I will need your help.

With nothing else to say at the moment … Maps!

Disclaimer: Views are my own!

Moabi at State of the Map US

Posted by mikelmaron on 17 April 2014 in English (English)

Update: we had an excellent time at SotM-US and the Sprint day. The presentation slides and video are now posted.

The Moabi development team is excited for State of the Map US this weekend. We are sharing a preview of the new Moabi (to be fully launched on Earth Day), and presenting our work on Sunday at 4pm, OpenStreetMap as Infrastructure, sharing the stage with the USGS National Map Corps project, and NPS Park Tiles. Hope to see you there! And if you want a demo any time this weekend, find one of the team, Sajjad, James, Leo, Chippy (virtually) and myself.

First why Moabi?

Moabi-DRC is an independent mapping initiative that collaboratively monitors land use in the Democratic Republic of the Congo. Our community works towards a more Transparent, Equitable, and Sustainable future for the environment and people of DRC. You can use Moabi DRC to explore, share, and create projects on a wide range of issues from REDD+ to community mapping and more.

Why OSM as Infrastructure?

OpenStreetMap's render stack, editor and web application can be used to power collaborative mapping efforts beyond OpenStreetMap. OpenStreetMap's software is unique and powerful as infrastructure for building communities of contributors. What happens when OpenStreetMap software is reused for new data sets and communities beyond OpenStreetMap.org?

We've been working on customizations like...

Preset Editor for iD

Tile management through GitHub and a OSM TileAPI

Map Story Building in OSM

Showcase Map Sites in Jekyll/GitPages

So much more this weekend, see you!

Running for the HOT Board

Posted by mikelmaron on 21 February 2014 in English (English)

I'm running again this year for the HOT Board, my third election. I sincerely ask HOT Members for their vote and the opportunity to serve them in a fourth year on the HOT Board.

My history with HOT is now getting long, and my dedication to HOT stronger than ever (not difficult, with HOT more amazing than ever). I appreciate Robert Banick's nomination and touching on some of that. I've also recently told the story of my HOT Year, including my time spent directly on HOT Board, which as Heather illustrated, is a substantial time commitment. While it's varied recently with time off for my family, pretty fair to say that I voluntarily dedicate 20% of working time to HOT.

So like many of us, I have so many ideas for HOT, it's hard to know where to start. Here's my priorities for myself over the next year, touching on where I think HOT needs to focus and grow.

Raised many times in discussion lately, our Bylaws need to be updated, to better reflect the shape of our community, and clarify ambiguities. I led the process of developing the HOT Code, and am ready to facilitate the process with all interested members for the Bylaws.

Connected, there are many HOT processes sitting collectively in our heads, that don't need to be enshrined in the Bylaws, but would be good to get these out objectively for us to better coordinate, in lightweight documentation. One recent example, would be a guidelines for administration of the tasking manager. We don't want to over proscribe the process, but as quickly and clearly collectively illustrate the minimum we need to know.

Building on that, the HOT Guide is something I'd like to help move to first version. Ourselves as members, our broad volunteer community, we all have so many skills to contribute, many outside the usual line of the mapping we do so well. Making it easy for everyone to find ways to contribute to HOT is what the HOT Guide is about.

Something we have not done consistently in our project and activation work is evaluation, yet incredibly valuable for us to steer our work. I want to help enshrine this practice. HOT as an organization itself came out of a series of evaluation and strategy building following Haiti. The recent Haiyan OSM Assessment gives one example of ways we can begin to examine our work, and I hope over the coming months, events and groups in Philippines and elsewhere can also contribute to our learning.

Humanitarian partners are what make our mapping work relevant to disaster response and preparedness, and those relationships are a big part of what HOT manages. We've reached a profile and gravity where there are a lot of organizations involved in what we do beyond just using the data, and great potential for coordination. Tapping into this more and figure out what this looks like is something I want to pursue.

Finally, want to step up more in helping our Board be productive, continue to serve in leadership of the Board, and facilitate our process and communication and meetings well. A big part of that will be collectively setting our strategy for the year, and increasing bidirectional communication with members and the community, especially with more frequent reports on our activities and finances.

Thanks for reading all of this, and considering a vote for me.

My HOT Year

Posted by mikelmaron on 5 February 2014 in English (English)

We've just seen a phenomenal group of folks join, the Humanitarian OpenStreetMap Team. And along the way, we are reading excellent stories from the nominations and new members recounting their personal HOT history, wonderful to learn about new folks and familiar faces.

HOT has adopted a long awaited Code of Conduct, and that's planned to be ratified as part of the upcoming Annual Meeting and Board Election. The Code makes clear how HOT membership operates, the rights and responsibilities that come with membership. Membership signifies another level of dedication to the incredible work of our community. One detail there is our expectation to share publicly our yearly contributions, for our quick growing and widely dispersed community, and for all partners and supporters, to have a window into our HOT lives.

At the end of each membership year, members will be asked to document their contributions to HOT over the past year, and their aspirations for the next

So here's My HOT Year. It got kinda long.

HOT Board

My main role in HOT is as Board President. Part of that responsibility is to organize and chair the monthly meetings, with the support of the Secretary. Board meetings are for making salient points on issues, making decisions or voting as needed, and assigning actions to complete outside of the meeting. We also discuss and report and decide on straight forward issues over email and OpenAtrium continually. Our duties as Board Directors are above all to hold the fiduciary duty of the HOT, working closely with our Executive Director Kate Chapman. We have oversight over HOT's finances and projects and fundraising. We help set strategic direction and draw up policy, to encapsulate as close to consensus as possible the operations, processes, and structures of HOT. We discuss emerging ideas and issues for the organization. We oversee membership processes. And we do this all while being active members of HOT. Yet it's important to note, our Board responsibilities are quite distinct from other activities in HOT; just being a Board member takes a great deal of dedication and time. I so appreciate our Board Members whom have really taken on this major responsibility.

One of my main projects for the Board this year was leading the writing of the Code of Conduct. This was a major group effort, and I'm really proud of the results. Big thanks go to those that contributed words and comments, Russell Deffner, Katrina Engelsted, Jaakko Helleranta, Tim Waters, Kate Chapman, and Joseph Reeves. I'm especially happy that Russell's efforts led to his joining as a HOT member.

Through the Board, I also took on responsibility to get the Tech Working Group up and rolling, holding regular scrum style meetings and getting our project management tools (otherwise known as a list of GitHub repos) in order. And before too long, was able to hand over to the capable hands of our new system administrator, Dražen Odobašić!

HOT at OpenGovHub

By "virtue" of being in DC, I'm the constant presence at our "world geodomination headquarters" in the OpenGovHub. We have lots of folks come through. HOTties, if you ever find yourself in DC and need a place to lay your laptop, you are so welcome. Less excitedly, I receive HOT's snail mail, and occassionally deposit a check or scan some important mail for our admin Kristen Egermeier or "file it away".

Many of HOT's partners are based in DC, so I have regular contact, some planned some serendipitous, with good folks from State HIU, USAID, World Bank, American Red Cross, Peace Corps, Wilson Center, GWU, among others. This is often just catching up, but sometimes leads to substantial things for the Board and ED to consider, so I then share for deliberation.

Tech

Outside of the Board, I spend a bit of time on technial issues ... though not as much as I'd like, it's a lot of fun to get into our code.

State of the Map US allowed a group of us to get together and start coding for HOT, focused on upgrading and relaunching the HOT Website and adding iD support to the OSM Tasking Manager. That continued through the OSM Birthday Sprint, which was great to co-organize with Kathleen Danielson. The website relaunched in December with the Drupal muscle of Felix Delattre and Clara, and is already improved with multilingual support. Steve Kashis dove into the OSM Tasking Manager, and made immediate progress, a tribute to the clean software design laid out by Pierre Giraud. The iD developers were very supportive, adding parameters to support OSMTM needs. Just today, Pierre, with a little help from me, finished up the iD integration by automatically loading imagery from the OSMTM.

I'm excited about the tasking manager especially, and think there's lots more it can be developed to do this coming year.

For LearnOSM, I facilitated improved documentation for translators. Daniel Joseph from the American Red Cross did an amazing job of synthesizing the process into something approachable for newcomers to GitHub.

Finally, I collaborated with Lars Bromley from UNOSAT and Edouard Legoupil from UNHCR, to import Zaatari refugee camp into OSM. This was a fairly interesting technical challenge, in that it involves synchronizing updates over time. And a solid example of collaboration across agencies within OpenStreetMap. Folks like Lars and Edouard, working within humanitarian agencies, are doing the truly heroic work of moving mapping mountains, freeing data for humanitarian benefit.

Outreach

SOTM-US HOT BOF

State of the Map US was a highlight. Schuyler Erle led a presentation on HOT, with contributions from Amy Noreuil from our partner USAID OTI, then EUROSHA volunteer and now HOT member Jorieke Vyncke, and surpise guest Presler Jean from Haiti. This led to a great Birds of Feather session.

I did less travelling last year. But one place I did get to was Democratic Republic of Congo, and got to finally meet Claire Halleux, who's about as bad ass as they come. It's a small mapping community in DRC, and we had some interesting meetings with the UN while I was there. Claire, we still need to import MONUC's road data!

Peace Corps

Closer to home, I connected with the Innovation Team at the Peace Corps, who are keen to see how Peace Corps volunteers can incorporate OSM into their service. We held an event and editing session at Peace Corps HQ, which led to a piece in the Peace Corps magazine. They've continued with editing competitions, and I've made 3 presentations/interviews remotely to trainings in Senegal.

Finally in the media, represented HOT and OSM in two excellent magazine pieces, in Wired and New Scientist. During Haiyan, I stepped for a couple interviews on NPR and PRI.

Next Year

Next year is now. With a growing community and more responsibility than ever, HOT can do more and will be called on to do more. For me, I want to focus on more clear ways for people to contribute to HOT; mapping and coding definitely, but in other organizational ways as well, in translation, in coordination. That means really good documentation of our processes and policies, a really clear understanding of our resources and how we work with the broader OSM community and partners, and better communication within our rapidly growing community.

Communication can be hard. We've had our share of hard knocks in communication this year. The source is often simple misunderstanding. If ever something arises which concerns you about HOT, I am personally always accessible to listen and share my point of view. Please do get in touch.

We also need to work on our Bylaws. The Code of Conduct process, and membership renewals, showed gaps in how we want to operate, and our present legal structure. This will be hard work, and we'll need some serious time and effort on it.

That all sounds serious, but seriously, after all, HOT is the most fun and valuable thing I do. Connecting with this global community of map nerd do-gooders, and moving serious mountains ourselves, seriously changing the world, HOT absolutely, thank you for the year, the years, and the year to come.

GeoGit and GitHub Geo

Posted by mikelmaron on 26 September 2013 in English (English)

As I've been exploring the OSM rails app for other data, Git has hovered in the background of my thoughts, and I've been watching GeoGit and GitHub Geo Features closely. The conceptual basis of Git, distributed version control, solves issues we come up against regularly in OpenStreetMap, like how to keep an "authoritative" data source and community data in sync or how do we support offline editing, in areas with bad or non-existent net (something to explore with BRCK perhaps). As Jeff Johnson says, "OSM is a geodata repository with just a single branch".

GeoGit

Chris Holmes gives a thorough recap of BoundlessGeo's rational and work so far with GeoGit (part 1 part 2) including experiments with using git itself. Git is built around managing revisions of individual files, and hits performance issues with very large files, or very large numbers of directories (which early GeoGit experimented with, using a directory hierarchy to support quad-tree indexing). So they worked to decouple Git's set of verbs from its backend, and implement those concepts on top of spatial databases, and provide special verbs particular to interact with OpenStreetMap (or even perhaps, OSM clones). Perhaps that's comparable to git integrating with svn.

The work looks really promising, though they are still working on the internal technical challenges, and they've set the bar high, to fork the entirety of OSM including all history! They're tremendously talented, so I expect they can get there. But what then? Git without the interface and social features of github is a frustrating experience. Replicating that kind of community space for GeoGit is another tall tall order, and I expect not the first order of business for a GIS oriented customer base.

Now, if interacting with OSM clones is a core part of GeoGit, then perhaps can simply use the OSM application ecosystem for editing and socializing on an individual branch. And in that case, makes sense to invest effort in improving OSM website features for Moabi, and leverage any future interoperability with GeoGit itself.

GitHub Geo

"GitHub Geo" takes another approach. Accept the file limitations of git, GitHub and browser client displays (split up large files, if needed, etc). For rendering GeoJSON on GitHub, the limits seem to be in the 5-10 MB range. They seem to be loading the data as a GeoJSON layer in Leaflet/MapboxJS, so a performance improvement would be rendering of that data into map tiles and utf8grids. I would guess that there's already been thinking into what kind of infrastructure would be needed to support that, and they're watching uptake of Geo features before making that kind of investment.

The approach takes full advantage of GitHub social functions, and there's been some fun examples of this, and it will be great to see if the city of Chicago will accept pull requests. Just as important, the GitHub API to build applications to interact with GeoJSON files. This is what MapBox has been doing with Prose.IO to edit gitpages, and their GeoJSON.io provides an excellent editing environment for geodata coupled to GitHub, and would be the first mapping example of this pattern. (And as a mindbender, the entire GeoJSON.io site is hosted with GitPages). GeoJSON.io also has performance limits, probably similar to what we've seen with iD (I assume some of the internals are similar, but I haven't looked).

Another service matching this pattern launched this week is GitSpatial, which selectively syncs GeoJSON files in your repos, and provides a simple geospatial query API, for example an API for Kenyan constituency boundaries counties near Nairobi.

This pattern could conceivably be used to build a rendering service, that could build those tile sets and grids, or vector tiles, from very large GitHub GeoJSON files. Or a combination of GitSpatial indexing & geo API and GeoJSON.io could build the ability to edit a select area of a large file, and commit just those changes. That would I suppose require GitSpatial reassembling the GeoJSON features in order they were received, or using some convention for ordering features based on a geographic index, and committing a diff to that file. Another useful service would be visualizing geographic diffs, something that OSM itself (or anything really) doesn't do particularly well. Though that feature is something I could expect from GitHub soon, seeing that they are now doing 3D file diffs.

Large geographic data collaboration today

This is all so new and untested, and so far, not really built with large data sets in mind. Unlike the OSM architecture, API and ecosystem, which can pretty solidly handle loads of data and provide lots of services. It's hard to get this kind of glimpse of the future, but also have the needs of today to grapple with. For now, I reckon OSM is a really good place to experiment and build, while we'll keep a close eye on these other approaches.

Tags are all too human

Posted by mikelmaron on 11 September 2013 in English (English)

This is a short write up of a small amount of thinking and investigation of potential ways Moabi could extend and contribute to tagging tools in OSM.

Tags are all too human

Tags are the magic tools that allow OSM to happen. There's no systematic restriction to how features are tagged. Rather it's based on experience, convention, conversation, invention. There's no universal representation of the world, and tags permit that. It's messy, not always rational, sometimes absurd like horse=yes, but also mapping social facilities in refugee camps. Using the common ones becomes an exercise in crawling through the wiki. Inventing new tags is a whole other adventure.

And so is software

Though, we make it easy on ourselves, and make tools to work a little bit with tags. There are presets for editors, style sheets for rendering, analysis tools.

How are tags read and managed by software? It's still an (impressive) grab bag.

Editors

For iD, presets are defined as json files. Each presets specifies a tag, and other tags to fill out for that feature ("fields"), as well as synonyms for search and a logo. It's a clean representation, focusing on the "data structure", and the collection of features is managed through github. Afaik, there haven't been many attempts yet to create custom sets of features. The would require compiling and hosting your own version of iD, there's no way to do it directly in the website.

JOSM presets have been around a while. It's an XML format that defines both tags and fields for features, and sets of features in a hierarchical menu, as well as labels and layout of dialogues. There's dozens of preset files out there, HOT maintains a set for the Humanitarian Data Model, Map Kibera has created them, tons more.

Forever, it's been a pain to create these JOSM presets. Hand writing fragile XML is not so fun. The [http://visualtags.hotosm.org/](visual tag chooser) is a tool to create these presets. Here's the code in github. Definitely a step up from hand crafted XML. Though still a little challenging working through the interface.

Rendering

Visualizing, there's the classic tool chain of osm2pgsql and mapnik and now carto for transforming tags into cartography. Other tools, including editors like JOSM, use the similar but different to carto MapCSS.

And..

There are some QA tools which monitor tags, to look at common issues in how tags are assigned, how they are used in combination with each other, etc. There's Validation tool built into JOSM. Keepright has a set of tag related tests. But there's not really a framework out there which allows for general application of quality checks on tags.

And of course, there's the mighty taginfo, which builds up global statistics on tag use, builds lists of combinations and geographic spread, and provides APIs for linking into other places tags are handled, like the wiki and editors, and links to other useful tools for querying on tags like Overpass Turbo. TagInfo is a sinatra app that others have set up, to focus on particular geographic places.

And what about Moabi?

Moabi wants to provide a bit more structure around tags. A group in Moabi (based off the OSM groups sketch perhaps), or a user, want to create a Feature within Maobi. That Feature is then made a searchable option in the editor. When a user creates or modifies a Feature in Moabi, it will be validated against that definition, and the results shown on the "browse" page. Further, there may be permission levels set on who can CRUD a Feature, or even at the level of specific Fields within a Feature. As well the definition of a Feature might have its own access rules.

Essentially, define, validate, visualize and restrict tags.

Pretty different then OSM right? Is and where is the overlap, and how do we best diverge?

And where do we do this again?

Honestly, I've always thought a central place to manage tags would be really useful. Well not to definitively enforce tags. But a place more functionally tuned to the problem then the wiki, that could provide some of the process of tag approval (even if tag approval has no functional meaning!), and then semi-automatically produce and integrate with the editors and rendering tools. TagInfo seemed like not a bad place to at least acquire some of these features ... again without changing the fundamentally free flowing nature of tags. Just make it easier for us to track what's going on.

Representation

Though "tags" are different from "features". One nice thing about the iD representation is that it introduces this concept. JOSM presets has this as well, though it includes groups of features, whole menus of features, along with layout and labels. iD has json just define that single feature, which is built up of references to reusable fields.

Editing

Somehow, I prefer the iD representation, cleaner to build off. But then, does it need to be represented in a file? Where would the creation of features take place? It could take place in an external application, like the Visual Tag Chooser, or TagInfo, or something else. It would then require some secondary transformation and distribution steps to make its way into the OSM applictions, to be used in editing, validation, and rendering.

The other option would be a MVC of a Feature directly in OSM, taking advantage of any group or user permissions developed there, etc. We'd still have some transformational steps to bring those options into iD (by producing those json files, or building in a step in iD to get options from the DB), as well as to produce custom tiles more easily. Could possibly even make use of some of the interface components of iD itself to manage the editing here.

Validation

Validation during editing could be accomplished by iD providing some feedback during Save based off the Feature representations it has loaded. If that was structured properly, that JS validation could be isolated and reused as a part of the browse page. In this model, as usual, feature components are not enforced, but are made more visible in more places.

Restriction

If Features, or even particular Tags within a Feature, have some access restriction, that would need to be included in the Model and/or representation. Perhaps Groups are a natural way to strucure these permissions, or even Roles within a Group, or a Platform as whole. For Moabi, might even need to apply restrictions on the geometry of an object, only allowing it updated through particular import accounts, and just allow editing on other particular fields.

There's two issues. This kind of access restrictions can be applied sometime during CRUD, and if a problem is detected, simply reject the entire Changeset, similarly to how editing out of data OSM objects throws a "conflict". The other is, how can Features be identified as Features, and not just a collection of tags. For that, we might use some kind of "hidden" tag, as JimmyRocks suggests in the comments, which specifies the Feature, and can link to restrictions defined elsewhere. This idea could also be expanded to manage "approval" of contributions, so that within a Group, some members can contribute, others can certify.

So where to start? I reckon, we start with what's more potentially generally useful, like tag definition and validation, and then move on to the more restrictive non-OSM options, when they're really more clearly needed. But might also make sense to start with the least OSM like features, to really make sure that this whole direction of using OSM for non-OSM data makes sense.

UN Collaborates on Zaatari Camp Data in OSM

Posted by mikelmaron on 4 September 2013 in English (English)

Zaatari, in northern Jordan, is facing huge humanitarian pressure from the Syrian conflict, considered the second largest refugee camp in the world. To aid the response and improve coordination of geographic data in the camp, a remarkable cooperation is underway there among UN agencies, the OSM community, and potentially, the refugees themselves.

Getting Here

As the Syrian crisis deepened, HOT was prompted to activate in January. It's naturally proven difficult to coordinate remote mapping inside Syria, though there has been significant and interesting activity over the past two years.

We've known Edouard Legoupil from UNHCR for a while, through even pre-Crisis Mappers days. We personally overlapped in Nairobi at the start of Map Kibera, and since, we've discussed ways of bringing open approaches to mapping and engagement in camps. Likewise, Lars Bromley from UNOSAT has been a colleague since we collaborated together in Darfur Google Earth layers project, and while he was at AAAS, supplied detailed historic satellite imagery of Kibera for our analysis of spatial extents of post-election violence. Both have innovated, very close to where things are built, and have created space within their organizations to work collaboratively. With the HOT activation, we've been able to come together and do practical work on those ideas for Zaatari.

The unhrc-jordan OSM user has imported roads, toilets, water points, health facilities, mosques, schools, offices, even restaurants. Another partner organization, UNICEF Jordan also uploaded child specific infrastructure (though this was later undone, and re-uploaded by UNHCR for some technical reason). Now, UNHCR is working with REACH to map on the ground in Zaatari, both confirming data and including "informal" infrastructure built by refugees; all the normal services of a city are growing in this place. In the future, there may be opportunity to engage refugees themselves in the processes. In traditional practice, data is shared through colleagues. Through OSM, they become openly collaboratively shared.

UNOSAT Import

UNOSAT is tasked with another piece of the geographic puzzle in Zaatari. They analyse satellite imagery to identify ALL structures in Zaatari, currently at 28174 structures. Lars confirmed that this data could be imported into OSM as well, and that I could help with that process through the UNOSAT OSM User.

Working off the July data, the first challenge of the import was transforming the ESRI Geodatabase released by UNOSAT into a format I could work with. Luckily, I was at OSFAC's office in Kinshasa where I had access to ArcGIS, and could do the conversion to Shapefile. Then, to my surprise, JOSM was able to read the Shapefile directly! In JOSM, I used the under-appreciated filters feature, to select only features with Shelter_St=1 (meaning the structure currently exists, the data contains shelters that no longer exist). Then, mapped the Structure_ column to either shelter (for 1) or administrative (for 2). Also stored the acquisition date, event code, and source, for reference (within a unosat: tag space). And also, the objectid, which could be used for applying updates later on. Removed the other tags. And because of the large size of the upload, used a script rather than JOSM. Here's an example structure from the import.

UNOSAT Conflation

So right before I was ready to do that import, Lars notified me that UNOSAT had released a new analysis. I could just redo the process above with the new data, or use the opportunity to test if it works to update based on unosat:objectid. Yea, did that!

Again, stuck with a GDB, until I was fortunately introduced to the simple and strong GDB Flee. From there, I went about writing a conflation script, which reads in the shapefile and OSM data, detects actions (create, update, or delete, either way), and then outputs OSM XML for examination in JOSM. There might be other tools to do this (please let me know in comments if so), but was useful for me to get my head around the process, a first step at abstracting conflation, and dig into some geo mangling code.

It's not fully tested against all cases, just the situations in this conflation, so putting together proper tests for this later is on the agenda. In this case, after the deletes and creates, the remaining number of structures added up precisely to what UNOSAT reported, so looks fine.

The thing to grapple with here are unexpected situations. The script deals with this by outputting OSM features with a NOTICE tag. Then in JOSM, can use filters to only show those features, and inspect and make decisions manually. For this import, there were about a dozen features that were supposedly collected in the previous analysis, but hadn't already been imported. This was strange, but didn't investigate further and since the numbers added up, decided to include them in the import.

So What?

Interesting results immediately came out from comparing multiple datasets, and putting many eyes on the data. Those dozen "missing" features were highlighting in JOSM Validation as Nodes at same position. The precisely same lat/lon, but different objectid tags. I confirmed they were definitely present in the current set of features, and that the total number of featues matched what's reported by UNOSAT. So perhaps I can conclude there may be duplicates in the source analysis, and hopefully this community view on the data can help improve the authoritative view.

Also interesting to look at how the UNHCR and UNOSAT data interact. Administrative buildings or compounds are polygons in UNHCR, points in UNOSAT. Some compounds contain multiple administrative structures; it might be interesting to associate them via a relation or tagging. Some shelter structures show up with administrative areas; do those structures need to be reinterpretted, or have the areas changed? Now that these are together in a single database, we can script or use GIS to analyse those kind of issues and prioritize further investigation.

One thing that might help is for OSM users to access the imagery directly. In fact, the most recent analysis come from State Dept HIU supplied imagery of Zaatari, so we can perhaps request access through Imagery to the Crowd. We'd want to have some good analysis first, and a clear idea of how mappers could help.

Finally, there's potential to do custom rendering which highlights all the data, and camp specific information. UNHCR is already rendering Zaatari on their data site. It would be good to also show the node buildings in an unobtrusive way, to show how the admin structures and shelter distributions interact.

Location: 32.292, 36.326

Skill Share: Map Photos Using OpenStreetMap and TileMill

Posted by mikelmaron on 14 August 2013 in English (English)

Recently, we organized a fun skill share with Map Kibera, and then again with Transparent Chennai. The idea is to use OpenStreetMap and TileMill to create an interactive photo map of OSM features. Here's the simple result from working with Map Kibera.

Want to learn to create something like this? This guide takes four steps, demonstrating an overview of the whole flow and connection between four tools to build a photo map. This guide will hit the main points, highlight how the tools connect up with open data, and I'll just link to other resources for the details on how to use each.

Note, this guide is to create photo maps of objects mapped in OSM, not arbitrary points where you have happened to take photos.

Upload Photos to Flickr

  • What you need: A photo of a place to map.
  • What you get: Link to the photo page and photo preview.
  • Learn more: Flickr Help

If you don't have a Flickr account, create one, it's an easy way to share images. Upload a photo of a place. For our example, we uploaded this photo of the iHub in Nairobi, taken during the Uchaguzi election project. We created a Map Kibera group to hold all our collective photos. MK mappers get in touch with me directly if you need access.

The result you should have is a link to the photo page. Something like:

http://www.flickr.com/photos/mikel_maron/9420814521/

Edit OpenStreetMap

  • What you need: Location of your photo.
  • What you get: OpenStreetMap feature linked to the image.
  • Learn more: LearnOSM

Now, over to OpenStreetMap. Sign up if needed, and review the basics in LearnOSM. Also, try the iD Walkthrough: navigate to an area, click Edit (specifically, click the down arrow, and select "Edit with iD"), and follow along the editing tutorial.

Now open the View tab again. Enter the name of the location of your photo, and search. In our case, it's "iHub". Look through the search results, and find the correct feature, there might be more than one feature with that name, and then click View Details to open the browse page for that node. This page shows the location and tags for the feature, and in the lower right, click the Edit Node link.

Often you'll need to add a new feature in OSM, rather than add to an existing feature. Features should be appropriate for OSM, not just an arbitrary point where your photo was taken. Check out the other guides to learn more.

In the left panel, click All tags, then the +, to add the tags to link the image on flickr. Add "image", then the link to the Flickr photo page. It will look like:

image=http://www.flickr.com/photos/mikel_maron/9420814521/

Click Save, add a commit message, and click Save again.

Caveats

Important: When you are taking photos, make sure to record which photo goes with which object! It can get confusing if there's a lot.

Extraction with Overpass Turbo

  • What you need: A Deep Breath (take one now!)
  • What you get: A GeoJSON file containing images and locations.
  • Learn more: Overpass Turbo wiki page

The OpenStreetMap website presents one view of the data you contribute. The powerful thing about OSM is that you can take the data and use it however you like. Extracting the data from OSM itself, for use in another tool, is what this step is about.

Open Overpass Turbo with the template to search for nodes with key "image". You can find links for other tags by exploring TagInfo, another great tool I'll cover in a later skill share. The interface has two panels, on the left the "Query" and the right the "Map". Search and navigate the map to the area where your features are located. For us, the is "Nairobi".

The query to do the extraction can get complicated; you'll see the template settings on the left, looking something like.

{{key=image}}
{{type=node}}
<osm-script output="json">
  <query type="{{type}}">
    <has-kv k="{{key}}"/>
    <bbox-query {{bbox}}/>
  </query>
  <print mode="body"/>
  <recurse type="down"/>
  <print mode="skeleton"/>
</osm-script>

Now click Run. Overpass will work for a few seconds, and then display the results on the map. You can click the points to get more info on each feature. Now click the Export menu, then the Data panel, and finally GeoJSON. This should trigger a download of the results, and a display of the results in the dialog (some browsers won't automatically download, so copy/paste the GeoJSON from the dialog into a file, and save locally). Note the location of the GeoJSON file, we need it for the next step.

Groom GeoJSON

  • What you need: The GeoJSON created with Overpass Turbo.
  • What you get: GeoJSON with image preview links

To make our map, we need not only links to the Flickr page for the uploaded images, but also a link to the preview image itself. The GeoJSON Flickr Groomer is a simple tool to create those links.

Open the GeoJSON file downloaded from Overpass Turbo in your text editor. Copy all the content, and paste into the first text box. Click here to Run, then Download groomed-geojson.json. Note the location of the GeoJSON file, we need it for the final step.

Make a Map With TileMill

TileMill is a tool for designing maps. There's a lot to it, so do check out the Crash Course, and other guides on the TileMill site. Install TileMill, start it up, and create a new project; uncheck Default data. Now, in the lower left, click the Layers buton, then Add Layer. For the Datasource field, browse and find the GeoJSON file downloaded from GeoJSON Flickr Groomer. Click Save & Style, and you should see a map of your points on the left side of the screen. To zoom to the area of just your data, reopen the Layers dialog, and click the magnifying lens next to the layer.

On the right, is CartoCSS, used to style your map. We want to make some small changes, making the background of the map transparent (by default it's sea blue). Here's what our CartoCSS looks like:

Map {
  background-color: transparent;
}

#images {
  marker-width:6;
  marker-fill:#f45;
  marker-line-color:#813;
  marker-allow-overlap:true;
}

Now, add Interactivity, the dialog is reached by clicking the small hand in the lower left tool set. Click Full. Here you can specify what appears when you click on a point. All of the tags for these features from OSM are available, you'll see them in a list at the bottom of the dialog in "Mustache" tags. We set the content to this below, to show the name of the feature, and a linked photo.

{{{name}}}
<a href="{{{image}}}"><img src="{{{image:href}}}" /></a>

The map is ready to share! Sign up for a free account on MapBox. Then back in TileMill, click the Export->Upload option in the upper right, configure the settings for the upload (try to minimize the upload size), and Upload. You can watch the progress of the export and upload, and then get a link to the map.

Here's ours!

Results and your challenge

So this is it! To get the streets background, you'll need to dive a little more into MapBox hosting configuration. Try it out on your own, with say, 10 images. If you have any questions about this guide, suggestions to improve it, or just want to share your results, get in touch. Thanks!

Is the OpenStreetMap Rails App Appropriate for Other Data Sets?

Posted by mikelmaron on 19 June 2013 in English (English)

I've begun technical advising on the next iteration of a collaborative mapping project, to collect, discuss and disseminate data and stories on deforestation in Democratic Republic of Congo. I first began thinking about it over a year ago, with this write up on Moabi and GeoWeb challenges, and have since helped WWF iterate on the old platform with Sigaptaru. Right now, I'm at the offices of IIASA, a global scientific research institute situated outside Vienna, in a castle. A few weeks from now, I'll be in Kinshasa, challenging most all of my assumptions about the project.

Right now, I want to challenge some of the technical direction this project is taking, and I hope you can help. I may be in danger of seeing every map through an OpenStreetMap lens; though it's possible I may be on to something.

Naturally, I want this project to be based on existing, active open source projects. Architecturally, we favor focused components integrated through appropriate data sharing and APIs, over a monolithic system. In Moabi, this means good functional separation between data management, and exploration presentation and communication, with links back to dig in if desired. We've starting talking about this as the difference between the kitchen and dining room (though our kitchen would be open plan, and anyone can come in and cook. The analogies are endless). There are groups of folks already collaborating on creating and sharing DRC data, but "bilaterally" and without a full community or tool for coordination. The data collected in Moabi is relatively specialized, like details of mineral right concessions, REDD+ project boundaries, field surveys, artisanal logging sites, proposed road projects, etc.

So the model of collaboration and collection of geographic data does have some parallels to OpenStreetMap (if a smaller potential contributor base), but the data itself does not always fit into what's appropriate for OpenStreetMap. Does it then make sense to fork the OSM rails port, and build off the same code base?? And further, model the community interaction and experience of working within this amazing project for the past 8 years? OSM is certainly the most successful mapping collaboration out there. MediaWiki, the software behind Wikipedia, is reused a tremendous amount. Perhaps there's a similar dynamic possible. The USGS took this approach when looking to work on the National Map. And it's not just the rails app, but the leverage of the entire architecture of planet dumps, map tiles and indexing, the great new iD editor, a well developed import process, flexible consensus based discussions of representations in tags. As the OSM app and associated tools get better, a system based off it can improve in parallel. The National Park Service has been working on beautiful and usable ways to distribute their maps as Park Tiles, and we could model that as well.

The question is, what's missing in OSM that's part of the Moabi requirements, how hard would it be to build these new features, how difficult will it be to maintain a branch, and how much of these features are useful to OSM core?

Here's some of the departures from current OSM that we've discussed. There needs to be some internal definition of a feature layer, and we may need permissions or at least focus on particular tags on particular features for particular user groups (ie some fields may be imported but need not be edited in Moabi). Perhaps the way iD is representing forms for objects could be useful here. Imports need to be properly marked and sourced; this might mean flagging user accounts as import or bot accounts. Collaborators need a space to discuss the focus of data campaigns, which could be a geographic region and/or a topic, a way to set a number of data goals and monitor progress of those goals. Some of those goals could mean a kind of quality control.This could be an implementation of Groups, as we started during the SOTMUS code sprint, integrated with an enhanced Tasking Server. A group might have a leader. Feature layers are published as tile sets, which are composited together by the tile server in different ways. A kind of CMS could power the front end, displaying various maps, highlighting useful work, telling stories behind the data, sharing out.

What's the alternative to building off OSM. Well, I guess that would be GeoDjango. Not a bad start, and flexible.

Would love to hear thoughts from those of you who experienced with building web mapping applications, developers of the OSM website, and anyone else who thinks this is a great idea or the first step towards painting ourselves into an OSM shaped corner.

Location: 48.068, 16.358