OpenStreetMap

mikelmaron's diary

Recent diary entries

Excited to Say, I am a Presidential Innovation Fellow

Posted by mikelmaron on 15 September 2014 in English (English)

Today, it can finally be said, I am a Presidential Innovation Fellow at the State Department working on OpenStreetMap for Diplomacy.

This is very exciting, and honestly a bit boggling, how it's all turned out.

9 years ago, I was living in Brighton, UK, and travelled to Nottingham for several days of hacking with some very creative technical people. Invitation was from Ben Russell, "author" of the Headmap Manifesto (read this). Ben was a kind of hero to me, so that was great, and we spent a lot of time with Steve Coast, I built a slippy map for OpenStreetMap. We blew our own minds. Ben summed it up ... OpenStreetMap was going to totally succeed, or fail spectacularly.

The people I've met through pursuing this crazy dream of OpenStreetMap, the adventures, the real places opened up ... I won't even try to sum up how this project has taken over my own life, and changed the whole world for the better. Just this. When we first started talking about OpenStreetMap for Disasters, the response was sometimes polite, often condescending, and always bewildered. Today, Humanitarian OpenStreetMap Team (HOT) provides geographic data direct to the ebola response. We didn't ask permission, but believed, listened very carefully, kept working, and created something entirely new.

And this year, the White House and State Department created a Fellowship for OpenStreetMap. The Humanitarian Information Unit (HIU) has already moved mountains with Imagery to the Crowd, providing satellite imagery for digitization into OSM for humanitarian response. Makes whatever comes next seem easy! I'm really looking forward to working with HIU on MapGive, with eDiplomacy, and with others at State, USAID, and other agencies working with OSM. Our class of Round 3 fellows are an impressive bunch, working on hard problems. We've met many folks at our sister groups, 18F, US Digital Service, White House Office of Science and Technology. There is a vibrant spirit here, true believers in transformation of government, but rooted in smart, sobering, hard work and commitment.

The problems facing the world, humanitarian, climate, environmental, economic, are so extreme, everyone must find a substantial way to work together. No matter if you are a slum dweller in Chennai, or a bureaucrat at the World Bank, you contribute to the same database in OpenStreetMap, and that is powerful. In fact this is the role of HOT, to serve as the interface between the quite different worlds of the OpenStreetMap community and humanitarian response, and bring people together. And this is why I put myself forward to serve as a PIF. There are entirely new ways to cooperate and organize now open to us, and while the answers are certainly not obvious, I felt a calling to explore the landscape of potential.

The specifics of what I'll work on are still being created. But all our discussions so far focus on supporting growth of OSM, building as usual in the open, with reusable work for all. I think there will be three parts. First, contribute to curriculum and guides for teaching OSM, and organizing OSM projects and communities (the "softer" stuff, mostly in our collective heads). Second, build software tools to support community organizing, improve data quality, and flow of imagery. Finally, trial all of this somewhere out there, including connecting with other groups working for OSM in DC, through cross agency and organizational collaborations.

I needed to make a few changes to focus on the PIF. I'm stepping down from my role as President of the Board of HOT (but will remain a dedicated Board member). As well, have minimized other commitments at GroundTruth Initiative, Map Kibera, and Moabi. Yet, I'm not going far. I'm still editing right here, contributing on GitHub, publishing, probably more than ever. And especially listening. In a very fundamental way, I can't do this fellowship alone, and rely on the energy and ideas of our collective. I will need your help.

With nothing else to say at the moment … Maps!

Disclaimer: Views are my own!

Moabi at State of the Map US

Posted by mikelmaron on 17 April 2014 in English (English)

Update: we had an excellent time at SotM-US and the Sprint day. The presentation slides and video are now posted.

The Moabi development team is excited for State of the Map US this weekend. We are sharing a preview of the new Moabi (to be fully launched on Earth Day), and presenting our work on Sunday at 4pm, OpenStreetMap as Infrastructure, sharing the stage with the USGS National Map Corps project, and NPS Park Tiles. Hope to see you there! And if you want a demo any time this weekend, find one of the team, Sajjad, James, Leo, Chippy (virtually) and myself.

First why Moabi?

Moabi-DRC is an independent mapping initiative that collaboratively monitors land use in the Democratic Republic of the Congo. Our community works towards a more Transparent, Equitable, and Sustainable future for the environment and people of DRC. You can use Moabi DRC to explore, share, and create projects on a wide range of issues from REDD+ to community mapping and more.

Why OSM as Infrastructure?

OpenStreetMap's render stack, editor and web application can be used to power collaborative mapping efforts beyond OpenStreetMap. OpenStreetMap's software is unique and powerful as infrastructure for building communities of contributors. What happens when OpenStreetMap software is reused for new data sets and communities beyond OpenStreetMap.org?

We've been working on customizations like...

Preset Editor for iD

Tile management through GitHub and a OSM TileAPI

Map Story Building in OSM

Showcase Map Sites in Jekyll/GitPages

So much more this weekend, see you!

Running for the HOT Board

Posted by mikelmaron on 21 February 2014 in English (English)

I'm running again this year for the HOT Board, my third election. I sincerely ask HOT Members for their vote and the opportunity to serve them in a fourth year on the HOT Board.

My history with HOT is now getting long, and my dedication to HOT stronger than ever (not difficult, with HOT more amazing than ever). I appreciate Robert Banick's nomination and touching on some of that. I've also recently told the story of my HOT Year, including my time spent directly on HOT Board, which as Heather illustrated, is a substantial time commitment. While it's varied recently with time off for my family, pretty fair to say that I voluntarily dedicate 20% of working time to HOT.

So like many of us, I have so many ideas for HOT, it's hard to know where to start. Here's my priorities for myself over the next year, touching on where I think HOT needs to focus and grow.

Raised many times in discussion lately, our Bylaws need to be updated, to better reflect the shape of our community, and clarify ambiguities. I led the process of developing the HOT Code, and am ready to facilitate the process with all interested members for the Bylaws.

Connected, there are many HOT processes sitting collectively in our heads, that don't need to be enshrined in the Bylaws, but would be good to get these out objectively for us to better coordinate, in lightweight documentation. One recent example, would be a guidelines for administration of the tasking manager. We don't want to over proscribe the process, but as quickly and clearly collectively illustrate the minimum we need to know.

Building on that, the HOT Guide is something I'd like to help move to first version. Ourselves as members, our broad volunteer community, we all have so many skills to contribute, many outside the usual line of the mapping we do so well. Making it easy for everyone to find ways to contribute to HOT is what the HOT Guide is about.

Something we have not done consistently in our project and activation work is evaluation, yet incredibly valuable for us to steer our work. I want to help enshrine this practice. HOT as an organization itself came out of a series of evaluation and strategy building following Haiti. The recent Haiyan OSM Assessment gives one example of ways we can begin to examine our work, and I hope over the coming months, events and groups in Philippines and elsewhere can also contribute to our learning.

Humanitarian partners are what make our mapping work relevant to disaster response and preparedness, and those relationships are a big part of what HOT manages. We've reached a profile and gravity where there are a lot of organizations involved in what we do beyond just using the data, and great potential for coordination. Tapping into this more and figure out what this looks like is something I want to pursue.

Finally, want to step up more in helping our Board be productive, continue to serve in leadership of the Board, and facilitate our process and communication and meetings well. A big part of that will be collectively setting our strategy for the year, and increasing bidirectional communication with members and the community, especially with more frequent reports on our activities and finances.

Thanks for reading all of this, and considering a vote for me.

My HOT Year

Posted by mikelmaron on 5 February 2014 in English (English)

We've just seen a phenomenal group of folks join, the Humanitarian OpenStreetMap Team. And along the way, we are reading excellent stories from the nominations and new members recounting their personal HOT history, wonderful to learn about new folks and familiar faces.

HOT has adopted a long awaited Code of Conduct, and that's planned to be ratified as part of the upcoming Annual Meeting and Board Election. The Code makes clear how HOT membership operates, the rights and responsibilities that come with membership. Membership signifies another level of dedication to the incredible work of our community. One detail there is our expectation to share publicly our yearly contributions, for our quick growing and widely dispersed community, and for all partners and supporters, to have a window into our HOT lives.

At the end of each membership year, members will be asked to document their contributions to HOT over the past year, and their aspirations for the next

So here's My HOT Year. It got kinda long.

HOT Board

My main role in HOT is as Board President. Part of that responsibility is to organize and chair the monthly meetings, with the support of the Secretary. Board meetings are for making salient points on issues, making decisions or voting as needed, and assigning actions to complete outside of the meeting. We also discuss and report and decide on straight forward issues over email and OpenAtrium continually. Our duties as Board Directors are above all to hold the fiduciary duty of the HOT, working closely with our Executive Director Kate Chapman. We have oversight over HOT's finances and projects and fundraising. We help set strategic direction and draw up policy, to encapsulate as close to consensus as possible the operations, processes, and structures of HOT. We discuss emerging ideas and issues for the organization. We oversee membership processes. And we do this all while being active members of HOT. Yet it's important to note, our Board responsibilities are quite distinct from other activities in HOT; just being a Board member takes a great deal of dedication and time. I so appreciate our Board Members whom have really taken on this major responsibility.

One of my main projects for the Board this year was leading the writing of the Code of Conduct. This was a major group effort, and I'm really proud of the results. Big thanks go to those that contributed words and comments, Russell Deffner, Katrina Engelsted, Jaakko Helleranta, Tim Waters, Kate Chapman, and Joseph Reeves. I'm especially happy that Russell's efforts led to his joining as a HOT member.

Through the Board, I also took on responsibility to get the Tech Working Group up and rolling, holding regular scrum style meetings and getting our project management tools (otherwise known as a list of GitHub repos) in order. And before too long, was able to hand over to the capable hands of our new system administrator, Dražen Odobašić!

HOT at OpenGovHub

By "virtue" of being in DC, I'm the constant presence at our "world geodomination headquarters" in the OpenGovHub. We have lots of folks come through. HOTties, if you ever find yourself in DC and need a place to lay your laptop, you are so welcome. Less excitedly, I receive HOT's snail mail, and occassionally deposit a check or scan some important mail for our admin Kristen Egermeier or "file it away".

Many of HOT's partners are based in DC, so I have regular contact, some planned some serendipitous, with good folks from State HIU, USAID, World Bank, American Red Cross, Peace Corps, Wilson Center, GWU, among others. This is often just catching up, but sometimes leads to substantial things for the Board and ED to consider, so I then share for deliberation.

Tech

Outside of the Board, I spend a bit of time on technial issues ... though not as much as I'd like, it's a lot of fun to get into our code.

State of the Map US allowed a group of us to get together and start coding for HOT, focused on upgrading and relaunching the HOT Website and adding iD support to the OSM Tasking Manager. That continued through the OSM Birthday Sprint, which was great to co-organize with Kathleen Danielson. The website relaunched in December with the Drupal muscle of Felix Delattre and Clara, and is already improved with multilingual support. Steve Kashis dove into the OSM Tasking Manager, and made immediate progress, a tribute to the clean software design laid out by Pierre Giraud. The iD developers were very supportive, adding parameters to support OSMTM needs. Just today, Pierre, with a little help from me, finished up the iD integration by automatically loading imagery from the OSMTM.

I'm excited about the tasking manager especially, and think there's lots more it can be developed to do this coming year.

For LearnOSM, I facilitated improved documentation for translators. Daniel Joseph from the American Red Cross did an amazing job of synthesizing the process into something approachable for newcomers to GitHub.

Finally, I collaborated with Lars Bromley from UNOSAT and Edouard Legoupil from UNHCR, to import Zaatari refugee camp into OSM. This was a fairly interesting technical challenge, in that it involves synchronizing updates over time. And a solid example of collaboration across agencies within OpenStreetMap. Folks like Lars and Edouard, working within humanitarian agencies, are doing the truly heroic work of moving mapping mountains, freeing data for humanitarian benefit.

Outreach

SOTM-US HOT BOF

State of the Map US was a highlight. Schuyler Erle led a presentation on HOT, with contributions from Amy Noreuil from our partner USAID OTI, then EUROSHA volunteer and now HOT member Jorieke Vyncke, and surpise guest Presler Jean from Haiti. This led to a great Birds of Feather session.

I did less travelling last year. But one place I did get to was Democratic Republic of Congo, and got to finally meet Claire Halleux, who's about as bad ass as they come. It's a small mapping community in DRC, and we had some interesting meetings with the UN while I was there. Claire, we still need to import MONUC's road data!

Peace Corps

Closer to home, I connected with the Innovation Team at the Peace Corps, who are keen to see how Peace Corps volunteers can incorporate OSM into their service. We held an event and editing session at Peace Corps HQ, which led to a piece in the Peace Corps magazine. They've continued with editing competitions, and I've made 3 presentations/interviews remotely to trainings in Senegal.

Finally in the media, represented HOT and OSM in two excellent magazine pieces, in Wired and New Scientist. During Haiyan, I stepped for a couple interviews on NPR and PRI.

Next Year

Next year is now. With a growing community and more responsibility than ever, HOT can do more and will be called on to do more. For me, I want to focus on more clear ways for people to contribute to HOT; mapping and coding definitely, but in other organizational ways as well, in translation, in coordination. That means really good documentation of our processes and policies, a really clear understanding of our resources and how we work with the broader OSM community and partners, and better communication within our rapidly growing community.

Communication can be hard. We've had our share of hard knocks in communication this year. The source is often simple misunderstanding. If ever something arises which concerns you about HOT, I am personally always accessible to listen and share my point of view. Please do get in touch.

We also need to work on our Bylaws. The Code of Conduct process, and membership renewals, showed gaps in how we want to operate, and our present legal structure. This will be hard work, and we'll need some serious time and effort on it.

That all sounds serious, but seriously, after all, HOT is the most fun and valuable thing I do. Connecting with this global community of map nerd do-gooders, and moving serious mountains ourselves, seriously changing the world, HOT absolutely, thank you for the year, the years, and the year to come.

GeoGit and GitHub Geo

Posted by mikelmaron on 26 September 2013 in English (English)

As I've been exploring the OSM rails app for other data, Git has hovered in the background of my thoughts, and I've been watching GeoGit and GitHub Geo Features closely. The conceptual basis of Git, distributed version control, solves issues we come up against regularly in OpenStreetMap, like how to keep an "authoritative" data source and community data in sync or how do we support offline editing, in areas with bad or non-existent net (something to explore with BRCK perhaps). As Jeff Johnson says, "OSM is a geodata repository with just a single branch".

GeoGit

Chris Holmes gives a thorough recap of BoundlessGeo's rational and work so far with GeoGit (part 1 part 2) including experiments with using git itself. Git is built around managing revisions of individual files, and hits performance issues with very large files, or very large numbers of directories (which early GeoGit experimented with, using a directory hierarchy to support quad-tree indexing). So they worked to decouple Git's set of verbs from its backend, and implement those concepts on top of spatial databases, and provide special verbs particular to interact with OpenStreetMap (or even perhaps, OSM clones). Perhaps that's comparable to git integrating with svn.

The work looks really promising, though they are still working on the internal technical challenges, and they've set the bar high, to fork the entirety of OSM including all history! They're tremendously talented, so I expect they can get there. But what then? Git without the interface and social features of github is a frustrating experience. Replicating that kind of community space for GeoGit is another tall tall order, and I expect not the first order of business for a GIS oriented customer base.

Now, if interacting with OSM clones is a core part of GeoGit, then perhaps can simply use the OSM application ecosystem for editing and socializing on an individual branch. And in that case, makes sense to invest effort in improving OSM website features for Moabi, and leverage any future interoperability with GeoGit itself.

GitHub Geo

"GitHub Geo" takes another approach. Accept the file limitations of git, GitHub and browser client displays (split up large files, if needed, etc). For rendering GeoJSON on GitHub, the limits seem to be in the 5-10 MB range. They seem to be loading the data as a GeoJSON layer in Leaflet/MapboxJS, so a performance improvement would be rendering of that data into map tiles and utf8grids. I would guess that there's already been thinking into what kind of infrastructure would be needed to support that, and they're watching uptake of Geo features before making that kind of investment.

The approach takes full advantage of GitHub social functions, and there's been some fun examples of this, and it will be great to see if the city of Chicago will accept pull requests. Just as important, the GitHub API to build applications to interact with GeoJSON files. This is what MapBox has been doing with Prose.IO to edit gitpages, and their GeoJSON.io provides an excellent editing environment for geodata coupled to GitHub, and would be the first mapping example of this pattern. (And as a mindbender, the entire GeoJSON.io site is hosted with GitPages). GeoJSON.io also has performance limits, probably similar to what we've seen with iD (I assume some of the internals are similar, but I haven't looked).

Another service matching this pattern launched this week is GitSpatial, which selectively syncs GeoJSON files in your repos, and provides a simple geospatial query API, for example an API for Kenyan constituency boundaries counties near Nairobi.

This pattern could conceivably be used to build a rendering service, that could build those tile sets and grids, or vector tiles, from very large GitHub GeoJSON files. Or a combination of GitSpatial indexing & geo API and GeoJSON.io could build the ability to edit a select area of a large file, and commit just those changes. That would I suppose require GitSpatial reassembling the GeoJSON features in order they were received, or using some convention for ordering features based on a geographic index, and committing a diff to that file. Another useful service would be visualizing geographic diffs, something that OSM itself (or anything really) doesn't do particularly well. Though that feature is something I could expect from GitHub soon, seeing that they are now doing 3D file diffs.

Large geographic data collaboration today

This is all so new and untested, and so far, not really built with large data sets in mind. Unlike the OSM architecture, API and ecosystem, which can pretty solidly handle loads of data and provide lots of services. It's hard to get this kind of glimpse of the future, but also have the needs of today to grapple with. For now, I reckon OSM is a really good place to experiment and build, while we'll keep a close eye on these other approaches.

Tags are all too human

Posted by mikelmaron on 11 September 2013 in English (English)

This is a short write up of a small amount of thinking and investigation of potential ways Moabi could extend and contribute to tagging tools in OSM.

Tags are all too human

Tags are the magic tools that allow OSM to happen. There's no systematic restriction to how features are tagged. Rather it's based on experience, convention, conversation, invention. There's no universal representation of the world, and tags permit that. It's messy, not always rational, sometimes absurd like horse=yes, but also mapping social facilities in refugee camps. Using the common ones becomes an exercise in crawling through the wiki. Inventing new tags is a whole other adventure.

And so is software

Though, we make it easy on ourselves, and make tools to work a little bit with tags. There are presets for editors, style sheets for rendering, analysis tools.

How are tags read and managed by software? It's still an (impressive) grab bag.

Editors

For iD, presets are defined as json files. Each presets specifies a tag, and other tags to fill out for that feature ("fields"), as well as synonyms for search and a logo. It's a clean representation, focusing on the "data structure", and the collection of features is managed through github. Afaik, there haven't been many attempts yet to create custom sets of features. The would require compiling and hosting your own version of iD, there's no way to do it directly in the website.

JOSM presets have been around a while. It's an XML format that defines both tags and fields for features, and sets of features in a hierarchical menu, as well as labels and layout of dialogues. There's dozens of preset files out there, HOT maintains a set for the Humanitarian Data Model, Map Kibera has created them, tons more.

Forever, it's been a pain to create these JOSM presets. Hand writing fragile XML is not so fun. The http://visualtags.hotosm.org/ is a tool to create these presets. Here's the code in github. Definitely a step up from hand crafted XML. Though still a little challenging working through the interface.

Rendering

Visualizing, there's the classic tool chain of osm2pgsql and mapnik and now carto for transforming tags into cartography. Other tools, including editors like JOSM, use the similar but different to carto MapCSS.

And..

There are some QA tools which monitor tags, to look at common issues in how tags are assigned, how they are used in combination with each other, etc. There's Validation tool built into JOSM. Keepright has a set of tag related tests. But there's not really a framework out there which allows for general application of quality checks on tags.

And of course, there's the mighty taginfo, which builds up global statistics on tag use, builds lists of combinations and geographic spread, and provides APIs for linking into other places tags are handled, like the wiki and editors, and links to other useful tools for querying on tags like Overpass Turbo. TagInfo is a sinatra app that others have set up, to focus on particular geographic places.

And what about Moabi?

Moabi wants to provide a bit more structure around tags. A group in Moabi (based off the OSM groups sketch perhaps), or a user, want to create a Feature within Maobi. That Feature is then made a searchable option in the editor. When a user creates or modifies a Feature in Moabi, it will be validated against that definition, and the results shown on the "browse" page. Further, there may be permission levels set on who can CRUD a Feature, or even at the level of specific Fields within a Feature. As well the definition of a Feature might have its own access rules.

Essentially, define, validate, visualize and restrict tags.

Pretty different then OSM right? Is and where is the overlap, and how do we best diverge?

And where do we do this again?

Honestly, I've always thought a central place to manage tags would be really useful. Well not to definitively enforce tags. But a place more functionally tuned to the problem then the wiki, that could provide some of the process of tag approval (even if tag approval has no functional meaning!), and then semi-automatically produce and integrate with the editors and rendering tools. TagInfo seemed like not a bad place to at least acquire some of these features ... again without changing the fundamentally free flowing nature of tags. Just make it easier for us to track what's going on.

Representation

Though "tags" are different from "features". One nice thing about the iD representation is that it introduces this concept. JOSM presets has this as well, though it includes groups of features, whole menus of features, along with layout and labels. iD has json just define that single feature, which is built up of references to reusable fields.

Editing

Somehow, I prefer the iD representation, cleaner to build off. But then, does it need to be represented in a file? Where would the creation of features take place? It could take place in an external application, like the Visual Tag Chooser, or TagInfo, or something else. It would then require some secondary transformation and distribution steps to make its way into the OSM applictions, to be used in editing, validation, and rendering.

The other option would be a MVC of a Feature directly in OSM, taking advantage of any group or user permissions developed there, etc. We'd still have some transformational steps to bring those options into iD (by producing those json files, or building in a step in iD to get options from the DB), as well as to produce custom tiles more easily. Could possibly even make use of some of the interface components of iD itself to manage the editing here.

Validation

Validation during editing could be accomplished by iD providing some feedback during Save based off the Feature representations it has loaded. If that was structured properly, that JS validation could be isolated and reused as a part of the browse page. In this model, as usual, feature components are not enforced, but are made more visible in more places.

Restriction

If Features, or even particular Tags within a Feature, have some access restriction, that would need to be included in the Model and/or representation. Perhaps Groups are a natural way to strucure these permissions, or even Roles within a Group, or a Platform as whole. For Moabi, might even need to apply restrictions on the geometry of an object, only allowing it updated through particular import accounts, and just allow editing on other particular fields.

There's two issues. This kind of access restrictions can be applied sometime during CRUD, and if a problem is detected, simply reject the entire Changeset, similarly to how editing out of data OSM objects throws a "conflict". The other is, how can Features be identified as Features, and not just a collection of tags. For that, we might use some kind of "hidden" tag, as JimmyRocks suggests in the comments, which specifies the Feature, and can link to restrictions defined elsewhere. This idea could also be expanded to manage "approval" of contributions, so that within a Group, some members can contribute, others can certify.

So where to start? I reckon, we start with what's more potentially generally useful, like tag definition and validation, and then move on to the more restrictive non-OSM options, when they're really more clearly needed. But might also make sense to start with the least OSM like features, to really make sure that this whole direction of using OSM for non-OSM data makes sense.

UN Collaborates on Zaatari Camp Data in OSM

Posted by mikelmaron on 4 September 2013 in English (English)

Zaatari, in northern Jordan, is facing huge humanitarian pressure from the Syrian conflict, considered the second largest refugee camp in the world. To aid the response and improve coordination of geographic data in the camp, a remarkable cooperation is underway there among UN agencies, the OSM community, and potentially, the refugees themselves.

Getting Here

As the Syrian crisis deepened, HOT was prompted to activate in January. It's naturally proven difficult to coordinate remote mapping inside Syria, though there has been significant and interesting activity over the past two years.

We've known Edouard Legoupil from UNHCR for a while, through even pre-Crisis Mappers days. We personally overlapped in Nairobi at the start of Map Kibera, and since, we've discussed ways of bringing open approaches to mapping and engagement in camps. Likewise, Lars Bromley from UNOSAT has been a colleague since we collaborated together in Darfur Google Earth layers project, and while he was at AAAS, supplied detailed historic satellite imagery of Kibera for our analysis of spatial extents of post-election violence. Both have innovated, very close to where things are built, and have created space within their organizations to work collaboratively. With the HOT activation, we've been able to come together and do practical work on those ideas for Zaatari.

The unhrc-jordan OSM user has imported roads, toilets, water points, health facilities, mosques, schools, offices, even restaurants. Another partner organization, UNICEF Jordan also uploaded child specific infrastructure (though this was later undone, and re-uploaded by UNHCR for some technical reason). Now, UNHCR is working with REACH to map on the ground in Zaatari, both confirming data and including "informal" infrastructure built by refugees; all the normal services of a city are growing in this place. In the future, there may be opportunity to engage refugees themselves in the processes. In traditional practice, data is shared through colleagues. Through OSM, they become openly collaboratively shared.

UNOSAT Import

UNOSAT is tasked with another piece of the geographic puzzle in Zaatari. They analyse satellite imagery to identify ALL structures in Zaatari, currently at 28174 structures. Lars confirmed that this data could be imported into OSM as well, and that I could help with that process through the UNOSAT OSM User.

Working off the July data, the first challenge of the import was transforming the ESRI Geodatabase released by UNOSAT into a format I could work with. Luckily, I was at OSFAC's office in Kinshasa where I had access to ArcGIS, and could do the conversion to Shapefile. Then, to my surprise, JOSM was able to read the Shapefile directly! In JOSM, I used the under-appreciated filters feature, to select only features with Shelter_St=1 (meaning the structure currently exists, the data contains shelters that no longer exist). Then, mapped the Structure_ column to either shelter (for 1) or administrative (for 2). Also stored the acquisition date, event code, and source, for reference (within a unosat: tag space). And also, the objectid, which could be used for applying updates later on. Removed the other tags. And because of the large size of the upload, used a script rather than JOSM. Here's an example structure from the import.

UNOSAT Conflation

So right before I was ready to do that import, Lars notified me that UNOSAT had released a new analysis. I could just redo the process above with the new data, or use the opportunity to test if it works to update based on unosat:objectid. Yea, did that!

Again, stuck with a GDB, until I was fortunately introduced to the simple and strong GDB Flee. From there, I went about writing a conflation script, which reads in the shapefile and OSM data, detects actions (create, update, or delete, either way), and then outputs OSM XML for examination in JOSM. There might be other tools to do this (please let me know in comments if so), but was useful for me to get my head around the process, a first step at abstracting conflation, and dig into some geo mangling code.

It's not fully tested against all cases, just the situations in this conflation, so putting together proper tests for this later is on the agenda. In this case, after the deletes and creates, the remaining number of structures added up precisely to what UNOSAT reported, so looks fine.

The thing to grapple with here are unexpected situations. The script deals with this by outputting OSM features with a NOTICE tag. Then in JOSM, can use filters to only show those features, and inspect and make decisions manually. For this import, there were about a dozen features that were supposedly collected in the previous analysis, but hadn't already been imported. This was strange, but didn't investigate further and since the numbers added up, decided to include them in the import.

So What?

Interesting results immediately came out from comparing multiple datasets, and putting many eyes on the data. Those dozen "missing" features were highlighting in JOSM Validation as Nodes at same position. The precisely same lat/lon, but different objectid tags. I confirmed they were definitely present in the current set of features, and that the total number of featues matched what's reported by UNOSAT. So perhaps I can conclude there may be duplicates in the source analysis, and hopefully this community view on the data can help improve the authoritative view.

Also interesting to look at how the UNHCR and UNOSAT data interact. Administrative buildings or compounds are polygons in UNHCR, points in UNOSAT. Some compounds contain multiple administrative structures; it might be interesting to associate them via a relation or tagging. Some shelter structures show up with administrative areas; do those structures need to be reinterpretted, or have the areas changed? Now that these are together in a single database, we can script or use GIS to analyse those kind of issues and prioritize further investigation.

One thing that might help is for OSM users to access the imagery directly. In fact, the most recent analysis come from State Dept HIU supplied imagery of Zaatari, so we can perhaps request access through Imagery to the Crowd. We'd want to have some good analysis first, and a clear idea of how mappers could help.

Finally, there's potential to do custom rendering which highlights all the data, and camp specific information. UNHCR is already rendering Zaatari on their data site. It would be good to also show the node buildings in an unobtrusive way, to show how the admin structures and shelter distributions interact.

Location: 11, Zaatari camp - District 11 - Street 22, 11, Mafraq, JORZATD011, Jordan

Skill Share: Map Photos Using OpenStreetMap and TileMill

Posted by mikelmaron on 14 August 2013 in English (English)

Recently, we organized a fun skill share with Map Kibera, and then again with Transparent Chennai. The idea is to use OpenStreetMap and TileMill to create an interactive photo map of OSM features. Here's the simple result from working with Map Kibera.

Want to learn to create something like this? This guide takes four steps, demonstrating an overview of the whole flow and connection between four tools to build a photo map. This guide will hit the main points, highlight how the tools connect up with open data, and I'll just link to other resources for the details on how to use each.

Note, this guide is to create photo maps of objects mapped in OSM, not arbitrary points where you have happened to take photos.

Upload Photos to Flickr

  • What you need: A photo of a place to map.
  • What you get: Link to the photo page and photo preview.
  • Learn more: Flickr Help

If you don't have a Flickr account, create one, it's an easy way to share images. Upload a photo of a place. For our example, we uploaded this photo of the iHub in Nairobi, taken during the Uchaguzi election project. We created a Map Kibera group to hold all our collective photos. MK mappers get in touch with me directly if you need access.

The result you should have is a link to the photo page. Something like:

http://www.flickr.com/photos/mikel_maron/9420814521/

Edit OpenStreetMap

  • What you need: Location of your photo.
  • What you get: OpenStreetMap feature linked to the image.
  • Learn more: LearnOSM

Now, over to OpenStreetMap. Sign up if needed, and review the basics in LearnOSM. Also, try the iD Walkthrough: navigate to an area, click Edit (specifically, click the down arrow, and select "Edit with iD"), and follow along the editing tutorial.

Now open the View tab again. Enter the name of the location of your photo, and search. In our case, it's "iHub". Look through the search results, and find the correct feature, there might be more than one feature with that name, and then click View Details to open the browse page for that node. This page shows the location and tags for the feature, and in the lower right, click the Edit Node link.

Often you'll need to add a new feature in OSM, rather than add to an existing feature. Features should be appropriate for OSM, not just an arbitrary point where your photo was taken. Check out the other guides to learn more.

In the left panel, click All tags, then the +, to add the tags to link the image on flickr. Add "image", then the link to the Flickr photo page. It will look like:

image=http://www.flickr.com/photos/mikel_maron/9420814521/

Click Save, add a commit message, and click Save again.

Caveats

Important: When you are taking photos, make sure to record which photo goes with which object! It can get confusing if there's a lot.

Extraction with Overpass Turbo

  • What you need: A Deep Breath (take one now!)
  • What you get: A GeoJSON file containing images and locations.
  • Learn more: Overpass Turbo wiki page

The OpenStreetMap website presents one view of the data you contribute. The powerful thing about OSM is that you can take the data and use it however you like. Extracting the data from OSM itself, for use in another tool, is what this step is about.

Open Overpass Turbo with the template to search for nodes with key "image". You can find links for other tags by exploring TagInfo, another great tool I'll cover in a later skill share. The interface has two panels, on the left the "Query" and the right the "Map". Search and navigate the map to the area where your features are located. For us, the is "Nairobi".

The query to do the extraction can get complicated; you'll see the template settings on the left, looking something like.

{{key=image}}
{{type=node}}
<osm-script output="json">
  <query type="{{type}}">
    <has-kv k="{{key}}"/>
    <bbox-query {{bbox}}/>
  </query>
  <print mode="body"/>
  <recurse type="down"/>
  <print mode="skeleton"/>
</osm-script>

Now click Run. Overpass will work for a few seconds, and then display the results on the map. You can click the points to get more info on each feature. Now click the Export menu, then the Data panel, and finally GeoJSON. This should trigger a download of the results, and a display of the results in the dialog (some browsers won't automatically download, so copy/paste the GeoJSON from the dialog into a file, and save locally). Note the location of the GeoJSON file, we need it for the next step.

Groom GeoJSON

  • What you need: The GeoJSON created with Overpass Turbo.
  • What you get: GeoJSON with image preview links

To make our map, we need not only links to the Flickr page for the uploaded images, but also a link to the preview image itself. The GeoJSON Flickr Groomer is a simple tool to create those links.

Open the GeoJSON file downloaded from Overpass Turbo in your text editor. Copy all the content, and paste into the first text box. Click here to Run, then Download groomed-geojson.json. Note the location of the GeoJSON file, we need it for the final step.

Make a Map With TileMill

TileMill is a tool for designing maps. There's a lot to it, so do check out the Crash Course, and other guides on the TileMill site. Install TileMill, start it up, and create a new project; uncheck Default data. Now, in the lower left, click the Layers buton, then Add Layer. For the Datasource field, browse and find the GeoJSON file downloaded from GeoJSON Flickr Groomer. Click Save & Style, and you should see a map of your points on the left side of the screen. To zoom to the area of just your data, reopen the Layers dialog, and click the magnifying lens next to the layer.

On the right, is CartoCSS, used to style your map. We want to make some small changes, making the background of the map transparent (by default it's sea blue). Here's what our CartoCSS looks like:

Map {
  background-color: transparent;
}

#images {
  marker-width:6;
  marker-fill:#f45;
  marker-line-color:#813;
  marker-allow-overlap:true;
}

Now, add Interactivity, the dialog is reached by clicking the small hand in the lower left tool set. Click Full. Here you can specify what appears when you click on a point. All of the tags for these features from OSM are available, you'll see them in a list at the bottom of the dialog in "Mustache" tags. We set the content to this below, to show the name of the feature, and a linked photo.

{{{name}}}
<a href="{{{image}}}"><img src="{{{image:href}}}" /></a>

The map is ready to share! Sign up for a free account on MapBox. Then back in TileMill, click the Export->Upload option in the upper right, configure the settings for the upload (try to minimize the upload size), and Upload. You can watch the progress of the export and upload, and then get a link to the map.

Here's ours!

Results and your challenge

So this is it! To get the streets background, you'll need to dive a little more into MapBox hosting configuration. Try it out on your own, with say, 10 images. If you have any questions about this guide, suggestions to improve it, or just want to share your results, get in touch. Thanks!

Is the OpenStreetMap Rails App Appropriate for Other Data Sets?

Posted by mikelmaron on 19 June 2013 in English (English)

I've begun technical advising on the next iteration of a collaborative mapping project, to collect, discuss and disseminate data and stories on deforestation in Democratic Republic of Congo. I first began thinking about it over a year ago, with this write up on Moabi and GeoWeb challenges, and have since helped WWF iterate on the old platform with Sigaptaru. Right now, I'm at the offices of IIASA, a global scientific research institute situated outside Vienna, in a castle. A few weeks from now, I'll be in Kinshasa, challenging most all of my assumptions about the project.

Right now, I want to challenge some of the technical direction this project is taking, and I hope you can help. I may be in danger of seeing every map through an OpenStreetMap lens; though it's possible I may be on to something.

Naturally, I want this project to be based on existing, active open source projects. Architecturally, we favor focused components integrated through appropriate data sharing and APIs, over a monolithic system. In Moabi, this means good functional separation between data management, and exploration presentation and communication, with links back to dig in if desired. We've starting talking about this as the difference between the kitchen and dining room (though our kitchen would be open plan, and anyone can come in and cook. The analogies are endless). There are groups of folks already collaborating on creating and sharing DRC data, but "bilaterally" and without a full community or tool for coordination. The data collected in Moabi is relatively specialized, like details of mineral right concessions, REDD+ project boundaries, field surveys, artisanal logging sites, proposed road projects, etc.

So the model of collaboration and collection of geographic data does have some parallels to OpenStreetMap (if a smaller potential contributor base), but the data itself does not always fit into what's appropriate for OpenStreetMap. Does it then make sense to fork the OSM rails port, and build off the same code base?? And further, model the community interaction and experience of working within this amazing project for the past 8 years? OSM is certainly the most successful mapping collaboration out there. MediaWiki, the software behind Wikipedia, is reused a tremendous amount. Perhaps there's a similar dynamic possible. The USGS took this approach when looking to work on the National Map. And it's not just the rails app, but the leverage of the entire architecture of planet dumps, map tiles and indexing, the great new iD editor, a well developed import process, flexible consensus based discussions of representations in tags. As the OSM app and associated tools get better, a system based off it can improve in parallel. The National Park Service has been working on beautiful and usable ways to distribute their maps as Park Tiles, and we could model that as well.

The question is, what's missing in OSM that's part of the Moabi requirements, how hard would it be to build these new features, how difficult will it be to maintain a branch, and how much of these features are useful to OSM core?

Here's some of the departures from current OSM that we've discussed. There needs to be some internal definition of a feature layer, and we may need permissions or at least focus on particular tags on particular features for particular user groups (ie some fields may be imported but need not be edited in Moabi). Perhaps the way iD is representing forms for objects could be useful here. Imports need to be properly marked and sourced; this might mean flagging user accounts as import or bot accounts. Collaborators need a space to discuss the focus of data campaigns, which could be a geographic region and/or a topic, a way to set a number of data goals and monitor progress of those goals. Some of those goals could mean a kind of quality control.This could be an implementation of Groups, as we started during the SOTMUS code sprint, integrated with an enhanced Tasking Server. A group might have a leader. Feature layers are published as tile sets, which are composited together by the tile server in different ways. A kind of CMS could power the front end, displaying various maps, highlighting useful work, telling stories behind the data, sharing out.

What's the alternative to building off OSM. Well, I guess that would be GeoDjango. Not a bad start, and flexible.

Would love to hear thoughts from those of you who experienced with building web mapping applications, developers of the OSM website, and anyone else who thinks this is a great idea or the first step towards painting ourselves into an OSM shaped corner.

Location: Neu-Guntramsdorf, Guntramsdorf, Mödling, Lower Austria, 2353, Austria

What a fantastic State of the Map US

Posted by mikelmaron on 12 June 2013 in English (English)

It's hard to know where to start. Most of the last four days, I wanted to be in at least three places at once.

Top of my agenda was moving forward the discussion and building of social tools in OSM, many of us were thinking the same, and we truly did make progress. Thanks so much for the great reception to the ideas in my presentation, and also happy for the interest in the slice of the early "history" of OSM. Talks by Saman, Richard, and Martijn all gave different approaches to a common vision, and I think there's broad acceptance of the direction in the OSM community. We got to work at the code sprint, on a two sided approach. Martijn, Steve Singer, Drew Dara Abrams, and Tom all contributed to the basic framework of Groups in the rails port. And Serge and Drew did some good exploratory architectural thinking into an Activity Server (OSM Antenna) to support News Feeds. I also started off on a more expressive User Profile page, including linking in JSON from HYDC (hoping to get JSONP). Lots of fun to be part of this and other ad hoc dev teams, and I really want to find ways to keep the pace on this and other development efforts.

Related were several good discussions on improving the community outreach and community tone of OSM. Alyssa Wright's investigation of gender and list participation totally hit home, and some solid ideas to address this imbalance are in formation. Also, think I recruited a couple new list moderators (Tom, Stephen, I'll be in touch!).

OpenHistoricalMap had an excellent BoF, lots of interest from historians working on real mapping problems. Jeff Meyer ramped us up with so much enthusiasm, and we set about fully getting OHM open for business. For the Code Sprint, Jeff set up our own forks on github for the rails app, mod_tile, the styles, osm2pgsql, and gotten everything current on the server. I started looking at adapting CartoCC to more easily build time aware map tiles, with the goal of manually providing time slices of 2008 and 2009 Burning Man. Tim Waters investigated adding time parameters to the maps API call, as well as better editing filters for dates in JOSM, and Sanjay was looking into time aware exploration tools of the data (along the lines of the in development historic gazateer. And Tim and Aaron worked on data imports of historic US counties and historic NYC buildings. The introduction of and work on Vector Tiles may be the ultimate way for OHM to handle time aware tile sets, by pushing filtering to the client; they'll also have application in low bandwidth environments (imagine global OSM installed on BRCK).

Another great focus of discussion was OSM in Education. Patrick Wilson presented Zombie Based OSM, and Noula and Richard from GWU shared their great work. Both sparked a lot of ideas, including the notion of TeachOSM, a simple curriculum site along the lines of LearnOSM, which Ian is charged up on. There's a lot to figure out here, but a good group. As well, the GWU presentation sparked some ideas for improvements to the Tasking Server, to make it easier to assign grid squares, based on a difficulty ranking, as well as improved individual statistics. This could eventually tie into the user profiles, giving badges to participants in HOT activations.

snkashis from Caerus took up the charge on the Tasking Server, starting to get familiar with the code base by implementing a link to iD. Peter Chin dove in as well. To support this, Tom MacWright added support for TMS in iD.

We had a good HOT BoF initiated by Amy Noreuil from USAID OTI. It was excellent to connect with Amy and hear about their experience working with HOT in Haiti, and what the future might hold. I also met Jorieke, one of the Eurosha volunteers, and absolutely great to hear about the experience from her. Presler from Haiti made a surprise appearance, was great to see him again, and see all the work with OSM he's doing at IOM. We all came together in Schuyler's presentation to give a full picture of HOT. Also good presentations from and good to meet Dale Kunce from Red Cross and Felix Delattre's work in Nicaragua. I made the plea to the assembled technically oriented hotties to join the Technical Working Group. One substantial new idea that came out of SotM was holding a small round table of agencies and organizations working with OSM in DC, to share and discuss broader strategic direction of HOT type work. As well, some early thinking on other ways to encourage coordination of working with OSM within the government. Finally, met Christine White and Bronwyn Bronwyn Agrios from ESRI, and there's going to be some great OSM stuff coming up there, starting with an event at the UC.

Connected with new folks using OSM for good in other ways. Gregor MacLennan from Digital Democracy is looking at mapping river networks in the Amazon, in order to model flows pollutants in indigenous people's territories, which overlaps with some of the notions of Open Watersheds in Kerela, flood mapping discussions with NASA< as well as mapping in other parts of the Amazon. Also learned about the New California Water Atlas, being built by Laci Videmsky and Chachasikes (finally good to meet this friend of Anselm's and developer of Lemonopoly) (update: great discussions taking place on Open Water Sheds Chris Natali from the Earth Institute recently connected FormHub to OSM, something Matt Berg and I had been bouncing around for a while, and I hope to figure some things out there. Jaak Laineste is looking at OSM to map global trash cleanup and landfills, something Liz Barry and I have been involved with before. All of these are informing my thinking on using the OSM model for conservation data in the DRC.

The National Park Service presented their full circle work with the OSM community, and while I didn't get enough time to talk to them, I finally met Nate Irwin in person, and we have a lead on connecting with Rock Creek Park; also hope to learn more about their feedback system from OSM, and perhaps see how it touches on Jeff Johnson's GeoGit work. There's a NPS mailing list. Craigslist shared their learnings with setting up tiles, and we caught up on their new experiments with Notes. Very proud to have been talking with them through the process and providing a little guidance into our community and tools. Apparently, CL poster feedback has led to significant new mapping, and I think this would be a great visualization (highlight changesets proximate in time and space to CL notes).

Was excellent so many of the core technical folks made it in from across the Atlantic. Grant delved into the details of OSM's hardware setup, Andy explained the move of the core OSM styles to CartoCSS, Frederik talked about the process of producing exports, and Tom Hughes was in attendance as well. With so many of them here, the Code Sprint was pretty effective. An idea that came up, but I didn't get chance to work on, was links to download services like GeoFabrik, Garmin exports, Overpass, from the Export Tab (would need to have metadata published on each service to direct to appropriate download spot, or use bbox, and explain fully the nature of the external sites).

With Henk Hoff, Oliver, and Steve Coast also in town for OSMPlus (which I unfortunately missed while enmeshed in the Sprint Day), was an opportune moment for the BoF on the ODbL and geocoding. I think this tricky topic was discussed really well, and there's probably a practical way forward. Also on my mind was the overall structure of OSM orgs, the roles of OSMF and OSMUS and HOT and other entities, what they're responsible for, how they operate and support themselves. Didn't get to talk as much as I'd like about federated structures, local chapters and funding models, but that can develop from here. But was good to hear Grant's relatively positive response to a question on future employment as an OSM sysadmin. Other things on my agenda that just couldn't possibly fit into all the going ons was thinking about better tool management for Tags, and a UN appropriate tile set.

And that's just what I remember right now. What an amazing few days, so much momentum, good ideas and spirit. Was great reconnecting with many folks. The organization of the conference was totally on point. I even connected with one of my favorite beers. I knew this was going to be a good conference, and it didn't disappoint. Thanks OSM!

Location: Mission Bay, San Francisco, San Francisco City and County, California, 94158, United States of America

Devouring the Landscape

Posted by mikelmaron on 3 April 2013 in English (English)

The weather in DC last Saturday was forecast to be warm and bright, and it was. I was desperate to escape the city, see expanses of budding trees and returning migratory birds, and have been planning for nature on my phone, digging into the NPS Chesepeake Explorer App since I stumbled onto the Star Spangled Banner Trail on the way to the Hyattsville Mapping party. It's a nice app, I hope they update soon to use NPS Park Tiles.

Patuxent River Park intrigued me. Trying to figure out how to bike out there, found the Patuxent Wildlife Center Loop on http://bikewashington.org/, a terrific source of information, originally inspired by biking in Atlanta for the 1996 Olympics (my friend Eric Manners and I were inspired to bike to Atlanta 96 from Chicago right after graduation, and became lifelong bicycle fanatics along the way). Would love to talk to the programmer of the Bike Washington about an update to use OSM and the latest infrastructure. So this Patuxent, was not the Patuxent I was looking for, but it sounded pretty nice.

Generated a fresh Garmin map for Maryland/DC/Virginia, and created a rough route on GPSies from the text description of the ride, and loaded that on with GPSBabel. Erica and I took the Metro out to Greenbelt, greeted by acres of very full parking lot, presumably the transport deritus of families taking the reverse route into DC for the day. Passing aspirational, exurban, mega office complexes ("your company here, 600,000 sq feet"), we turned into a still-but-not, rural but vaguely militaristic, US Property warning, federal landscape of the USDA Beltsville Agricultural Research Center.

Of course, there is a map, out in the landscape, to easily find our way to the beef, pork or chicken research stations. A landscape out of E.T., quiet and peaceful, yet clearly something was happening. Proud buildings, echoes of rolling English countryside, revealed to be the US by the gauche MIT style numbering prominent on every facade. A banner on the side of building, "Celebrating 100 Years" ... 100 years? Such a mammoth federal complex, I could only have imagined emerging in New Deal America, as the Wildlife Research Center itself was founded in the 1930s. We passed on, turned off down Scarlet Tangier Loop, and arrived at the National Wildlife Visitor Center. Tired from charging our single speed up the hills, we relaxed on the tram ride, saw beaver, turtles, and passed along Telegraph Road, along which the old Washington-Baltimore telegraph line travelled, famous for carrying news of Lincoln's assassination North (Lincoln will make another appearance in this landscape portrait later); laid in the sun as the birds and bugs came to life again in the late afternoon; and boggled at the "state of the art" amusementpark/nightmare/psychedelic/mournful education exhibit. From the command center-sh1t your pants visualizations information overload of the planet's unsustainable human development, to a hall of stuffed endangered species displayed inside room-high/triangular/funereal/obscured glass columns, to the final escape launch via a full size kaleidoscopic Koyaanisqatsi-lite video booth, and face on portraits of adorable endangered animals crying for your concern, it is a journey sure to traumatize generations of school children into deep lifelong passion for conservation, if they can ever find their way outside the exhibit again.

We turned back, not following the full ride loop through Bowie, back to the Metro, back to some of DC's fine pupusa. Here's the whole ride on Strava, including our time laying in the sun, and tooting around on the tram ride; hope I scored some of the slowest riders place on some segments!

And the map. OSM had the "North Tract" of Patuxent mapped out, traced from Maryland parcel data. The "South Tract" and "Central Tract" were missing completely. The boundary is not exactly evident from satellite imagery, and I figured as federal data, somewhere out there was a shapefile of the boundary. Patuxent and other Wildlife Refuges are part of the US Fish and Wildlife Service, and they maintain a GIS data download site. It wasn't exactly obvious how to find this on geo.data.gov, Google and human facing text won this one. Downloaded the USFWS Interest Shapefile, and opened in QGIS. Zoomed to the area of Patuxent, selected all the numerous polygons (seemed to be broken down by parcel), and merged into three polygons (North, Central, South) based on the rough map in the Patuxent brochure. North had about a dozen very tiny, still unexplained holes. Projected, then saved that shapefile, then opened in JOSM, which worked just perfectly (including setting up a multipolygon relation for the North). Tagged and merged with the existing OSM data. Fixed up the road network to the Visitor Center (mostly untouched since Tiger import), and added a few more features. There's plenty more work to do, just around the visitor center, tracing the lakes, adding walking paths, info boards, dozens of benches, and 2 beaver lodges.

USDA Original Location on the Mall

I remained curious about the USDA Research Center ... how the heck did it end up out here? What do they do? Found Beltsville Agricultural Research Center history exhibit in one corner of the delightfully Web 1.0 USDA web presence. Lincoln, a farmer himself, signed the USDA into existance in 1862. In the middle of the Civil War, a turning point in the balance of federal power (though let's not forget the real cause, Lincoln created a new federal agency, and it found its first home right on The Mall. That included ample space for livestock research; for two decades, right in the heart of DC the USDA was experimenting on cows and pigs and chickens. This must be Historically Mapped. Later the livestock research was moved east, near to the present site of the National Arboretum, also a USDA property, and hopefully the site of a mapping party soon. Then Bethesda, and finally Beltsville in 1910. Though Beltsville felt like a New Deal landscape, it's historical precedent went decades back.

And its 100+ years, BARC has pretty much created food as we know it in the country, scientific exploration of greater yield (quadrupling milk output per cow), adjusting breeds to consumer preferences (most every store sold turkey in the US descends from a lineage developed here, a 8-10 pound turkey with more breast meat) (later they discovered some of these turkeys reproducing asexually). Oh, they also developed DDT here, which later Patuxent proved harmful to reproduction of birds, including our national bird (nice government landscape symbiosis here). Don't get me wrong, I'm sure the USDA has time and again made absolutely beneficial discoveries to our agricultural. Even Thomas Jefferson said "The greatest service which can be rendered any country is to add a useful plant to its [agri]culture." Pacifists during WW2 undertook their national service here. This is the largest contiguous green area in the metro area. It just gives me plenty of pause to see what change this place has brought about in what we eat. Would be great to map the place more, using that georectified sign board, and the boundaries of government land ownership in Maryland ordered up from the USDA GeospatialDataGateway.

And that's another pass through the physical and data landscape in the DC metropolis. Ever fascinating, the process of the historic shaping of our landscape, the ways we understand symbols and concepts and their physical presence, our fleeting feelings and impressions, our communication and sharing of place, in data and stories.

Location: Scarlet Tangier Loop, Sandy Hill Acres, Prince George's County, Maryland, 20708, United States of America

StudioX Mapping Party

Posted by mikelmaron on 7 March 2013 in English (English)

Last month, we held a great mapping party in Mumbai. I was in town for TechCampMumbai, and had introduced OpenStreetMap and discussed how to use open source tools with dozens of South Asian civil society organizations. Overwhelming and inspiring, lots of grassroots potential, worthy of several more posts. However, we didn't get to do much mapping (except for adding the outline of our very nice hotel), so I was eager to get out in the city.

map of new data from the party

Good friends from ChaloBest and the OSM India community pitched in to bring something together at the very nice space of StudioX, and invite interesting civic minded folks like Walking Project to come out to learn the tech and community. I had also extended the invitation to all the TechCamp attendees, and the Bangladesh contingent came through on their tourist tour of south Mumbai (most everyone else had immediately left after the camp concluded the night before).

We started off with a my 5 minute lightning talk from TechCamp Mumbai, and Shekhar gave a tour of the latest from ChaloBest, before doing a super quick surveying tutorial, distribution of GPS units, and down to the street. We decided to focus on the immediate area of Dr Dadabhai Naoroji Road, and split into thematic groups. One group focused on the major buildings, another on transport/bus stops, another on heritage.

Myself and Rishi from the Walking Project mapped the very very local level, the literally dozens of street vendors within the few blocks we covered. We also discussed pedestrian access issues, and how mapping could help in advocacy and planning, and Rishi took dozens of photos like the one above documenting the condition of the street.

Street vendors take a few forms, like relatively substantial wheeled stands and miscellaneous accessories, like padapav vendors who have a flat surface for holding the assembled ingredients, and a separate oil cooking apparatus. (Padapav is the typical Mumbai street snack, basically a potato burger on a Portuguese style bun, with accompanying sauces and chutney. Delicious and cheap, responsible for the daily sustenance for millions). Others simply arrange their wares using the available street scape, like the vendor of interior design magazines who shelved them behind string tied tight to a fence.

You might think that street vendors are too ephemeral to map in OSM. Turns out, despite their movable physicality, they entirely dependably end up in the same spots every day, by some kind of common agreement and logic of the streets. They serve such a vital purpose, an integral part of the street life that makes many parts of India so enjoyable (at least to someone like me who usually walks the relatively vacant streets of American cities).

Of course, there's the tagging. Nothing in the Mapping Features page quite covers this case. There are things like shop=stationary, but that doesn't quite capture the reality of a street vendor. For now, I decided to consistently apply "shop=street_vendor", and add "shop:type" to describe in more detail what they are selling. (Note, we mapped a few shop=kiosk, which are distinguished by having a permanent structure). This could use more discussion and deliberation on the tagging@ mailing list, and in the proposed features section of the wiki.

Here's all the shop:type we encountered.

  • 'betel_paan' (yum?)
  • 'books'
  • 'cane_juice'
  • 'chana_bhel' (street snacks)
  • 'cigarette, public telephone'
  • 'cobbler'
  • 'fruit_chat,sweet_lime'
  • 'interior_design_magazine'
  • 'juice'
  • 'newstand'
  • 'sandwich'
  • 'stationary'
  • 'vadapav'
  • 'wallet'

Fort is the center of colonial Mumbai, so hosts a number of historic buildings, and though the walls of the Fort itself are mostly gone, the layout of the area still echoes its footprint. If only we had OpenHistoricalMap! There was this very solid map right on the street, and historic buildings plaqued at their entrances. Rishi was thoroughly stopped by an overeager security guard from taking a photograph of only the plaque, authoritatively leaning on the vague warnings against photographing secure areas throughout India. This was all with the backdrop of bombings the day before in Hyderabad. We at least avoided unwanted attention by not taking GPS points around CST/VT.

As is the usual practice, we headed to a pub after the mapping party. Shekhar painted the picture of Mumbai's past as we walked, the bustling market zone, still housing many of the main paper suppliers for the city, where we would have printed out our mapping party map had we finished everything in time. Cafe Universal is an old style Parsi cafe, and things just got fun and cloudy there. We finished the night at a still unmapped small fish restaurant across the street which fed four of us deliciously for 600 rupees!

Spent the night above the Worli seaside, gazing on may soon be the expanded sea link encircling the island city, blocking incredible exclusive views, but also perhaps further degrading the seaside environment, and not investing in pedestrian and mass transit Mumbai. Arun Ganesh and spent the morning walking to Haji Ali Dargah, chatting on how both the rich and poor of Mumbai maintain their spaces, how mapping might take hold in Dharavi, India's unmapped conservation areas, the transformative power of maps right in public spaces. Haji Ali was the site of intense holy frivolity, these young boys floating mini-mosques to crash on the rocks, old men describing with certain "rocket ships" washed up on the rocks, Bollywood-built young dudes posing in the crashing waves. A stimulating environment, that drove us back to Sanjay's to hack.

In a quick sprint, we got thing running on Topomancy's server, so OSM India now has it's own India Tile Server. The plan is to have stylesheets managed on github, with a script to manage pulling updates, configuration and restarts on the server. This all didn't quite work until this morning, after Jeff Meyer install mapnik2 and update everything. And that also means that OpenHistoricalMap is nearly ready to go, since it's installed and running on the same server.

And then the trip was over. Can't wait for more.

Location: Kala Ghoda, Churchgate, Mumbai, Greater Bombay, Maharashtra, 400001, India

(Not) Finding Communities

Posted by mikelmaron on 15 February 2013 in English (English)

I often get asked for connections to local OSM mappers and communities, or a listing of countries in a region with strong OSM presence. Depending on the place, sometimes it's easy, sometimes it's hard.

Today, I was asked about Jordan. It's proven hard.

That's a place where I have a little direct experience (http://brainoff.com/weblog/2010/02/26/1532) and some history of direct connections. I know there hasn't been a strong local effort, but there have been a few individual locals, and some interesting events. So I went about trying to finding people.

First stop OSM History Tab. This quickly proved useless as mostly it captured recent, big, global efforts.

I then tried the beta OWL History Browser. This work has been lately pushed by Pawel, with some UI ideas from MapBox, and building on the infrastructure developed by Matt Amos. It's looking awesome, and I hope that the hard last steps to make it production ready get a push soon. As of now, it's offline, so no use for my current needs.

Then over to ITO Map. I created an area for Amman, and it created a nice visualization of the top mappers. Clicking through to their OSM profiles, I can see their details, and the history of the mapping. Turns out the top mappers are all foreign, tracing from imagery globally; which is awesome, but not helpful for local community connection. A dozen folks down, and someone who looks to be local, though inactive. One gap here is that ITO is created statistics only for the last revision of every object in the area, not for all time; so it doesn't pick up the historic top mappers. I think that the license redaction hit this area hard, so there has been a lot of remapping. (correction: lyx, who's been heavily involved remote mapping, says that Jordan wasn't hit hard by redaction bot, just never much data to begin with)

Pascal's work on Results Map seems promising. But here again the issue is that most everyone positioned here is a global mapper to some extent. Would be nice to be able to filter to those people who seem to focus on an area.

In OSM itself, you can specify your home location, and on your profile it will show mappers near you. I'm based in DC, and can't easily browse other areas. So I repositioned myself in Amman (had to search for Amman on osm.org main page, and cut/paste permalink into boxes; map widget itself can't zoom out or search). This brings up a bunch of folks who only mapped 0-2 times, signed up in 2009/2010 (must have been a workshop).

I then tried WikiProject Jordan. Sometimes you'll see links here to country websites, discussion lists, forums. Nothing like that for Jordan. There's one active mapper listed there, again a global mapper. But I'm trying to connect with them via the OSM messaging feature.

Yet, I'm absolutely sure I have connected with Jordanian mappers. I checked my Friends list (a totally underused feature) on OSM, and I don't see anyone I recognize from Jordan. There's no way to tag or group friends here anyway.

Then, checked my OSM messaging inbox/outbox. At last. One set of emails from a global mapper working on redaction, and one from a mapper actually in Amman! Checking his profile, well he hasn't been active since 2010, hardly mapped at all, and didn't accept the CTs. Dropping a message just in case.

Finally, searching through my mail archives for "Jordan OpenStreetMap". A bunch of email from when JumpStart tried to get going there. Otherwise, a couple rounds of emails about events in Amman from a few years ago, where there was some small OSM presence: Tactical Tech held "Visualising Women's Rights in the Arab World", and Unicef/MobileActive held a Mobile Innovation workshop where Jeffrey Warren did some kite mapping. These were more regionally focused events, so didn't leave much of a local OSM footprint.

And that's where it is. Maybe there's no one active in Jordan locally. Or maybe it's just difficult to reach a nascent community, either already doing OSM, or interested but not in OSM yet; community building is tough work, takes a lot of direct relationships and understanding. No matter, I think OSM tools can be a lot better. Connection around a place is diffused across tons of tools, none of which work particularly well for the general purpose. I have a lot of ideas on how to improve the notion of places and groups, overdue to write that up. At least today's search serves as a guide for how tricky it is right now.

National Park Service Mapping

Posted by mikelmaron on 4 February 2013 in English (English)

At the mapping party in Hyattsville a few months ago, I stumbled apon and mapped part of the Star Spangled Banner Trail, a National Park Service Historic Trail covering sites throughout the region. Gave me the idea to organize a mapping party around the trail, and maybe coordinate with the NPS. I'm a history geek, and love getting outside, so OSM is basically an excuse to explore.

So I cast out onto the mappingdc list and twitter to get in touch with some folks at the NPS who might be into the idea. Most everyone lead me to Nate Irwin and Mamata Akella. I had met Mamata super briefly at SOTM-US, simply hailing the greatness of the NPS being there; I didn't get a chance to learn about their work then. We had a chat the a couple weeks ago, and I'm really impressed.

The NPMap team are working to create a tile set specifically designed for use by the National Park Service. Out of the box tile sets have their place, but there's a need for maps that highlight detailed park information, in the beautiful cartographic style we know and love in NPS paper maps. Their software and data stack is familiar to us, built around TileMill. The results so far are stunning, familiar from trips to the parks, and they're still working hard to improve the style and rendering process. And they're documenting the process, introducing the rational for the tiles, and delving into the technicals.

Just the other day, they used all this to roll out a real time Blue Ride Highway road closure map!

If you look closely, you'll see that OpenStreetMap is a key part of these tiles. Right now, they bring in OSM mostly for detailed and correct roads. But more is possible. The level of data comprehensiveness and freshness varies a lot across the parks, depending on whether they have an active GIS resource; some places are up to date, others not. OpenStreetMap is similar. Some parks have amazing detail collected, beyond what's traditionally available in NPS Maps, while others are barely mapped at all in OSM. There's potential for a virtuous cycle of sharing between the NPS and OSM. To that, there's been work led by User:Glassman to identify candidate NPS data for import, and devise a tagging scheme for the National Parks.

This allows public domain NPS data to flow into OSM, but what about the other way. Of course, there are license issues. But otherwise, there's the interesting technical question of how to manage, conflate, and synchronize multiple master databases. The NPMaps team are looking at setting up their own infrastructure (ie local osm_website) to manage and collect change. GeoGit is an emerging approach to integrating OSM into GIS workflows. Or perhaps OSM can be something like a monitoring tool for NPS, changes providing details to their own surveyors and cartographers. What's helpful about the NPS case is the very well defined data boundaries, which will make it easier to try different legal and technical approaches to data sharing.

I suggested that MappingDC would be able to experiment and collaborate, by organizing mapping parties focused on NPS data collection and management, with local park managers and mapping folks. The Star Spangled Banner Trail might be too diffuse. Rock Creek Park, right in DC, is convenient and enjoyable and interesting. So I'm ready to discuss more with the park management at Rock Creek, and see if we can organize an event for sometime when things warm up a bit!

Hyattsville Mapping Party

Posted by mikelmaron on 2 December 2012 in English (English)

Yesterday MappingDC did a mapping party in Hyattsville, MD. Good results. Was fun catching up with DC mappers, and meeting a bunch of folks from US Census (good job Steven Johnson!). Heard fascinating project fior informal settlement mapping undertaken by US Census in colonias, along the Mexico border, in Texas where there are no zoning laws, coordinating with researchers and activists ... never imagined anything like this in the US.

I biked from there from the OpenGovHub. The Sanitation Hackathon was there this weekend, exceeded my expectations largely due to learning about the Peace Corps Innovation Program and seeing some energy in the hub. This problem on mapping medical facilities in OpenStreetMap provoked a lot of ideas I've had on tagging in Kerala responsible tourism sites, and this problem on organizing directories of local projects linked to lots of thoughts from the Kibera Organizational Directory. I pitched the problem to finish off integration of OSM into the open source Google Crisis Map (yes cooperation between OSM and Google). There was also someone looking to do some drone work in less visited parts of DC.

Here's my mapping ride on Strava and the changeset. Would be cool to see more stats like what you have Strava in OSM.org. With a little analysis, you could see how much time spent recording waypoints, show other active and lead mappers in the area, etc.

The ride was interesting. Big roads seemed only viable, since bike path connectivity along the Anacostia is patchy from DC. I spun through the National Arboretum, and mapped a couple trees in the National Grove of State Trees. This stumpy Giant Sequoia was not exactly impressive, but I was inspired to think about mapping parties here come the spring, there is so much detail on particular special trees and collections of trees to map. Further on, past several cemeteries and one funeral procession (which made biking just a little more pleasant) I mapped a couple historic markers, one from the newly created Star Spangled Banner National Historic Trail. This trail covers a large area from the War of 1812, and mapping would be a pretty great means to experiencing it.

I don't think I would have ever made it to Hyattsville if not for OpenStreetMap. As ever, it's a perfect way to pay more attention and see the hidden world just around the corner.

Location: Highland, Brentwood, Prince George's County, Maryland, United States of America

Dolly Sods

Posted by mikelmaron on 20 August 2012 in English (English)

In early August, spent three days backpacking in the Dolly Sods, a beautiful and unusual wilderness area in West Virginia. Highly recommended, especially if you want fresh blueberries in your oatmeal.

For maps, I downloaded OSM data to my Garmin. It was only partially complete. Searching, I found maps at http://www.jtphillips.com/DollySodsMaps/, good comprehensive mapping of Dolly Sods. Way back in 2004, they created maps using only GPS and open source software (sounds familiar :). I downloaded and image and printed it out for use on the trail.

After I got back I wanted to update OSM with a few of my tracks. I contacted a few of the previous mappers in Dolly Sods, as well as reaching out to the Dolly Sods Mapping site. Fantastically, John supported us using their data in OSM. To start, I converted the DollySods KML map to raw OSM data, using gpsbabel, and brought it into JOSM as a background layer, to compare to existing OSM data.

Fantastically, John was able to upload the GPS data to OSM, and then (joined in mapping!)[http://www.openstreetmap.org/user/johntrudy/edits]

At this point

  • all of the trails in Dolly Sods have been better aligned to the GPS traces and to the Bing satellite
  • trails were connected properly into a network
  • tagging is consistent
  • some data that was imported with TIGER was fixed up (removed tiger:reviewed)
  • tagged the boundary of DollySods with additional tags ala http://wiki.openstreetmap.org/wiki/Tag:boundary=protected_area
  • added Red Creek
  • adding parking areas

There's always more mapping to do, maybe later...

  • a more comprehensive survey of campgrounds
  • add more water features
  • add trail numbers

This was great fun. Looking at the imagery after our trip was a great way to see it again. Hope I get to do more backpacking and mapping soon.

-Mikel

Location: Red Creek Trail, Tucker County, West Virginia, United States of America
Older Entries | Newer Entries