OpenStreetMap

manings's Diary

Recent diary entries

Welcoming new mappers around your mapping patch

Posted by manings on 10 April 2017 in English (English).

OSMCha has this great feature of watching edits around your neighborhood using the bounding box (bbox) filter. Today, I reviewed several edits using the combination of bbox and the new mapper flag.

screen shot 2017-04-10 at 16 07 54 </br>New mappers in the Philippines in the last couple of days

Note: Being a new mapper does not mean making bad edits per se, but by being new, we can commit errors because of unfamiliarity with the best practices of editing. Admit it old mappers, we made mistakes before. ;)

As I review each changeset, I often fix common errors based on my familiarity with the area. After fixing, I usually add a comment to the user’s changeset explaining my edits and encouraging them to continue contributing to the project.

screen shot 2017-04-10 at 16 16 46 </br>My changeset comments

While re-reading my comments, I realized I’m doing something wrong! My comments were peppered with OSM technical jargon new mappers may not even know of. I’m imagining this new user’s reaction 🤔 to my comment.

  • What is a changeset?
  • What do you mean by tags? Amenity, shop, building; what’s the difference?
  • How can I go back to my edits and correct them?

Next time, I’ll try to simplify my comments and avoid hard to understand terminologies. It is challenging to simplify without losing the substance of the comment, but, it is equally important to engage new users as first step in engaging in a conversation.

How do you welcome new users in your neighborhood?

LA Building import hit >2M buildings and next steps

Posted by manings on 12 December 2016 in English (English).

Back in July, we talked about the Los Angeles building import hitting > 1 million mark for LA City and last month, we already imported 2 million buildings covering the rest of the county. We are not finished yet ;)

Read on for what we are doing in the last couple of months.

la_3d
Downtown LA City in 3d

MaptimeLA in Catalina Island

The local community through @MaptimeLA has also been busy validating the data on the ground. A few of them took a ferry to Catalina Island for an awesome mapping weekend! Aside from validating the imported buildings, they added local features and Mapillary street/trail level photos across the island.


MaptimeLA explorers in Catalina Island, photo by J. Regan Hutson

Updated imported data for 2008-2014

Also last month, LA County GIS released an update to the building data covering changes from 2008-2014. The team decided to pause importing buildings to check the new dataset. The data is now ready and mappers can continue the import starting with Malibu & Santa Monica Mountains.

2008-2016
New buildings (2008-2014) are now available for import

What about the previously imported buildings?

We are reviewing the best strategy to update areas with already imported buildings. Conflation is challenging and we need to make sure that data already improved by mappers won’t be damaged unnecessarily. If you have thoughts on this, hit us on gitter or the project’s repo.

Location: Middle Ranch, California, United States

Fixing split buildings in LA

Posted by manings on 9 August 2016 in English (English). Last updated on 10 August 2016.

Last time, we talked about how we imported over 1 million buildings in LA. Watch this video from our SOTM-US talk. In this post, we’ll talk about our ongoing cleanup.

No data is perfect, the quality of what we imported in OpenStreetMap was generally good, but in all things data, there will always be unexpected cases.

During the import trials, we discovered that the LA City data was split to the parcel boundaries resulting to small polygons that should be part of the larger building (see: #71). We fix this during the import by using the Auto-tools plugin in JOSM but there were cases when it wasn’t fixed.

screen shot 2016-07-21 at 11 47 25

Detecting split buildings

We ran a detection for split buildings by analyzing size and shapes of buildings using OSM-QA-tiles, turf and tilereduce. A sample output looks like this:

screen shot 2016-08-04 at 15 07 21

The general idea is that:

  • small buildings are more compact (low area, high shape value);
  • large buildings are complex and wiggly (high area, low shape value).

The reverse is true for invalid/split buildings.

After several trials, we came up with an acceptable threshold for split buildings in LA.

Here are some examples of valid detection:

valid_small_splits

valid_large_split
Thin/narrow rectangles

valid_traingles
Triangles

valid_diagonal
Diagonal cuts

Fixing with Maproulette

The script detected ~4K buildings and is available as a task in Maproulette: http://maproulette.org/map/419/460642

Workflow for JOSM

  1. Go to http://maproulette.org/map/419/460642 and login with your OpenStreetMap account.
  2. Open JOSM and activate Remote Control tool.
  3. Download the Auto-tools plugin.
  4. Start fixing by merging the split building to the adjacent larger building using Auto-tools.

fix_split_josm

For ideas how to do this in iD, let me know.

We are continuously improving how we detect split buildings, if you have ideas, comment here or directly in the ticket.

Thank you for fixing!

Location: Nevin, Central-Alameda, Los Angeles, California, 90011-2519, United States

1 million buildings in Los Angeles

Posted by manings on 19 July 2016 in English (English).

Part of our series of diaries sharing experiences on the ongoing LA Building Import into OpenStreetMap. Last month, we talked about the tools. Today, it’s all about the data.

Just last week, we’ve hit 1.1 million buildings imported in LA City!
A great milestone as we are in the final stretch of import, validation and clean-up for LA City. Here’s a few map-shots on what happened in the last couple of months.

What happened?

We made sure that the import process followed the community accepted guidelines. Instead of importing everything with scripts/bots, we used the Tasking Manager to allow volunteers to take part. We divided LA City into four TM projects and organized mapathons within LA City to kick start the process. The animation below shows the weekly progress starting from Southside all the way to SF Valley. Large chunks of buildings were added during and after every mapathon.

time
Weekly progress, March - July 2016.

We asked volunteers to use a dedicated account and over 100 usernames participated. You can see this checkerboard pattern when you color the buildings by username/id.

user
Colored by user ID, > 100 usernames.

Before the import, a lot of buildings already exist. In recognition and respect to the mappers before us, we made sure that existing buildings were properly merged/conflated (in JOSM, we use the Replace Geometry tool). This is a tedious process but this is how it should be done. By doing this, we are keeping the editing history of existing buildings.

version
Feature version. Blue = v1 to Yellow > v4.

What was imported?

Aside from the good quality of building footprint, we included several tags that describes every building. This is not visible in our default map rendering, but these tags exist in most of the data we imported.

Building types

We’ve included building types based on LA Country’s Assessors information.

types
Colored by building type.

Year the building was originally constructed

There’s a lot of history in the urban expansion of Los Angeles starting from the pueblo in 1700s up to today. The import data has a year_built attribute which we included in the import. You can now see the city’s settlement history in OpenStreetMap.

year_built
Colored by year built, 1800 - 2015.

Building heights!


Rendered in Mapbox gl-js

These are a few of the tags we’ve added. I’m excited to see what the community creates out of this data. Our friends at MaptimeLA started experimenting with the data already!

cm0fn3_vyaaws-x jpg_large

Photo by MaptimeLA.

Are you planning on using this data in your own map? Catch the team this weekend at the SOTM-US and show us what you created!

More info about the import is available in the following links:

Location: Exposition Park, Los Angeles, California, United States

[WIP] A new neighborhood, a new mapping quest!

Posted by manings on 3 July 2016 in English (English).

living_street
A typical living_street in Bangalore.

It’s been a couple of months since I moved to Bangalore. Naturally, with a new neighborhood comes an opportunity to map and improve the data! When I first came to this side of the city, OSM coverage was fairly good. Streets were marked and buildings were traced, but no other detail was there. It lacked street names, POIs and neighborhood names. So during my daily commute, I try to add a few details to make it much more updated and useful.

Mapping a totally different place comes with challenges that break many of your assumptions. Here’s a few things I found amusing, interesting and at times confusing as I navigated around this Indian city.

Colours everywhere

colors
Everywhere you go, you see colours. From the temples, to the rangolis, to how vegetables are stacked like a treemap infographic. It seems that colour defines the city’s daily lives.

shop=ironingshop?

press
The last time I saw a working charcoal iron was when my grandmother was using it. Here, there are roadside shops in carts that iron your clothes while you wait.

Different beliefs can always get along

worship
Roadside place_of_worship, left-side: Jesus and Mary, right-side: Lord Shiva.

Chai shops everywhere

tender
You don’t need to worry about refreshments, chai shops are everywhere. This one even sells fresh coconuts.

Signs maybe confusing

hotels
This got me confused for a while, it turned out, not all hotels in India are for sleeping. They are eating hotels (amenity=fast_food) NOT sleeping hotels (tourism=hotel)!

There are exceptions to the rule!

commissariat
Just as when I’m getting used to cycling on the left hand side, I stumbled into Commissariat. I initially thought the data was wrong but apparently, you drive on right at Commissariat.

Mapping is still a work in progress, I don’t have much of a plan. In my free time, I simply walk and mark anything I find. But the map is now beyond just streets and buildings. Check it out and let me know what’s missing.

emoji
Emojimap.net

Location: Maruthi Seva Nagar, East Zone, Bengaluru, Bangalore North, Bangalore Urban, Karnataka, 560084, India

Building tools for LABuildings Import

Posted by manings on 30 June 2016 in English (English).

This is a part of a series of diaries sharing our experience on the ongoing LA Building Import into OpenStreetMap.

In the last 2.5 months we started importing building footprints over Los Angeles from open data available in LA county. Discussions about this import started early last year, after several discussions, planning and trial runs, we finally started the import this April. From its start, the import team agreed that this will be a community managed import. The goal is not only to improve building coverage of OpenStreetMap within the county but also to invite local mappers to actively participate in the whole process.

In this post, I will talk about the tools we built to coordinate this massive import. Many of the processes were based on an earlier buildings and addresses import in New York City with modifications needed due to the difference of data and the context of the local community.

Data sources


3 million buildings in LA County.

The data came from several open data sources provided by the Government of Los Angeles:

  • Building geometry (LA City, LA County) - building footprints digitized using high resolution stereo imagery under the Los Angeles Region Imagery Acquisition Consortium (LARIAC) Program.
  • Building type and use - from the Assessor’s parcels database. Each building from has an associated parcel identifier (AIN), This identifier allowed us to link the type and use of each building.
  • Census blocks - this is not directly part of the data to be imported, we simply used the census blocks for dividing the data into manageable chunks.

We combined the building geometry and parcel information using the usual join by attribute GIS operation. This creates a single shapefile of buildings with all the parcel information for each building.

In total, we have >3 million buildings that needs to be checked for quality and a workflow to coordinate a massive community import process.

Data attributes to OSM tags

The Assessor’s parcel data contains detailed usage of each property. We used this data to identify the type of building. To do this, we compared each attribute to taginfo and adopted tags already used by the community. For all other attributes that didn’t have corresponding tags we used the generic building=yes. A CSV was created as a lookup to convert the shapefile attribute to OpenStreetMap key/value pairs.

From shapefile to .osm format

Like the NYC building import, we scripted the entire process pipeline so that we can execute in a single command. In general, the script performs the following steps:

  • Download and extract data in shapefile format.
  • Reproject from source spatial referencing system to long/lat WGS-84 (EPSG:4326).
  • Split buildings into smaller chunks based on census blocks.
  • Convert attributes to OpenStreetMap tags and export as osm format.
  • Upload to S3.

This automated process allowed us to easily re-run the conversion if we need to. For example, when we discovered data issues specific to buildings in Pasadena, we were able to exclude Pasadena in the next run of the script.

Coordinating the import via the Tasking Manager

To manage the coordination of import by the community, we used a separate instance of the OSM Tasking Manager. By using the Tasking Manager we can:

  • Split the import into smaller TM projects;
  • Track the progress of the import and;
  • Introduce a two-step process of import and validation.

But the current Tasking Manager does not allow downloading of import data from arbitrary polygons. To make this work, we implemented a new feature in the Tasking Manager that allows it to:

  • Upload a GeoJSON file with a property referencing the source URL of the .osm file,
  • When a contributor takes a task, the Extra Instructions section will have download URL for the import data. This URL will automatically download the data into JOSM.

These changes were merged into the main Tasking Manager code so anyone using the latest TM codebase can use this feature as well.

Once we had everything ready, we did a couple of trial runs to evaluate the mapping workflow.

During one of our trial runs, we discovered that since we are using arbitrary polygons instead of the usual square tasks, mappers find it difficult to visualize the edges of the task they are working on. This can introduce data conflicts especially when the imported data is now merged to the current data within OpenStreetMap. To solve this issue we added a background layer of census block boundaries that is automatically loaded when a user downloads the data.


Downloading data for import.

Automated merging of the split buildings

The trials also encountered cases where a small sliver of building was split from the main building.
Upon further investigation, these buildings are cut by the property lines of the assessor, during the LARIAC mapping.


Split buildings.

Since this came directly from the source data, we have no way of fixing this during the data conversion process. This required the user to manually merge the split buildings during the importing process.

While combining overlapping buildings in JOSM works, it requires several mouse clicks to merge two buildings. To make this process faster, we built a plugin in JOSM that merges two buildings and automatically assign the correct tags.


Merge LA buildings in 3 clicks with auto-tools plugin in JOSM.

As of this writing, 105 usernames added ~580K buildings. This is more than half for LA City.


500K buildings added. Map by F4map.

As we go along, we continue to improve the workflow and tooling, let us know how we can make this import better! Head over to labuildingsimport.com/ and grab any task available. If you’re a local, join our mapathons happening through MaptimeLA.


MaptimeLA building import mapathon. Photo by MaptimeLA.

In the next post, we will talk about how we interacted with the mapping community and the response by the local LA mappers on this import.

If you’re in Seattle next month, catch the team at the SOTM-US where we talk more about this import.

More info about the import is available in the following links:

Location: Baldwin Hills/Crenshaw, Los Angeles, California, United States

Your OpenStreetMap Story

Posted by manings on 19 March 2016 in English (English).

Over the years, people ask me how I started and what have I done so far in OpenStreetMap. I tried to capture this in an earlier post.

Wouldn’t it be nice to have a template for your OpenStreetMap elevator pitch? Inspired by an old discussion and The Open Source Report Card, I looked at Pascale’s HDYC numbers to tell a story about your OSM journey.

Stories unlike numbers, are kind of imprecise, fuzzy and sometimes exaggerated. ;-)
This is the challenge of this exercise. How can you tell your OpenStreetMap story this way?

So I made something out of OpenStreetMap data, and, it’s not a map.

prosm

Click the image to see video

Try it out! Use this as a boilerplate and fill in the details of your OpenStreetMap story.

The code is messy/buggy but somehow works. Don’t take the numbers seriously, this was just a fun hack. :)

OSM-PH Error Dashboard using OSMLint

Posted by manings on 16 January 2016 in English (English).

Using OSMLint and vector tiles, I made this simple dashboard to display and fix common errors.

http://osmph.github.io/errors/

errors

Dashboard is based on Fulcrum Map’s Geojson Dashboard. I hope the OSM-PH community can find it useful.

You too can build your own for your local mapping community.

Basic ingredients are:

  1. Get data as vector tiles: http://osmlab.github.io/osm-qa-tiles/
  2. Process data with OSMLint: https://github.com/osmlab/osmlint
  3. Configure your dashboard: https://github.com/osmph/errors/tree/gh-pages

Keep on fixing the map!

Missing Maps Workshop at Mapbox-BLR

Posted by manings on 9 November 2015 in English (English).

Our Mapbox-Bengaluru office celebrated International Open Access Week last month with a data gallery and a mapping workshop. I co-lead the mapping workshop as part of the Missing Maps project. I penned these notes to remind me in the future how to run our workshops even better. I figured, why not share to everyone?

2015-10-21 15 32 25

What we talked about

  • OSM is a volunteer/community driven project, mainly for fun and exciting ways.

screen shot 2015-10-26 at 3 24 16 pm

  • Coincidentally, we can use it for crisis mapping.

2015-10-21 14 54 03

  • But, we should map not only during/after a crisis but even before a crisis happens. Preparedness is key.

  • Here’s how OSM works (show basic tagging but do not go into complex technical details).

screen shot 2015-10-26 at 3 23 58 pm

During mapping

  • Let them experience mapping first hand. Answer the technical details as they go through the task.

2015-10-21 15 10 33

2015-10-21 15 41 24

After mapping (wrap-up)

  • At the end of the workshop, always show the results no matter how small (changesets, number of edits, before/after screenshots). For example, http://resultmaps.neis-one.org/osm-changesets?comment=project-1250#10/-4.1786/28.7883
  • Point them to links where they can read more about OSM or how to improve their skills. For example, http://github.com/mapbox/mapping/wiki.
  • Invite them to be part of the community (link to mailinglist, forum or social network groups the local community uses).
  • Always find time to socialize. Mapping workshops are not just about collecting data but also about expanding our community.

2015-10-21 17 26 17 2015-10-21 18 46 33

See also