Recent diary entries
Building detection at Villa Imelda, MacArthur.
For the last couple of weekends I’ve been tinkering with Robosat to detect features from aerial imagery. At its core RoboSat is using state of the art fully convolutional neural network architectures for semantic segmentation.
Daniel posted an excellent walk-through to run the RoboSat pipeline on openly available drone imagery in Tanzania.
This post follows Daniel’s guide for detecting buildings in drone imagery in the Philippines. The goal of this exercise is for me to understand the basics of the pipeline and find ways to use the tool in identifying remote settlements from high resolution imagery (i.e drones). I’m not aiming for pixel-perfect detection (i.e precise geometry of the building). My main question is whether it can help direct a human mapper focus on specific areas in the imagery to map in OpenStreetMap.
Daniel already outlined the step-by-step process. This post will focus primarily on the data preparation. To be able to build a robust model for feature extraction it is important to create a clean dataset as input to training your model.
Dan Joseph from the American Red Cross collected a lot of drone imagery in Visayas, PH summer of last year. Thanks Dan and ARC! 🙇. I chose these images to start with my experiment because of the amazing resolution (up to 2 — 3 cm) and Dan and his team used this imagery to trace buildings which I can use for training the model.
The imagery is available in OpenAerialMap as individual TMS or GeoTIFFs — no composite seems to be available. In order for me to collect enough samples I needed to get images and masks from several of these GeoTIFFs or their corresponding TMS endpoint.
I downloaded several of the GeoTIFFs and clipped the OSM buildings from the bounding boxes of each imagery. These buildings will be used as training labels. But this is not enough, “presence” (positive samples) data will only give so much reliability, we also need to provide our model with “absence” (negative samples) data. Here’s an example of the data I traced using QGIS in addition to the buildings from OSM to bootstrap my negative samples.
imagery bounds > imagery > building > other cover types.
Cleaning the “presence” data
As illustrated above, drone imagery usually don’t have a squarish boundary so using the bounding boxes to clip data from OSM is not optimal. You will get black tiles with buildings which can affect your model. In my initial data prep I got images like this in my training samples.
No imagery, but with buildings in OSM.
The are two options to clean the data, you can:
rs rasterizefor all the images then delete these tiles or,
- delete the buildings outside of the actual boundary of the imagery in QGS (you can do this too in JOSM, but make sure you don’t upload 😉 )
I chose the former approach because its faster. There may also be cases when the building polygon is not aligned to the imagery particularly when it was traced from another imagery source. Fortunately ,this is very minimal in my sampling areas and these smaller problems won’t be a problem during training as long as there is not a large constant bias in the dataset.
Adding “absence” data
For adding negative samples, I traced several polygons representing common landcover types such as roads, water/riverbeds, orchards, farmland (wet and dry) and bare areas. I added various landcover types so that the model can learn that these are not the features it should detect. In order to avoid overlapping with the building tiles I overlayed a z21 tile boundary in QGIS and only traced landcover type the does not overlap with a building poly.
Now that I have both positive and negative polygons, I need to run
rs download and
rs rasterize to create a slippymap directory with images and rasterized buildings. Since I have to do this in all individual imagery, I created a script that loop through each imagery list from OpenAerialMap. Run the script for your positive and and negative samples snd combine samples in a single directory using
# Combine negative and postive samples to one slippymap directory mkdir combine-images rsync -vrh positive-images/21 combine-images/ rsync -vrh negative-images/21 combine-images/ # do the same for the rasterize output
In total I have 15,237 samples at zoom level 21. 7104 (46%) for presence of buildings and, 8,133 (54%) absence (negatives).
Training the model
Once data is ready, I can start training the model. I need to split the imagery and mask tiles into training, validation and a final evaluation dataset using
rs subset. For my sample I split them into training 80%, validation 10%, and evaluation 10%.
rs weight to calculate the class distribution of the masks in your dataset. Save the result of
rs weights in your RoboSat dataset configuration file (
Finally, start training the model. I used a GPU capable (aws p2x.large) machine to run the training.
./rs train --model config/model-unet.toml --dataset config/dataset-building.toml
In my training, I chose 4 as my batch size and 50 epochs. The checkpoints are saved after each epoch for evaluation. Once the training is done, the results are visualized on the saved plot files like below.
The main indicator to choosing which checkpoint to use for prediction is the validation mean IoU. In my trained model the highest mean validation IoU (~0.82) was at epoch 40.
Visualizing the result
Before running the trained model to another imagery, I visually inspected the results using my evaluation dataset. This is a good dataset to use since it was never “seen” by the model during the training process.
# Run prediction to the evaluation data ./rs predict --tile_size 256 --model config/model-unet-maning.toml --dataset config/dataset-building.toml --checkpoint tmp/pth/checkpoint-00040-of-00050.pth dataset/evaluation/images/ evaluation-segmentation/ # Get masks from segmentation probabilities ./rs masks evaluation-masks/ evaluation-segmenttation # Compare images, segmentation and masks ./rs compare evaluation-compare/ dataset/evaluation/images/ evaluation-segmentation/ evaluation-masks/
Here are some results of the prediction (imagery-left, segmentation probability-middle, mask-right).
Not so good
In most cases, the prediction looks pretty good except for landcover types that are very similar to building rooftops like paved highways, vehicle or when the structure is covered by trees. But overall I’m satisfied with the initial result.
Putting it all together
Now that I have trained model, I used it to predict detection in another imagery. Instead of downloading the tiles from OAM, I used Daniel’s tiler script to create tiles from a GeoTIFF.
python3 tiler.py --zoom 20 660c5321-0334-471f-bca5-829d85fb1d40.tif 660c5321-images/
Delete completely blank tiles to prevent running the prediction on them (these are mostly found on the edges of the imagery boundary). This speeds up your prediction time.
find . -name "*.webp" -size -190c -delete #the filesize will depend on your imagery, make sure to delete only the blank tiles.
Run the prediction.
./rs predict --tile_size 256 --model config/model-unet-maning.toml --dataset config/dataset-building.toml --checkpoint tmp/pth/checkpoint-00040-of-00050.pth 660c5321-images 660c5321-segmentation
To visualize in a map, I created a leaflet side-by-side webmap to compare the predicted segmentation probabilities and the imagery. See the result here.
Of course, I had to load the data into an OSM editor! So I blended the imagery and segmentation together using imagemagick.
Give it a try! Go to OSM and copy the TMS endpoint below in ID’s custom background imagery.
- The process is straightforward, thanks to Daniel’s and Bhargav’s excellent README and diary posts. Part of this experiment is to confirm that anyone can do it without any knowledge of machine learning and I confirm it is possible. I have started reading some ML basics though to know more. This fastai lesson 0 video is a great introduction to the concept.
- We can leverage open data and open source to build ML based detection models. All data (buildings from OSM, imagery from OpenAerilaMap) and tools (Robosat, QGIS) all free and openly accessible. This experiment won’t be possible without access to these data and tools.
- It is critical that you have a clean data that you feed to the model prediction (GIGO). In my initial iterations the results were really bad because I did not include enough negative training samples. The outcome of your model highly depends on the training data you provide it.
- A lot of data preparation can be done on a modest laptop. I only used a GPU capable machine for model training. What will take weeks to do the training on a CPU will take only hours with a GPU. In my case I deployed a p2.xlarge instance from AWS. This cost money, an ondemand p2.xlarge in in aws costs USD 0.9/hour. My current bill is around USD 50 after 4 iterations of model training.
- I had fun! At first running a full machine learning stack seems daunting but RoboSat makes it so simple.
The past few weekends was really fun, I plan to continue iterating on the training models and share back results (more visualizations in this repo). I had discussions with several friends doing drone capture and they think automated detection like this can help in the initial assessment of an area say after a crisis. Next time, I want to explore:
- add more training samples into my current model using various drone imagery;
- creating models for other feature types;
- convert the result into vector (
rs features) and feed it directly to the editor or a tasking tool.
- publish my model for anyone to use, let me know in comments 👋 here if you want to test my models.
Give robosat a try! The devs are more than happy to help you along the way!
I created way which has _link tag for the first time.I did not have ever edited road.So, I may have made a mistake.
Change set is here
MAPS.ME is a great app, and Ashgabat users suffering from high internet costs (the highest in the world, we are told) have found it a cost-effective way of uploading POIs. In certain important respects, this is a boon to mapping Ashgabat and other cities in Turkmenistan.
There are, however, two issues. First, the mappers often mis-tag POIs, either because they don't know English too well (some mappers tag all medical or health facilities "amenity=hospital", for example, including pharmacies, clinics, and ambulance stations), or because MAPS.ME's limited universe of point tags doesn't lend itself to proper tagging. Second, because MAPS.ME's offline maps are updated relatively infrequently, multiple mappers add a POI multiple times, so you end up with two or three new entries for the same POI. This has created a fair amount of work for me, cleaning up after entry of POIs that either duplicate an existing node, usually consolidating all data from multiple entries, or simply correcting tags. I'm not complaining--there are worse problems (like vandalism), but it does take time.
The good news is that this weekend temperatures in Ashgabat will be well over 100 Fahrenheit, mid-40s Celsius, so I have a great excuse to stay close to the air conditioner and edit the map.
In Argentina we discussed and finally have a new highway hierarchy criterion.
Proposal is kind of loose so we can adapt it in different situations. I used it on my city and I think is an opportunity for other to understand the process. Specially if you live in a city similar to mine.
Context: Resistencia has 300,000 inhabitants, at East there is an important satellite city Barranqueras that has 50,000 inhabitants, at Southeast Puerto Vilelas has only 8000 inhabitants, at West you find Fontana, 20 years ago a town with 10,000 people, now 40,000 people. Resistencia is a wide city with a weak infrastructure, less than 40% of its streets are paved, and there are big areas where main roads are unpaved avenues. Its checkerboard city plan has "avenidas" (avenues) each 1 km, inside that avenues there are very few avenues. It is situated on a flooding plain, there is not any rocky outcrop, average precipitation is 1200 mm a year and it never snowed. Unpaved streets are maintained with gravel, mainly when they are in a public transport route. There are so many unpaved streets that it is very difficult to have them in good conditions all the time, more difficult when it's rainy season, so public transport streets are heavily used because it's very common that are the only one in good conditions.
Trunk: in a medium size city we shouldn't have any trunk street. Exceptions are roads that get here: RN11 and RN16. In big cities it could be possible we find trunk internal roads.
Primary: under our scheme they are "trunk avenues". In Greater Resistencia they are the main access to the cities, except for Puerto Vilelas.
- Resistencia has 7 primary links. 2 direct links to RN16 at North; two links to Barranqueras at East; two that go West, one of them remains primary up to RN11, the other one links to Fontana. Also I set 1 km sections in two avenues, which have 3 lanes for each direction.
- Fontana has only one primary link, that takes you to Resistencia and RN11.
- Barranqueras has 3: two linking Resistencia and another one that links to RN16, where they get together there is a fourt highway that arrives to city's downtown.
- Puerto Vilelas: it hasn't any internal primary road, only avenida Castelli that is mainly an access to Barranqueras. Nonetheless, I doubted about setting main avenue as primary, this his logical since our limit for a city to have a primary or secondary access is in 10,000 inhabitants. It is very likely in some years I have to set it a primary.
- All of this highways have in common that are heavy transit highways, they have higher priority in junctions. They are the only roads where traffic light are sync (so it is clear that townhalls are more interested in transit flow on this avenues). They have many shops, this is clear in 25 de Mayo avenue, which falls from primary to secondary without you can see any physical change. Avenida Sabín -which links Resistencia to RN16- has a short section in which they split up into 2 one-way streets, this streets have the same heavy and fast traffic, so they are primary too. But when they get at downtown there isn't any avenue that is undoubtly its follow-up, I mean, when you arrive downtown you can go to main square by Lavalle or Ávalos, so they are both secondary..
Secondary: in proposal they are "paved avenues". In my city they are paved avenues, 1 km length or more, and also unpaved avenues that are as wide as a paved avenue. I will explain this, in city plan checkerboard there is an avenue each kilometer, in some cases this avenues are not wide, so you can distinguish it from a regular street. It could also happen that they are wide, but townhall only maintains a regular street width, so you have a big non-used space between building lines, it is common when neighbourhood linked are not so much populous. In big areas without pavement you find wide unpaved avenues that have a heavy transit like a paved avenue, so in Resistencia unpaved avenues set as "secondary" are not an exception as it should be in more developed cities. I found just three secondary avenues without a physical barrier between directions: Combate de Obligado, 9 de Julio in Barranqueras and Soberanía Nacional/Malvinas Argentinas; in the first one you can't park, in the other two there is enough land at botn sides to have a physucal barrier, they look like non-urban roads that were surrounded by the urbanization.
Tertiary: in our classification they are short paved avenues, unpaved avenues and major streets. In this hierarchy I put avenues wide as a street and without physical barriers for each direction; very short avenues (less than 1 km), and streets that are internal corridors between avenues, no matter if they are paved or nor. When they are part of public transport routes Town hall put its effort on them, they are better enlightened so they are preferred streets for cars and motorbikes. As a special case side streets along RN16 and Soberanía/Malvinas Argentinas are tertiary.
You can now preview OSM notes in iD by pressing 'F' to open the map data pane, enabling notes, and zooming in. Much like on osm.org, you can toggle on and off the notes layer, close / reopen notes, comment on them, and follow links to a note or its creator on osm.org. Give it a try and let me know what you think! We are compiling feature requests such as filtering, note templating, etc. Also, feel free to search and open issues related to bugs or desirable features. Before the next release of iD, expect an add
Note button as well (as seen in the image below).
Additionally, along with Bryan Housel and Marc Farra (my GSoC 2018 mentors), I am researching how OSM users use notes. We hope to make notes more useful, more consistent, and potentially a rejuvenated outlet for stronger socialization and communication across the OSM community. Therefore, if you have feedback on how to improve notes (or what you use notes for), please comment below! In my next blog post, I'll provide more fuel for ideas by sharing note features that we have already started working on.
Until next time,
OSM processes over 3.1 million feature changes per day. With the help of OSMCha and the ever-vigilant OSM community, many low-quality changes are found and corrected. However, sometimes low-quality labels are not detected and slip into the map.
Towards ensuring a high-quality map, I have been applying natural language processing (NLP) techniques to detect low-quality labels. Part of applying these NLP techniques requires:
- Defining low-quality labels and understanding their properties
- Determining the motivation of a user who submits a low-quality label
First, towards defining a low-quality label, we typically think of labels that include profanity, such as the following (also! 🙂, all of the screenshots you see below are examples of low-quality labels that I have found with my filtering logic):
Profane label says: “The Shit”
Profane label says: “Too Damn Far Rd”
However, labels can also be low-quality but not profane, such as the following:
Assertive label says: “Arguably the best location to see evening Manhattan Henge”
Humorous/assertive label says: “I’ve replaced the batteries but my smoke detector won’t stop beeping!”
Flag planting label says: “Jen is here”
Through inspecting millions of labels, I found that the three main motivators of intentional vandalism are:
- Flag planting, or the idea that someone wants to assert that they were present in a place
- Assertion of quality, or a user who wants to express their opinion of quality about someone or something
- Humor, such as meme-based labels
On the other hand, the only motivation of unintentional vandalism seems to be naiveté.
Specifically, I inspected over 11 million map labels from the United States of America. With filtering logic applied, I was able to narrow down the potential list of poor-quality labels to just 256 labels. Of these 256 labels, approximately 30% were true positives. As a result of these findings, I submitted my first OSM edits! https://www.openstreetmap.org/changeset/59951250
However, these are only preliminary results and I am excited to continue expanding my filtering logic! 🙂
The workshop: Orientation and Consultation on Geographic Information System (GIS) and its Applicability to Social Welfare and Development of the DSWD GIS Unit was conducted on July 18-20, 2018 in Quezon City. It was attended by the Planning Officers and Technical Staff from the Central Office, Field Offices and Attached Agencies of the Department.
The program consisted of several talks from various resource speakers of different sectors (academe, social/community development, environment, Indigenous Peoples, disaster response, and private sector).
Being one of the main sources of free spatial data, OpenStreetMap was briefly presented by OSM mapper advocate, Ms. Jen Ayco (@jenjereren). She encouraged the participants to sign up for an account to increase visibility of each of their communities, and raise diversity (gender, ethnicity, PWD's) on the map. Surprisingly, few participants were already mapping volunteers of OSM! :)
During the afternoon break, I found myself surrounded with the three women mappers/resource speakers we invited. They've shared experiences in the field of GIS and discussed the importance of an integrated and free geographic data source in the country. There is an existing one though, PhilGIS.org, but still needs to be populated and updated.
Ms. Kail's remark on this topic was something to ponder about: "No one owns spatial data. Nobody can have copyrights on your location."
Maybe the act of gathering location information (purchasing of tools and investing effort) gives data collectors the right to own it. Giving proper credits to data collectors would facilitate sharing of data. And there are still privacy issues. It's still a case to case basis. And maybe, spatial data can be also be tagged as "special" data depending on its purpose.
On the last day of activity, we asked participants to identify data which can be translated spatially. Data were classified as data for internal use or data for sharing. It’s great to know that they’ve appreciated how GIS can be applied to their work. Not only for mapping/visualzation purposes, but also for planning purposes such as attending to the needs of vulnerable sectors in times of disaster by building inclusive evaculation centers and other gaps on social facilities.
Indeed, transforming their special data to spatial data to communicate and serve the peolple better.
In 2012, the 'new' public_transport scheme was voted. What seemed like a good idea at first, has not led to unification though. I was not smart enough at the time to see how this would develop, but today with the benefit of 20/20 hindsight it it's apparent that we are on the wrong track.
The way the wiki formulates it at present, even though I have tried over the years to keep it in check, we should supposedly map 2 objects for each and every stop and then add both of these to the route relations.
I'm sorry, but that is unacceptable! For one thing, I need a SINGLE object, preferably a node where all the details of the stop go. That has always been the way we do things in OpenStreetMap. The node that gets the stop_position tag is NOT suitable for that purpose, as it is not immediately apparent on what side of the street the stop is.
So, over the years I have tried to work with public_transport=platform and stop_position tags on the stop nodes I've been mapping. The idea was that, in time, highway=bus_stop would be replaced by those tags. Only that is not possible, for one thing, those nodes we're mapping to represent the stops, were not supposed to have a mode of transport on them. So it is utterly impossible to drop highway=bus_stop from those nodes.
OK, let's step back for a moment: highway=bus_stop is here to stay, wait for it... it's coming... the conclusion: public_transport=platform tags are NOT ADDING any information. highway=bus_stop says it all. Sorry that it took me a few years to figure that out!
Of course, now the epiphany dawned on me, I decided to revisit all 70000 bus stops around Belgium. So since yesterday I started to remove public_transport=platform/bus=yes tags from all those stops, in packages of about 100-150. Obviously that's not all I'm doing, some stops have moved, we have better imagery now than we did 5 years ago, there is a new bus_bay tag since last summer. It will take 6 to 12 months to complete this mission, but soon those tags that don't add actual information will disappear from sight.
It's time we came up with a scheme to map public transport that is: * easy to work with * easy to maintain * easy to visualise and that: * does not duplicate information * does not force mappers to add 2 objects to route relations
I have a clear vision of how we can accomplish that.
இந்தியா என்பது ஹிந்தியா அல்ல. பாரத் என்று இந்தியாவை குறிப்பது தவறு. இந்தி தேசிய மொழி அலல. India என்றிருப்பதே சரி.
India is not Hindia. Please remove the word "Bharat" (1000km) in hindi from osm. Indians speak 100's of languages. Hindi is not a national language.
There is a lot of prepared and approved SVG icons for OSM-Carto just wainting to be added, but there is unfortunately also lack of people ready to add these new features. We would like to invite you to improving default OSM style.
Here is a task list: https://github.com/gravitystorm/openstreetmap-carto/issues/3298
Coding instruction links:
You can also ask for any help in certain tickets and PRs :)
I have been OSM mapper since February 2015 and got enthusiastic about the change I was making in #MapLesotho. Being the spatial planner I realised the importance of OpenstreetMap data and how it would be useful in my job.
Being number one mapper in Lesotho opened doors for me, and in September 2016 I won a scholarship to study in Ireland doing MSc in Spatial Planning at Dublin Institute of Technology. I adapted easily in Ireland & met Ireland OSM community and also continued training school kids to map, and also how to use different Phone Apps such as mapillary, OSM Tracker, street complete and maps.me.
I wanted to be able to map more than what #MapLesotho taught me, so simaltaneous to moving to Ireland for town planning scholarship and internship I started mapping Ireland. I adapted to different ways of mapping and started mapping things in Ireland that aren't there in Lesotho such as the coast and heritage sites, different tags for different road uses, such as bicycle lanes, bus lanes, footpath etc. It was also fascinating to map high buildings because they are frequent in Ireland using maps.me and the sudden change when editing height of the buildings using the app was entertaining as well. Streetcomplete is another app that always got me curious because the more I used it the more I got different kinds of stuff to edit which I would say widened my experience with Tags also.
Being in a new country of course I wanted to get out and about, so I turned my curiosity into mapillary trips in all 32 counties and I became number 2 mapillary contributor in Ireland. The truth about me is when I do mapping stuff I make sure I do it well and conquer, hahah😊.
In June I became mapper of the month and that took me by surprise when I was contacted by a person I have never talked to before Marc Gemis because I never thought there are people out there who notice the effort and the changes I was making.
I know a lot of countries and places I have never been through mapping and I never want to stop. Anyday without mapping is a bad day for me. The mappers in the places I have lived know I show up and do my part and help others. I don't disguise that I am also happy to map on my own and I like to help new mappers all the time. Vote for me if you are like me, or want there to be more of me. If I win this I will show it to others to inspire them to start mapping.
Thanks for the ongoing discussions about gender in OSM. At SOTM, we will build on our shared efforts to make improvements. Your support and participation is most welcome. At SOTM we will host the Open Gender Monologues. Our allies, GeoChicas, are also running a session on diversity.
Three ways you can become involved in the Open Gender Monologues:
Submit your experiences (optional to be anonymous) for us to read at SOTM on July 28th. We want to be sure that your voice is heard, even if you can't join us in person. add your experiences here
Join us in person on July 28th to share your experiences. Just let Heather and Kate know so that we can add you to the agenda of speakers. SOTM - Open Gender Monologues session
Be an ally - share this diary entry on your local/topical mailing lists and social media. We want to reach as many people as possible.
Thank you and see you soon
I took a tip from Dexter from the City of Detroit Office of Innovation, and started on the Detroit Mapping Challenge by browsing City of Detroit Open Data. What open data could help make OpenStreetMap Detroit the best map in the world? BikeShare locations looked like a useful and straightforward starting point. And now OSM Detroit has very accurate MoGo bike share docking stations. There turned out to be a few surprises getting there, and lessons to absorb for mapping all of Detroit.
The data looked decent on quick inspection, and is licensed public domain. Maybe a very small human supervised import is in order. I browsed OSM to see what was already there, and turned out this already all 43 docking stations were already added by mapper175, with the changeset comment
Added nodes for MoGo Bike Share system stations (resurvey needed for most of them). Are they in the right place? Is there something a remote mapper could do here?
I picked the Second Ave & Prentis St docking station at random, and opened it up in iD.
I cycled through all the imagery options available to OpenStreetMap, and no evidence of a bike share dock station. Turns out MoGo launched just over a year ago and all of the aerial imagery is apparently older than that.
I then tried street level imagery. Bing Streetside is comprehensive but collected back in 2014. Fortunately in Detroit, Mapillary and OpenStreetCam have extensive and recent coverage. Clicking through the street level images in iD to find something with the correct alignment to capture the dock was sometimes tricky — depended on the precision of the capture position, the direction, the field of view, and distance. Confirming features seen in the street level imagery against aerial imagery was helpful to choose a well located shot.
For Second Ave and Prentis Street, the docking station moved just a few meters, from on street to inside a parking lot. While the accuracy of the first location in OSM was probably ok enough to find the docking station, the position of on the street vs sidewalk has substantial meaning for high definition mapping and analysis.
I next checked out the dock at Cass Ave & W Hancock St. The original position on West Hancock looked ok, but the OpenStreetCam imagery quickly confused me. One image showed the doc on West Hancock, and another from a few months later on Cass Avenue. I couldn’t trust what my eyes told me here — is the location or direction of the images incorrect? Am I looking at the same dock?
I searched and found that this dock had moved due to street construction. The reported move date didn’t quite match up with the first image but seemed reasonable to explain what I was seeing. And it also mentions the move of ****2nd Ave. and Prentis — somehow luckily the first two stations I picked had substantial changes.
The MoGo site also had a map, and browsing it, the locations for the above two dock stations were very accurate. It looked like the location in the MoGo map corresponded exactly with the docking station kiosk unit. Perhaps I could simply use these locations to correct the data in OSM. But what about the license? Asked Dexter and ..
.. he confirmed that I could us it in OSM and that they would now update the data set on the main Open Data site. Open Data is more than a data source, it’s a conversation.
But first, had to screen scrape. Viewing source, the dock locations were stored in two lists of coordinates. I wrote a quick script to scrape this and transform into GeoJSON and then load into iD as a local file.
I had confidence in this data from the first two stations, but decided to confirm each location from street level imagery before adjusting. I zoomed in on each station from the local file in turn, enabled street level imagery, clicked to find the best view and recency of imagery, until I had a confirmation of the MoGo location.
It became a bit monotonous, though still interesting to investigate the streetscape across Detroit. I began day dreaming about a process that would make it easier for me to confirm. Something that would show the current and new point, automatically determine the best source of imagery from all available — biased towards most recent, and including machine learning to filter images that probably have a dock in view. Step through each point, reposition if needed, and confirm.
For the most part, the locations were spot on. This one confirmed with Mapillary imagery contributed by, in fact, Dexter!
And in just a couple cases, very close but slightly off. Like the station above, where the MoGo map has the station on the street side of the sidewalk, but OpenStreetCam had the station closer to the building. I decided to go as close as I could tell from the imagery.
I learned a lot from the short exercise.
- OSM is iterative. The first version of this data was pretty good, now it’s great. And we’ll need to update again as MoGo changes and grows.
- Open data is a conversation, not just a download site. Connect with data holders and it will help unlock more data and possibilities.
- Street level imagery is a superb source. Utilize as many sources as possible, research from multiple angles, and pay close attention to recency.
- Human review is always key, but we need to make it very easy with processes that minimize drudgery and take best advantage of human intellect.
- You can learn a lot about a city, even from far away. Takes patience but it’s rewarding to understanding the geography of a city.
There’s going to be a lot to learn about the city, and especially about the process of mapping, from making Detroit the best map in the world. Excited to see what happens next.
Mural by Ukuu Tafari of feeding crows art
If any of you all live in/around Richmond/Greater Richmond Region/Central Virginia (or any other localities in the commonwealth) and are interested in collaborating on open data/open mapping/civic tech/etc., email me: email@example.com.
Yesterday I had too many other obligations to travel far, so I took about three hours of Saturday afternoon to cruise parts of Ashgabat I'd not visited for a while and to collect Mapillary imagery where there were gaps (uploads currently underway). The dacha community north of the Karakum Canal on the far east end of the city is gone. It was apparently demolished in the past months and has been bulldozed. I had no idea, but will have to revise the map. The road into the community has been blocked with a dirt pile. This is why ground truth is so important.
I have completed corrections to the A-381, P-1, P-15, P-16, and P-18 national highway routes in Turkmenistan. I have consulted with local sources to nail down exactly which roads are part of each route and think these five are now pretty accurate.
Cet article est également disponible en français : http://www.openstreetmap.org/user/jbelien/diary/44358
Article written for open Summer of code 2018
Google is in charge
When Google Maps started more than 10 years ago, everything was completely free to use. These last few months that has changed drastically.
The first decision made by Google was to limit the free number of requests per day. It didn't really impact the small users that were still under the limit but some big players decided to switch to other solution like OpenStreetMap (for instance Foursquare and Pinterest).
The latest change (June 2018) has much more impact : Google decided to reduce the limit of free requests (25000 map display per day to 28000 per month - that's around 1000 map display per day, so 25 times less) but also made it mandatory to give your credit card number even if you do not go over the free limit !
Impact on Belgian Prime Minister website :
Money is definitely an issue but worst, Google can decide what's displayed or hidden or highlighted and how it is displayed. What if Google decide to "hide" or "highlight" certain kind of shops for instance ; even worse, what if they decide to draw the border between countries wherever they want. All of this could quickly lead to ethical issues !
Long story short, you have absolutely no control about this, Google can change the term of use of their service when they want how they want and it will most probably not be in your advantage !
Why OpenStreetMap ?
OpenStreetMap is a free, editable map of the whole world that is being built by volunteers largely from scratch and released with an open-content license.
The OpenStreetMap License allows free (or almost free) access to our map images and all of our underlying map data. The project aims to promote new and interesting uses of this data.
OpenStreetMap use is free as in free beer but also as in free speech !
Everyone can contribute and everyone can use it ! OpenStreetMap is open data ! Whether you're a developer, a scientist, an activist, a cartographer, ..., you can use OpenStreetMap database to fulfil your need.
You have access to the data
Google Maps doesn't give you access to the data behind the map (and that's the case for most proprietary solutions) but you have full access to the OpenStreetMap database !
Access to the data means you can build a map that suits you and not depend on decisions made by the map provider.
If you need some information, just download it ! Let's say you need all the buildings from Brussels for a research project, or you want to find the nearest ATM, or you want all the information from the country you're in vacation and you need it offline, ... ; those are all simple queries ! OpenStreetMap database is open !
OpenStreetMap covers the whole world with the same mapping rules so it's really easy to switch from local to global !
If you see that something is wrong in the map, just fix it, or notify the community and we'll fix it ! Fixing the map will not only benefit one big player but everyone in the world. OpenStreetMap is made by citizens just like you.
Power of the community
Thanks to hundreds of thousand contributors all over the world OpenStreetMap is probably the most up-to-date and complete map of the world.
When the city center of Brussels became officially accessible for pedestrian only, the update was immediately made in OpenStreetMap the day it became official ; same thing for the recent mobility changes in Ghent. Google Maps of course also made the change but it took a few weeks, sometimes months ... and that can lead to some issues (navigation, ...).
Third world countries are also a great example of the power of the community : humanitarian organization (like Doctors Without Borders, Red Cross, ...) often need maps to be able to prepare and coordinate their actions on the field. It usually costs a lot to buy maps from those regions (if there is a map available) so they made the switch to OpenStreetMap. They call us on a regular basis to ask the community to map part of the world where they have to take actions.
That's nice but let's get technical ...
Many libraries and tools are already using OpenStreetMap data.
You're looking for a baselayer, there are so much options available depending on your needs (standard basemap, basemap focused on cycling, on accessibility, ...). Have a look at the list available on the wiki : https://wiki.openstreetmap.org/wiki/Tiles#Servers and if you want to play with Vector Tiles, have a look at https://openmaptiles.org/.
OpenStreetMap Belgium also provides its own baselayers covering Belgium and surroundings : https://tile.osm.be/.
You're looking for a geocoding API, have a look at Nominatim : https://nominatim.openstreetmap.org/.
Everything that's possible with Google Maps is possible with OpenStreetMap and (almost) as easy to use and set up !
You can find more information on https://switch2osm.org/ !
You can also have a look at all the great libraries made by Mapbox : https://www.mapbox.com/help/how-web-apps-work/
That's probably the easiest way to switch from Google Maps to OpenStreetMap : Mapbox libraries and API allow you to have all the function provided by Google Maps (basemap - you can even style your own -, directions, geocoding, ...) and are all based on OpenStreetMap data. Most of Mapbox products are open-source but some services and functions come with a downside, you'll have to pay if you go over the request limits.
Some more reading (and sources used for this article) :
- http://geoawesomeness.com/why-would-you-use-openstreetmap-if-there-is-google-maps/ (by A. Buczkowski)
- https://www.citylab.com/design/2016/08/google-maps-areas-of-interest/493670/ (by L. Bliss)
In French :
- https://medium.com/@cq94/dont-be-evil-until-95f2e8dfaaad (by C. Quest from OpenStreetMap France)
- https://medium.com/@cq94/tarifs-google-maps-pour-les-nuls-d47c1833c242 (by C. Quest from OpenStreetMap France)
I have found an authoritative but slightly out-of-date list of provinces, districts, and municipalities in Turkmenistan, have corrected and updated it based on recent press releases, and posted it to the OSM wiki:
I am also working with my embassy's motor pool to identify the national highways, which are either not tagged or are in some cases mistagged. This will be a bit of a project over the summer as we research the Turkmen national highway system and bring the OSM map up to date. My ambition is to have a complete map of Turkmenistan's national highway network, all properly tagged.