Scalable Aerial Imagery Generation from Phone Lidar and 360° Point Clouds
Posted by tordans on 24 June 2023 in English. Last updated on 25 June 2023.Image Source: Jake Coppinger
Companies like Mapillary and Kartaview have played a significant role in advancing OpenStreetMap (OSM) and enabling detailed mapping efforts, particularly in urban areas. While 360° street-level images have been crucial for capturing high-quality data, there is a growing need for a scalable solution that leverages the power of aerial imagery to further enhance community mapping. In this blog post, we delve into the potential of aerial imaging and its implications for OSM mapping. Interestingly, emerging companies in the 360° imagery space, such as Mapillio (Commercial) and GeoViso (Open Source), may view this as an opportunity to add a unique selling point to their portfolio. The process of generating detailed aerial-like imagery for specific smaller areas not only benefits OSM mapping but also proves highly valuable for city planners involved in intersection redesigns or the addition of bike paths to streets.
The Power of Mapillary and Street-Level Mapping and the Need for Aarial Imagery
Mapillary, in particular, has been crucial for mapping efforts in cities. It allows the mapping of intricate details on sidewalks and bike lanes, empowering communities to collect data and map it later. Unlike professional street-level projects that primarily rely on car-based perspectives, Mapillary enables data collection from the viewpoint of pedestrians and cyclists. This perspective is essential in cities with numerous parked cars, as it offers better visibility of sidewalks.
However, despite the benefits of 360° images, placing objects accurately on the map remains a challenging and manual process. Mappers often rely on guesswork and visual alignment with other objects, making it time-consuming and prone to errors. This is where aerial images shine, as they simplify the process by allowing mappers to place objects directly on the image, eliminating the need for guesswork and enhancing accuracy.
Classic Aerial Images Are Great, but Imagine Creating Them Yourself
In Berlin, Germany, we are fortunate to have access to fresh aerial images every year, which greatly aids our OSM mapping efforts. You can check out an overview of these images here. This valuable resource allows us to create accurate and highly detailed maps of the city and its streets. However, many places around the world lack this kind of readily available data. Even in Berlin, there are streets that are constantly shrouded in shadows or obscured by angles, building shadows, and tree cover, making it challenging to accurately map the street space.
When it comes to mapping intersections in intricate detail, incorporating 360° images and local knowledge becomes essential to create a comprehensive map. However, this process can be time-consuming and reliant on on-site visits or contributions from local community members.
This is where the potential of self-created aerial imagery becomes truly exciting. Not only would it offer high accuracy in mapping, but it would also open up possibilities for remote mapping in areas where it was previously impossible. By generating aerial images on our own, we can overcome the limitations imposed by the availability and quality of existing aerial imagery, enabling us to map with precision and detail from anywhere in the world.
Drones: A Complex Solution
Traditionally, the community relied on drones to collect aerial images. However, drone operation is expensive, complex, and often requires permissions and road closures in urban areas. The high cost and logistical challenges associated with drones make them less accessible for community mapping purposes.
A Scalable Solution: Generating Aerial Imagery from 360° Images
A scalable solution that leverages existing 360° images or point clouds to generate aerial imagery would revolutionize community mapping. This approach offers several advantages, including lower costs compared to drones and greater accessibility, as 360° cameras are more affordable and can be used by almost anyone, anywhere. This makes it an ideal choice for mapping in low-income areas. It’s worth noting that Humanitarian OpenStreetMap Team (HOT) should be particularly interested in this type of processing, given its potential impact.
The Ideal Workflow: Uploading, Processing, and Geo-Referencing
In an ideal scenario, a website would facilitate the upload of 360° images, handle the processing, and provide tools for easy geo-referencing. Ideally, the geo-referencing process should be automated, with an option to make manual adjustments if needed. The end result would be a flat image that serves as a basemap for mapping purposes in tools like iD and JOSM.
Proof of Concept: Aerial Imaging with 360° Images
A proof of concept by Jake Coppinger demonstrated the feasibility of creating aerial imagery based on 360° camera footage. In his blog post, he outlined the technical steps involved in generating such imagery. Although the proof of concept showcases the potential, the current complexity of the process hinders immediate implementation. However, if a project dedicated to processing 360° images were to integrate this workflow, it could offer a seamless experience for contributors worldwide.
Image Source: Jake Coppinger
Lidar: An Even More Promising Technique?
While 360° images have shown promise, Lidar technology presents an even more compelling option. With modern phones equipped with Lidar sensors, such as the iPhone 12 Pro (2020) and iPhone 13 Pro (2021), one can skip a step in the processing pipeline. Again, Jake Coppinger documented a proof of concept in his blog post, showcasing the generation of aerial imagery using an iPhone’s Lidar sensor. However, several challenges, including device availability and issues in the Open Source Ecosystem, need to be addressed to make this process more accessible. Additionally, platforms like OpenAerialMap need further development to support this type of data. Nevertheless, integrating Lidar processing into existing 360° image platforms could streamline the workflow for contributors.
Image Source: Jake Coppinger
Evaluating 360° Images vs. Lidar
Ideally, processing pipelines should be compatible with both 360° images and Lidar data. Further experimentation and testing are required to evaluate which technique works best in different scenarios. Lidar offers the advantage of skipping a step in the process but has limitations in terms of device availability. On the other hand, 360° images are well-established and provide additional street-level information, but determining the optimal number of images for generating high-quality aerial imagery requires more investigation.
Exploring Future Possibilities
While Mapillary has experimented in this area, it is uncertain whether they will introduce a feature for aerial imaging in the near future. However, other players like Mapillio and GeoViso could potentially embrace this feature, differentiating themselves from existing platforms. For now, this blog post aims to inspire further experimentation in the field of aerial imaging for detailed OSM mapping. Share your findings and contribute to the advancement of community mapping efforts!
PS: ChatGPT assisted in writing this blog post. I initially created a hierarchical outline of rough notes, not focusing too much on grammar or typos. I then inputted this into ChatGPT to generate a first draft, which required some paragraph-by-paragraph adjustments. I used ChatGPT once again to fix any typos and grammar issues. You can read the whole Chat that created this post. Overall, it was a helpful process. While the tone of this post may not entirely reflect my own, the information is presented more effectively than I could have managed given the limited time I have available for writing such a post. — Update: It turns out, ChatGPT does not know how to type aerial imagery but instead applied a typo I had in my input notes.
Discussion
Comment from tordans on 1 July 2023 at 13:07
Update: GeoViso project wrote in https://gitlab.com/geovisio/blurring/-/issues/3
Comment from cbeddow on 14 August 2023 at 17:04
This is fantastic research and documentation! I know some experiments have happened with Mapillary imagery in the past in this regard.
What also could be very interested is to track pixels in this process, in order to turn detections of roads, buildings, sidewalks, vegetation etc into 2d vector polygons, since Mapillary already has open data about the image segmentations and labels.
I don’t see this being pursued in depth during 2023 via Mapillary, as the agenda is quite full with other great stuff, but I hope to share this around and see where it can generate interest with colleagues!
Comment from tordans on 14 August 2023 at 20:41
Thanks you for sharing your insights, cbeddow!
One thing I notice when talking about this topic: From the outside it looks like since the beginning a lot of Mapillary’s work is spent with extracting features from the images via AI. However, in my OSM mapping experience, the data generated from this is still not a relevant factor when mapping (and I tried many different approached over the years). On the other hand, good areal imagery is the base for more or less every map edit in OSM. And together with the 360°-point of view its the basis for most of our mirco mapping efforts in Berlin (eg. https://strassenraumkarte.osm-berlin.org/?map=micromap).
Looking at it from a “cost of data production” vs. “features added to the map” point of view it looks to me that this aerial imagery approach could win by a huge margin. I just hope we are not looking at the cool AI-kit all the time missing the boring image processing tool sitting right there, ready to be useful right away.
I am glad to hear you see a possibility for more experimentation in this area!
Comment from tordans on 12 October 2023 at 05:55
https://toot.cafe/@impiaaa/111218551039688662 Shows a great example how to micro map a newly opened park using Jake‘s tutorial.