Recent diary entries
This summer I missed DebConf16 in Cape Town and, being completely offline, not even the videos I followed this time. I hope I can meet many of my old Debian friends next year in Montreal, and I’ll try to slowly catch up with what happened during those days in Africa.
Luckily I had attended CubaConf and FOSS4G NA that fulfilled my needed dose of conference days and brought me many other friends and projects that I hope to keep up when I go back to real life. After almost 3 months of comings and goings of all kinds of trips, between conferences and family commitments, now I’m finally packing to go back home. It seems more than time to publish some late reports.
CubaConf took place in Havana from April 25th to 27th. It was the first international free software conference in the Island, with people from 17 countries, mainly from Latin America. All sessions in the main room were simultaneously translated to/from English/Spanish. The conference had a colorful and diverse environment, above the average of all the other tech meetings I’ve already attended.
CubaConf was an eye-opening experience for me with respect to:
The barriers for participation of Cuban people in free software development, namely poor connectivity and scarce hardware resources. Putting in perspective, can you imagine what would be your contribution to free software if you did not have a personal computer? Or if you could only access the Internet twice a week, for a short period of time?
The impressive capacity of Cubans to deal with and overcome obstacles. “We need to work with what we have. A crime would be to close our arms and do nothing”.
The alignment between the Free Software movement and non-capitalist values, and how Cuba is a fertile ground for Free Software flourishing. “Free Software for us is not a technical choice but a philosophy of life”.
The power of a community working together, which is much greater than the sum of all individual efforts. That was CubaConf.
Here are some highlights of the conference:
- Hamlet Lopez Garcia: “Las tecnologías libres en Cuba. Una mirada desde la cultura.” This was an awesome choice of keynote to open the conference and set up the solidarity environment we could feel throughout the meeting. Hamlet talked about the adoption and development of Free Software in the context of Cuban history and culture. If you read Spanish, I highly recommend Hamlet’s article “Las comunidades virtuales de software libre en Cuba” in the book Bienes Comunes.
OSM community sessions: OSM activities started before I arrived in Havana, with a talk given by PB during Flisol (Festival Latinoamericano de Instalación de Software Libre) and a bike tour in El Vedado to map POIs and take photos to mappillary. During the conference there were many OSM-related sessions: Introduction to OpenStreetMap by Wille, Mapping Ecuador after the Earthquake, by Ivan Terceros, OSM to map risk of flooding in Costa Rica, and a mapping workshop at the last day. The conference was offline, and Wille did a good job in preparing enough screenshots to show how to map using certain applications, and he also had the iD editor running locally so that we could simulate an edition in OSM server. The fact that we didn’t have satellite imagery as a background really puzzled me, cause the lack of Internet was not an issue of CubaConf only. Cubans in general have very little access to Internet, so in practice most of them cannot engage in base map drawing. Local mapathon activities need to consider that and plan in advance, either acquiring imagery in advance or using drones/balloons/kites for local aerial surveying. If none of this is possible, local contributions will be limited to filling nodes’ attributes, which is for sure a great contribution, but doesn’t use all the potential workforce Cubans could provide as OSM mappers.
Debian community sessions: Valessio gave a talk on his experience as a Debian contributor and how it was important for his trajectory from Jacobina, a small town in the Northeast of Brazil, to the world. I translated his speech to English, while he was speaking in “Portugnol”. During the unconference day, Maykel gave the talk “MiSOX - Personalización de Debian GNU/Linux, a su medida o necesidad”, where he presented this tool that allows to create Debian Derivatives using local mini repositories that contains most popular packages among their users. Then I gave a talk on the Universality of Debian Contributors, where I exposed the demographics of Debian Developers (which I suspect coincide with many other Free software projects) and talked about how it is important for the project to have contributors from diverse backgrounds if we want to move towards the utopia of an Universal Operating System. At the last day of the conference we made a Debian community meeting, which was named “the first MicroDebconf in Cuba”. There we shared our experiences, cases of successes and frustrations. By the end of the meeting we were all very inspired and planning a miniDebConf in the Island soon.
Cubaconf was organized by User Group of Free Technologies (GUTL) from Cuba and Best Of Open Technologies (BOOT e.V.) from Germany. The conference in itself was a very collaborative effort and faced many issues that are unthinkable for most people outside Cuba. It was with no doubt a great great success, many thanks for all the organizers and participants that made it real. Here is a nice video put together by Valessio, to share with the world the energy of those special days.
Back from Cuba, I spent a couple of days in Montreal then I flew to Raleigh, North Carolina. FOSS4G NA happened between May 2nd and 5th, followed by a geotour at the university on May 6th.
My highlights of FOSS4G NA were the contact with the GRASS community, playing with tangible interfaces, meeting OSM/HOT members and figuring out many free software solutions and challenges for fields that are largely dominated by proprietary software.
Here is my list of memorable sessions, you can follow the links for more info and slides:
- Using Grass GIS through python and tangible interfaces
- Managing Versioned Data with QGIS using GeoGig
- HOT-related sessions:
- Portable OSM - OSM in the Disconnected Wilds
- Putting 14 Million People on the Map: Revolutionizing crisis response through open mapping tools
- UAV-related sessions:
- From UAV to Orthomosaic: Building a Toolchain
- Balloons, Kites & Poles - Low-cost ways to capture and analyze open geodata
- Open Source Photogrammetry with OpenDroneMap
- Open the classroom window for… DRONES!
- GRASS-related sesions:
- Tangible interaction for GIS
- GRASS GIS loves lidar
- Other projects I want to follow:
- Visualizing GPS Tracks with RikiTraki
- GIPPY: A high performance geospatial image processing library for Python
- Terra Populus: Free Human-Environment Spatial Data
- Bringing OpenStreetMap to the third dimension with 3D Tiles and Cesium
I came back from both conferences inspired and full-charged to keep my efforts towards lowering the barriers for people from all backgrounds to engage actively in Free Software projects. Yes we always need more contributors, but more important than our needs, the opportunity to contribute to a Free Software project is everyone’s rights.
This is my last post of the series, which is also my last pending task of the Outreachy internship. I’m late with this wrap-up, sorry if you were waiting for that, but during the school year I had a hard time to keep up with my activities related to the plugin. Now that I’m on vacations I’m closing the cycle.
On the 1st of October 2015 we presented to the community the work accomplished by HOT Outreachy interns. My intervention can be seen through the HOT Youtube page, and I’ll try to summarize it here.
Looking back at the experience with Outreachy, I realized that my journey was not very typical. I have a background with computers and I’m currently studying environment, but I’ve been a free software enthusiast for many years now. I’ve started contributing to HOT as a remote mapper in early 2015, then I’ve chosen Outreachy as my summer project. With HOT I had the chance to use my technical skills for something that I was already interested by other means (its humanitarian aspect).
Since many of the projects proposed by HOT interested me, I’d say my first challenge was the choice of the topic I would be working on during my whole vacations. HOT as an organization had proposed 17 ideas, from which I selected the following 3: creation of moodle courses, translation workflow for LearnOSM and OAM QGIS plugin. The last one was a good opportunity to connect both of my areas of studies, and that was the main reasoning of my choice.
After that, my next challenge was to produce a Hello World QGIS plugin as a prove of concept, also to serve as my first contribution to OAM. Since I had very few experience with GUI programming, this was not a very easy task, specially considering that the application deadline was at my exam period for the spring term. The successful application came only 2 days before the HOT Summit, where I had the change to meet the international HOT community, specially Cristiano and Drazen who were my mentors.
From there, I had 3 months of challenges, from learning the best practices in the development of QGIS python plugins, to choosing my development tools, and learning more about OAM and the many acronyms that surround the project. My previous posts outline the development phases, please refer to them for more details:
- Project setup and research
- Plugin features and designed
- Upload functions
- From tabs to wizard
- Outreachy wrap-up
The release of the experimental package happened on September 12th 2015. At this moment, 1383 downloads were made, and a couple of bugs were reported by users. We currently have 20 open bugs, which we consider to be outside the scope of the internship and will be dealt by the development team as the plugin evolves. A new developer has joined us, Yogi, as I already mentioned in a previous post.
In terms of the internship experience, I can point out as skills learned during those 3 months (and more): python and Qt development, basic concepts of remote sensing and image processing, writing and communication skills, time and tasks management, and the experience with the OAM team, through which I could follow and contribute with the project being shaped, since the beginning.
Now that the internship is over I need to keep up with my studies, so development will definitely slow down from my side. At the same time we have plans of refactoring the code and many open bugs, so we really need more people to get involved. I also want to make space in my schedule to go back to mapping, as I first contributed to HOT. And as a final future plan, I’d love to work on the field. I’d like to be in contact with people on the ground, work with outreach and training, follow the results of our collaborative work from closer and closer, and help with whatever skills I have to make it be better used.
Thank you OAM, HOT, OSM and the Outreachy program. Thank you everyone who helped me along the way, and thank you those who will join the team and help us with development, testing, bug reports, or in any other way.
For 2 weeks in August I was “abducted” by DebConf and that slowed down the development of the plugin from my side. Fortunately we have a new coding contributor in the team who kept the ball rolling while I was busy, thanks @yojiyojiyoji!
At DebConf I had the chance to show the plugin to some folks who gave us some good ideas of how to make it more user-friendly. I also managed to gather some potential translator contributors, who I’ll poke as soon as we have the strings file ready to be translated.
As a result of that, we decided to change the design of the uploader from tabs to wizard, due to its step-by-step nature. Apparently users expect an unrelated and non sequencial flow with tabs, which I was unaware of, but it makes a lot of sense indeed. With the help of yojiyojiyoji I migrated all the uploader code to this wizard design.
Here is how the new design looks like for the final wizard page:
Now we have space for short instructions at the top of each page, this is very helpful. Also, buttons for page navigation are already implemented by the wizard, and are always at the same place, which is a good usability point. The biggest challenge of this design change was to place the QgisMessageBar inside the wizard. After some workaround it is working, but we definitely need to improve the code, there should be a better way of doing that :-/
I’ve also designed the dialogs for “Edit settings” and “Search imagery”, take a look at those screenshots too:
I was hardly blocked to propose a design for the search dialog. Anything I started to do I threw away after a couple of edits. But I’ve finally accepted that this is what I have for now, it is a starting point, and I’m happy to receive comments and suggestions of improvement. If you look back at my first mockups for the uploader you can have the feel of how things evolve along the development process ;-)
My idea is to have a thumbnails browsing as the result of the search and checkboxes beside each one of them, so the user can select the ones to be downloaded. I still don’t know how to implement that with Qt and I couldn’t find those elements in Qt designer, that’s why the mockup has only this blank widget.
My following tasks are basically packaging, testing and documenting. With this I’ll close my internship commitment, but I really hope to continue the development with the rest of the team after that.
On a side note, HOT Outreachy interns will present the results of this experience to the HOT community. It will happen by mid September, stay tunned for more details!
If you are a drone mapper tired to see your imagery sitting in your own hard drive, stay tuned on OAM, it is coming to make sharing a much easier task!
OAM stands for OpenAerialMap. It is a collection of services related to sharing open imagery: uploading, hosting, indexing, searching, and using tile map services. Not only drone mappers, but also government agencies and satelitte imagery provider are expected to join the network, enriching the pool of available resources to be easily searchable by users.
HOT currently leads the development of OAM aiming to provide a simple solution to host and share imagery for humanitarian response and disaster preparedness. But it is worth noting that the final products won’t be restricted to humanitarian use. All the available imagery will be released freely under a Creative Common license, so one could use it for a variety of purposes. For instance, students and researchers will greatly benefit from OAM when looking for open imagery of specific areas.
I am working now on the development of a Quantum GIS (QGIS) plugin to communicate with OAM services. It will basically function as a desktop client for OAM. Using this plugin, GIS users wil be able to upload imagery to OAM, trigger the creation of web map services, search, browse and download imagery from the catalog.
This week I’ve just finished implementing the upload functions. Let’s have a tour in what we have so far.
Through the Imagery tab the user can select data to be uploaded. It is possible to select layers currently loaded on the QGIS map canvas or any file existing on the computer, either a single GeoTiff image or a collection of files also known as VRT. Multiple sources can be selected and added to the upload list. Reordering of sources is also possible using the up/down buttons, and the upload will be performed on the order provided.
The “metadata” tab has been designed to allow the user input information that will compose the metadata sent to OAM catalog, along with automatically extracted data from the imagery files. The user must first select one or multiple data sources for which the current input refers to, and go on with filling the form.
The last tab, “uplodad”, is where the user will review the full metadata file (generated + input info), select optional upload options, choose the upload destination and finally perform the upload.
If you want to have the full experience of our plugin in its current experimental stage, you can download it from our repository and follow the install instructions. Our next milestone is to develop the search and browsing functions. So stay tuned for the upcoming commits!
The OAM QGIS plugin team holds regular meetings on Wednesdays at 19:30 UTC and the general OAM team meeting happens on Thursdays at 18:00 UTC. Please don’t hesitate to show up and give your contribution, in the development side or with reports of user experience.
Thanks for reading!
It’s been a while since my last post, so I just wanted to give a short update about the plugin development stage.
This week I’ve finally managed to have a working code for the upload feature ;-)
All signals and functions for the imagery tab are implemented:
- Select files from file system
- Load QGIS layers (getting the corresponding file system path)
- Add/remove selected sources
- Rearrange order of sources to upload (up/down buttons)
I still need to work on the package for QGIS which will include all the dependencies, and upload it to qgis repository with the tag experimental. In the meantime, you get get the code at our Github repository and follow the install instructions at the README file.
The interface has also evolved a bit since my last post, and an update on that will come in the short future.
As always, feedbacks are very very welcome, please don’t hesitate to play with the plugin and get in touch.
Plugin required features
We have listed the following features to be included in the plugin first version:
- select input data (from individual files, a VRT or a loaded layer)
- insert/load/change metadata
- validate metadata
- choose OIN upload destination
- upload transaction
- notify OAM of new OIN resource
- trigger tile service on OAM
- authenticate to OAM (if requesting tile service)
- provide additional OAM metadata (if requesting tile service)
- re-projecting to EPSG:3857 (optional)
- convert format to GeoTIFF RGB (optional)
Regarding the metadata acquisition, we didn’t discuss yet about what can be extracted automatically from the GeoTIFF and what needs to be informed by the user.
The list of features guided the plugin GUI design. I’ve first chosen Pencil as a GUI prototyping and sketching tool. It is a cross-platform tool based on mozilla firefox which can be installed as a Firefox extension, being accessible through its tools menu.
These quick tutorials demonstrate what can be done with pencil:
The draft mockups I’ve made with pencil are available in our repository. Here is the extracted png image:
Later on, when I was studying Qt-creator I discovered Qt-designer, a GUI prototying tool that does exactly what I needed for my draft mockups with the advantage that it generates code. With Qt-designer, the generated .ui file (a xml) can be later compiled into python code using the command pyuic4.
The other advantage of using Qt-designer is that we can be sure that every element placed in the GUI will have a Qt representation, while with Pencil we could be planning to use elements in a way that might be difficult or even not possible to implement. And even though both programs generate xml files to represent graphical elements, the one genereted by Qt-designer is much cleaner and can be easily edited with a text editor if needed.
The wireframes developed with qt-designer (xml and images) are also [available in our repository] (https://github.com/hotosm/oam-qgis-plugin/tree/master/GUI_design/pencil). Below you have a snapshot of the current version. We are willing to have feedbacks to evolve the design, so if you have any comments or criticism, don’t hesitate to get in touch!
Debbugging and version control system are some of the features integrated in the development enviroment (within the IDE). Another important tool is the management of translations. We can mark strings as translatable let Qt-creator take care of the rest. Among other features there are code highlighting, various navigation/documentation short-cuts, sessions saving, kits (set of settings) and screen split.
Qt-creator manual can be consulted as a reference. For a quick demostration of the IDE I’ve watched some videos, such as “Introduction to Qt Creator” by Tobias Hunger, a software engineer from the Qt Company.
Here is the demo summary, for future reference:
- Ctrl-K: powerfull tool for searching code (“c “ search for classes)
- ESC: go (closer) to code
- Alt-Left/Alt-Right: history-based cursor navigation
- Ctrl-Tab/Ctrl-Shift-Tab: pops-up and rotate among open documents
- F1,F2,F4: Help, follow symbol, switch header/source files
- Ctrl-Shift-U, Ctrl-Shift-R: find usages, rename
- Alt-Return: quick fix
4. About Qt development
My next task in the project, scheduled for the following two weeks, is to rewrite the trial plugin used in my Outreachy application to include the list of required features listed above, but this time using qt-creator and QgisPluginCreator. So I thought it was time to study Qt more seriously.
I’ve decided to go for the basics and searched for a book, which usually builds up concepts without assuming previous knowledge from the reader. I am reading now “Rapid GUI Programming with Python and Qt: The Definitive Guide to PyQt Programming”, and I still have a lot of reading to do…
Basic project setup
- Use qtcreator for GUI layout/development and for defining most of the common signals/slots for the GUI components
- Do not change any generated code directly but create a subclass and then override/extend it, to keep our code and the automatically generated code separated
- Use QgisPluginCreator that has some basics setup (eg. internationalzation support)
- Package any external python modules as part of the plugin
OpenAerialMap (OAM) is an open service that will provide storage and search within a collection of openly licensed satellite and unmanned aerial vehicle (UAV) imagery. OAM will facilitate the processing and provision of imagery for humanitarian response and disaster preparedness, making it easier to determine what imagery is available and where it is. [OAM Catalog] (https://github.com/hotosm/oam-catalog) will be used for indexing and OAM Browser for searching.
The Open Imagery Network (OIN) is a consortium and specification of imagery providers led by HOT and Planet . Imagery indexes from all participating providers will be merged into the main OIN index through a GitHub repository. Providers will host an index of all their dataset and add the URI to the main OIN index in GitHub.
The OAM Server will be responsible for processing data directly from providers of aerial imagery or from existing Open Imagery Network (OIN) nodes . Once a new imagery is uploaded, the Server will trigger a tile engine instance to create a tiled map service and save it to a specific storage (eg. S3 bucket). It will also create the needed metadata and submit it to the OAM catalog to be indexed and easily searcheable. Check specifications for OAM metadata and OIN metadata for more details. Each stored image will carry such metadata.
The development of the OAM server is schedule for the period between June 15h and August 15th 2015 . Gitter chat is used as the main channel of communication with weekly meetings on Thursdays at 18h00 UTC.
There are so many new terms involved in this work that having a list of acronyms will give a hand to our lazy memory.
I’m one of the OSM Outreachy interns and the aim of my project is to develop a QGIS plugin for OpenAerialMap.
I’ll use this diary to document my progress, and I’d love to have feedbacks from the community along the way.