Libosmscout in 2015

Posted by Framstag on 3 January 2016 in English (English)

The end of the year (And the start of a new one, as I did not finish this article in time ;-)) is a good time to review the progress of libosmscout in the year 2015. As with likely every project at the end of the year one has the feeling, much more should have been done and there is even more left to be done in the next year - the more project time passes the more time is required to finish the project.

A retrospective is a good approach to correct this feeling and move things into a more objective perspective. After writing down all the things achieved in 2015 I'm of the opinion, that 2015 was a good year for libosmscout :-)

What happened in 2015?

Bug fixes: As always 2015 brought a number of important bug fixes. Things didn't work or broke - we fixed it.

Code cleanup and C++11 only: We did a huge number of code cleanups. Libosmscout is now C++11 only, requiring recent C++ compilers. On the other hand C++11 allowed us to (unconditionally) use things like unordered_map/set, lambdas, auto, range-based for loops and most important std::shared_ptr. Code got simpler and readable because of this. Using std:.shared_ptr allowed us to drop our own custom reference counting solution.

"Feature"-feature: Long planed and finally realized in 2015: The "Feature"-feature. Features allow the application developer to evaluate its own OSM attribute combinations, inject resulting data into the database and retrieve them back during rendering. Libosmscout itself now uses features to map attribute information into the database. Obvious candidates for further custom Attribute are for example : ele, wikipedia links or similar. Features also allow you to add custom labels and the rendering of multiple labels.

Disk size, performance, memory footprint: As every year 2015 also brought further optimizations of the on disk database format, resulting in less disk space been required and faster loading. We also did a number of performance optimizations, changing and improving the various indexes, data loading and rendering code. We also did improve the style sheet. Compared to the performance at the beginning of the year we got measurably faster. We also reduced the memory footprint by dropping some internal caches. Memory footprint however will increase again with more tile based caching and more caching in the renderer.

Automatic builds and more supported platforms: See the recent blog article regarding use of cloud based CI platforms. We now use Appveyor and Travis-CI to regularly builds libosmscout for various platforms. The build in general was improved, too. Configure scripts have on one hand been improved and on the other hand been simplified. A global Makefile is available to build everything in one go. 2015 also brought Visual Studio 2015 support for the core libraries and tools and reintroduced MinGW/MSYS2 support. Using above services allows use to better support these platforms, while spending less time for it. Thanks for the people at Appveyor and Travis-CI for their support for open source software! Note also the OpenHub web page for libosmscout.

CMake: We now have simple CMake support for all important sub projects. The support is not enough to actually build libosmscout but is a good start for anybody interested in further improving the CMake support. It is currently used to import libosmscout into Clion.

Logging-API: We now have a simple logging API allowing us to redirect output in to to a file or some platform specific logging sink.

Better Location-API: Location API got some major bug fix resulting in better handling of regions. The API also got new methods for describing the current location.

Import of multiple data sets: The importer now allows import of multiple data files. This allows to enhance the database with additional information, one use case being addition of SRTM contours.

Conditionals in style sheet and "night"-mode: The style sheet was enhanced with new rendering options. Especially its is now possible to to use conditional statements. Conditionals can be influenced by the application code. One practical use was the additional of a (rudimentary) night mode variant for the default style sheet.

New tools: We added new tools (DumpOSS, performance tests) and improvements of existing debugging tools (DumpData).

More documentation: We added more documentation to the project. The installation documentation at the OSM wiki has been revised, new README files were added. Also inline source code documentation was improved.

OSMScout2: The OSMScout2 demo was enhanced, now looks nicer and better shows the libosmscout features.

Multi-threading and caching: The end of the year brought initial multi-threading support for database access. Most data files can now be accessed from multiple threads. A new cache based on tile cells was introduced, improving the database access time by better and faster caching. The cache also uses multiple threads to load and process data from different data files in parallel. 2016 will see more improvements in this area.

Finally I was able to join two hacker weekends organized and (payed) by the the German FOSSGIS e.V. Thanks for that. The participation allowed me to work an bigger features in one go, giving libosmscout a burst.

My thanks also to all the people that showed their interest in libosmscout by writing on the mailing list or connecting me privately. Some of them also offered patches and improvements. Please continue to do so. I also encourage people interested in libosmscout or who are interested in general to work on new renderer implementations, routing or who would just like to try new things without starting from scratch, to contact the project. Libosmscout is not only a good choice for app development, but the amount of time already spent to define simple APIs and get things "fast" is also a good starting point for your experiments. Concentrate on your goals while using libosmscout. Libosmscout allows you to get to the point fast and concentrate on the things that are important for you. Please join, you are welcome!

So what is planed for 2016?

More optimizations: Of cause there are still ideas to make things faster. Libosmscout rendering is still a little bit too slow on older devices.

Multi-threading: 2015 will bring more support for multi-threading. More things will be don in parallel, more things will be callable from multiple threads.

More caching: The tiled data cache will get further improvements, allowing background loading of tile data. But I plan to work on caching of data in the renderer, too, likely improving rendering performance, too.

Improved data layout: Also on the list for 2016 are enhancements in the way the importer preprocesses data. Some planed changes will better allow using multiple databases for rendering and routing. Being able to embed node data into way and area data another point on the list. I also would like to experiment with enhance the data format to make use of complex ways consisting of multiple segments (with different attribute values), thus reducing the number of object to be rendered.

Support for Java using SWIG: I would also like to create an initial project setup for adding Java support via SWIG. If anybody has general SWIG project setup example code, please contact me.

More style options: After changing the renderer API to make direct use of tiling and adding support for caching within the renderer I would start to work on adding more features to the style sheet syntax and the renderer. Better label placement and rendering is high priority, but there other features like rendering labels on or beneath area borders, background images, stripes that are on the list.

Exception handling: I would like to switch to error handling via exceptions. The current solution to return bool results in ugly if-cascades and in general produces difficult to read code.

Borderlines: Better support for rendering of borderlines was also requested in 2015: IMHO the best approach is to generalize the approach implemented in the water index, allowing a more generic generation of indexes for similar cases.

Generation of mapping error reports: During processing a number of mapping errors are detected. This are complex things like resolving of relations or errors in the generation of the location index (most often spelling mistakes) but also simple errors occurring during parsing of individual attributes. Scanning these errors (and fixing them) can improve mapping and will (if feedback is given) also improve the quality of libosmscout.

Google Summer of Code 2016: Like nearly every other open source project libosmscout lacks people that actively participate on development. I'm in principle willing to support people that would like to work on libosmscout during the Google Summer of Code as mentor. Experience shows however that to make a good proposal and in result successfully get chosen, people should already have some experience with the project they would like to help out beforehand. IMHO in consequence now is the time to contact the project.

Location: Unionviertel, Dorstfeld, Dortmund, Regierungsbezirk Arnsberg, North Rhine-Westphalia, 44137, Germany

Comment from stephan75 on 5 January 2016 at 17:59

Nice to see progress, and thanks for all your efforts!

By the way: Where does main development of Libosmscout take place actually? on github or still sourceforge? Maybe give a hint on the sourceforge website?

And is the OSM wiki page about Libosmscout still up-to date? see

Greets, Stephan

Comment from Framstag on 5 January 2016 at 18:15

Development takes still place at sourceforge (repository, mailing list). The GitHub repository is only a manually synced copy of the main repository an was required to get Travis and Appveyor to work (the sync could get automatised though with some effort but currently it is more convenient to not have such automatism).

The OSM Wiki page is still current (I updated it recently), the sourceforge web page on the other hand is badly out of date.

I'm lacking time for pushing development and enhancing the web page at the same time :-/

Best place for getting up to date information is thus the sourceforge mailing list.

Comment from Framstag on 1 July 2016 at 08:35

There has now been a recent update of the libosmscout web pages. I also spend some effort to make it much easier to keep the web pages current.

Login to leave a comment