OpenStreetMap

OSM Route Manager and History Viewer

Posted by Candid Dauth on 30 July 2010 in English (English).

OSM Route Manager and History Viewer have been down for about three months now. As I have written earlier, the reason for me to disable them was thir massive memory consumption. I have just put them online again, although they don’t work well yet.

Originally, Route Manager was entirely written in PHP. As splitting large relations into segments is a CPU-intensive operation, I rewrote the logic in Java but kept the PHP web pages. The PHP page would start the Route Manager Java executable for each request and after its completion read out the result from an SQLite database. I adapted the same concept for OSM History Viewer.

The problem with this approach was the massive memory consumption that had two reasons:
1. There was no limit on concurrent requests, so 10 concurrent route analysations would have caused 10 Java Virtual Machines to run, and each of them takes a lot of memory by itself.
2. To analyse a large relation or changeset, all of its members had to be loaded into memory.

I solved these problems more or less in the following way:
1. I rewrote the PHP pages in JSP so that both applications and all simultaneous requests would run in one JVM.
2. If too many objects are in the memory, some of them are dumped into a Postgres database (which increases the CPU usage and thus the analysation time).

The server I run the two applications on has 512 MB RAM and a very slow SAN hard disk connection (with 1–2 MB/s), so swapping is very slow. With the old PHP versions of the applications, my server would completely hang up when too many JVMs were running because of the lot of swapping to do. This was unacceptable because I also use the server for e-mails. With the new JSP versions, the memory usage of the applications is limited. I have not yet found a way to find out the actual memory consumption of a process under Linux, but the VIRT column currently shows 368 MB and the RES column 186 MB for the servlet container, so the actual usage must be somewhere in between.

This is still high memory usage, but at least it prevents my server from hanging up completely. Unfortunately, due to the excessive usage of the Postgres database to minimise the memory usage, everything takes very very long. Somewhere, there seems to be a limit of 5 minutes after which the web pages stop loading, although the servlet container in the background still keeps calculating. So at the moment, you still cannot analyse large relations or changesets. I am planning to create some asynchronous user interface where you are notified via e-mail or similar once the analysation is complete.

If you have a server running a servlet container and have some resources left where you want to host the two applications, that would make things a lot easier for me. Please contact me in that case.


Login to leave a comment