Recent diary entries
There have been some API changes in OSM some months ago, which caused OSM History Viewer to fail with NumberFormatExceptions on all changesets where something was deleted. This basically meant that History Viewer had been unusable for several months now. The issue is now fixed.
I have finally had the time to build in a timeout for OSM Route Manager and History Viewer.
Previously, the only limit was the memory of the Java process, which caused some requests to take really really long. The queue got very long and the service became unusable. (And before that, there hadn’t even been a memory limit, the service would just dump cached items from the memory and re-fetch them from the API when needed.)
Now, I have set the timeout to 10 minutes. I know that this might not be enough for the “Relation blame” of medium to big relations, but at least it keeps the service usable.
Since the introduction of the queue in Route Manager and History Viewer, there have been several down-times where the queue became very very long without anything being processed. The reason why nothing was processed was that someone tried to use the “Relation Blame” on very big relations, such as http://www.openstreetmap.org/browse/relation/107796. I have therefore disabled the Relation Blame function for now.
The reason why the queue got so long (sometimes several thousand requests) is probably caused by bots. Probably beacuse of the links that the Wiki automatically creates to the two tools. I suppose I will have to add a robots.txt, and if that does not help, a Captcha.
For a long time, I’ve wanted to implement something similar to the relatively new Google Maps feature where you can click on POIs (such as companies or restaurants) and a popup displaying details opens.
Now I’ve come accross OpenLinkMap (olm.openstreetmap.de, Wiki), which seems to do exactly this. So I’ve created an OpenLayers layer for it and added it to the Beta version of FacilMap available on beta.facilmap.org. On zoom levels >= 13, you’ll see these transparent dark-blue circles around some POIs. Click on them and a popup appears with some information about the POI.
At the moment, OpenLinkMap seems to only contain POIs that have either a telephone number, a website or a Wikipedia article. Hopefully, in the future other POIs will be included as well so that one can see their address and opening hours.
(Once the OpenLinkMap layer is considered stable, it will be added to the stable FacilMap API on http://api.facilmap.org/facilmap.js.)
Then add the layer to your map using the following code:
This feature is documented on http://wiki.openstreetmap.org/wiki/FacilMap/API/POIs.
OSM Route Manager and History Viewer are now hosted on the FOSSGIS development server under the new URLs http://osmrm.openstreetmap.de/ and http://osmhv.openstreetmap.de/. I will continue to use the old URL (http://osm.cdauth.eu/route-manager/ and http://osm.cdauth.eu/history-viewer/ ), which now redirect to the new locations.
I have implemented some new caching and queueing mechanism so that only one request is processed at each time (to avoid a too extensive use of resources and API calls). Hopefully, the service will finally run acceptably stably due to these changes.
Dominik Mahrer (http://www.teddy.ch/) has offered me to host OSM Route Manager and History Viewer, as I was experiencing a lack of resources that forced me to take them offline. Many thanks to him!
The tools are available on the old URLs (http://osm.cdauth.eu/route-manager/ and http://osm.cdauth.eu/history-viewer/ ). Some things don’t work quite right yet (such as the nasty Google Maps API key warning appearing all the time), but at least the tools are usable again.
I am currently working on improving the speed a little bit by using better caching, hopefully it will be as usable as it used to be some day again.
cdauth’s map and its API are now called FacilMap and is available on http://facilmap.org/. I will try to keep all old URLs alive. People who want to make use of new features have to migrate their code following the guide on http://wiki.openstreetmap.org/wiki/FacilMap/API/Migration_guide_from_cdauth%E2%80%99s_map.
The old locations will redirect to the new one for some time.
The source code is now developed on http://gitorious.org/cdauths-map/openstreetbugs.
OSM Route Manager and History Viewer have been down for about three months now. As I have written earlier, the reason for me to disable them was thir massive memory consumption. I have just put them online again, although they don’t work well yet.
Originally, Route Manager was entirely written in PHP. As splitting large relations into segments is a CPU-intensive operation, I rewrote the logic in Java but kept the PHP web pages. The PHP page would start the Route Manager Java executable for each request and after its completion read out the result from an SQLite database. I adapted the same concept for OSM History Viewer.
The problem with this approach was the massive memory consumption that had two reasons:
1. There was no limit on concurrent requests, so 10 concurrent route analysations would have caused 10 Java Virtual Machines to run, and each of them takes a lot of memory by itself.
2. To analyse a large relation or changeset, all of its members had to be loaded into memory.
I solved these problems more or less in the following way:
1. I rewrote the PHP pages in JSP so that both applications and all simultaneous requests would run in one JVM.
2. If too many objects are in the memory, some of them are dumped into a Postgres database (which increases the CPU usage and thus the analysation time).
The server I run the two applications on has 512 MB RAM and a very slow SAN hard disk connection (with 1–2 MB/s), so swapping is very slow. With the old PHP versions of the applications, my server would completely hang up when too many JVMs were running because of the lot of swapping to do. This was unacceptable because I also use the server for e-mails. With the new JSP versions, the memory usage of the applications is limited. I have not yet found a way to find out the actual memory consumption of a process under Linux, but the VIRT column currently shows 368 MB and the RES column 186 MB for the servlet container, so the actual usage must be somewhere in between.
This is still high memory usage, but at least it prevents my server from hanging up completely. Unfortunately, due to the excessive usage of the Postgres database to minimise the memory usage, everything takes very very long. Somewhere, there seems to be a limit of 5 minutes after which the web pages stop loading, although the servlet container in the background still keeps calculating. So at the moment, you still cannot analyse large relations or changesets. I am planning to create some asynchronous user interface where you are notified via e-mail or similar once the analysation is complete.
If you have a server running a servlet container and have some resources left where you want to host the two applications, that would make things a lot easier for me. Please contact me in that case.
The routing support of cdauth’s map, my OpenLayers map with additional features (routing beta preview on http://osm.cdauth.eu/map-routing/), has made some progress. I have switched the routing provider from YOURS to CloudMade, which is almost comparable to Google Maps in speed (the only problem seems to be that it cannot route over ferries yet).
I have fixed some bugs considering the drag and drop support. You can move the start and destination markers on the map as well as drag parts of the route to change its course (creating “via” points, remove them again by double-clicking them).
I have tested in the newest versions of Firefox, Opera and Konqueror. Some bugs are yet to be fixed:
* Removing “via” points by double-click does not work in Konqueror
* The different versions of Internet Explorer need to be tested (I suspect that it works in none of them properly.)
* Test in Opera 10.0.
As some of you might have noticed, Route Manager and History Viewer have been down for a while. I had to take them down because with the current implementation -- a PHP page starting a new Java Virtual Machine on each request -- my server would hang up at too many concurrent requests because of too little memory, which is unacceptable because I am expecting some very important mails on that server. I am on a journey right now with very limited Internet access and unfortunately did not quite manage to finish the new version, completely Java-based, before leaving home. I will be back not earlier than beginning or middle of July. Maybe someone manages to run the applications on their server until then? Sorry for the inconvenience and greetings from Greece, Candid
I was a bit upset about the fact that everytime you wanted to write an AJAX application using the OSM API or the Namefinder, you had to set up a local proxy on your server because of the Same Origin Policy.
Thus I have written an AJAX proxy that circumvents the Same Origin Policy by using HTML
I have added experimental routing support to my map. I am using the YOURS API to calculate the routes and try to provide an easy user interface that is similar to Google’s. You can drag and drop the route start and end point and also drag parts of the route to create Via points. It works pretty well (only tested on Firefox so far), but a few bugs still occurr some time.
var layerRouting = new OpenLayers.Layer.cdauth.XML.Routing("Routing");
layerRouting.setFrom(new OpenLayers.LonLat(5.963946, 52.215676));
layerRouting.setTo(new OpenLayers.LonLat(6.1799, 52.2573));
The drag’n’drop will work automatically.
Nominatim has a nice feature to provide an icon URL for search results, for example a shop icon if you search for supermarkets. I use these icons now for the search result markers on my map (http://osm.cdauth.de/map/). Although I don’t know if I will not restore the old marker icons, because these new icons aren’t very prominent on the map.
I have simplified the syntax to add Live Permalinks to an OpenLayers map with my cdauth’s map API. These Permalinks are stored in the part after the # in your browser’s address bar, so when you reload the page, everything will be exactly the same and you can easily send the link to your friends. You can then also use your browser history, so when you hit the “Back” button in your browser, the previous map view will be loaded. Try it out on http://osm.cdauth.de/map/.
To add the functionality to your map, include http://osm.cdauth.de/map/prototypes.js into your web page and replace the constructor “OpenLayers.Map” with “OpenLayers.Map.cdauth”. Then add an OpenLayers.Control.cdauth.URLHashHandler object to your map either by putting it into your controls array or by using the following code:
var hashHandler = OpenLayers.Control.cdauth.URLHashHandler();
This will keep the location hash part in sync with the current map view, so the location will be updated when the map view is changed and the map view is updated when the user changes the location hash part manually (or opens a Permalink someone sent him).
Read more about my API on http://wiki.openstreetmap.org/wiki/Cdauth’s_map/API.
There are 16 labels to be translated, currently an English and a German translation are available. If you have some time left over, help translating it on http://wiki.openstreetmap.org/wiki/OpenStreetBugs/New_Client/Translation.
Just wanted to say that I’ve been back from my journey through Italy for one week now.
OSM Route Manager and OSM History Viewer are working again, they had problems due to a full server disk. Furthermore, the server was down for about two weeks in August.
The cdauth’s map layers are working again, many of them had been broken due to OpenLayers API changes.
During the next weeks, I will upload the GPS traces of my journey (and probably create lots of highway=road as well as camping sites and supermarkets) and continue the work on OSM History Viewer and the new OpenStreetBugs client.
Unfortunately, mapping is kind of hard for me at the moment, as both the displays of my camera and my GPS device don’t work anymore (the latter is actually “broken”…). Fortunately though, I’ve got to know some mappers in my area (as well as have gotten mappers some people I know), so maybe some of us will do one or another tiny mapping party some day.
The first part of OSM History Viewer, the changeset visualisation tool, is ready for testing on http://osm.cdauth.de/history-viewer/ (I haven’t tested it much yet, so it could be really buggy).
During the analysis of the changeset, all ways that were changed are split into segments (connections of two nodes). Then the versions before and after the changeset are checked if they contain the segment. Segments that only exist in the old version are marked red, those that only exist in the new version are marked green, segments that exist in both versions (and thus were not changed) are marked blue. This way, you can see what exactly has changed, if someone only combined or split a way, this will not be marked as a changed (instead, it will be coloured blue).
One problem I am facing right now is that a single node movement can cause a way to change without the way being mentioned in the changeset. There is no way in the current API to find out which ways a node belonged to at a specified point in time. I can only guess this at the moment, maybe I will parse the history view on the OSM homepage in the future, as the way membership seems to be listed there.
A tag change visualisation will also be part of the changeset visualisation in the future, visualising which tags have been added, removed or changed on which way or node.
The second feature coming with OSM History Viewer will be a relation history analysis tool. It will be possible to show the changes on a relation (with all its members and sub-members) in a specified changeset as well as displaying who added which part to the current version of the relation (“blame” view).
In reply to davidearl, who suggested to use the planet diff files instead of the OSM API in my last diary entry about this: As the hard disk space on my web server is very limited, I would not be able to keep many versions of the diff files. A want the changeset visualisation tool to work with all changesets, no matter how long ago they were commited, and that is only possible by using the API.
I am currently developing a web tool to analyse the history of OSM elements visually. There will be two modes available:
1. Visualise changeset: This will highlight any changes made in a changeset on the map.
2. “Blame”: This will visualise who is responsible for removing or adding which part of a relation.
The reason why I need this tool is that recently, lots of errors are being added to cycle route relations I maintain. Sometimes, this happens by mistake (because someone extended a way that was part of the relation for example), sometimes on purpose (someone is not aware that the route runs another way now). In both cases, it is useful to write to the person that did the change to inform him about what he did wrong. But it is not always easy to find out who made a certain change, one main reason seems to be people splitting ways and thus adding certain ways to the relation. The person who has added a way to a relation that should not belong to it may not always be the person who did this mistake, it might be someone who split a way that errorneously was part of the relation.
Thus my History Analyser will work on per-node basis: if the nodes of a way added to a relation were previously part of another way belonging to the relation, this will not be considered to be a change. Accordingly, a way whose tags but not whose nodes were changed will not be visualised as having changed. Only the part of a way whose nodes have changed will be considered to have changed. Sub-relation support will also be implemented for better route analysis.
The visualisation of a way or a relation could take quite a while, as many requests are needed to find out what has changed. To get the changes of each node, you need to get the previous version of that node. As you can only get one item per request, it could take several minutes up to several hours to find out the exact changes (or maybe the API is faster than I think when getting the history of objects?). I’m not sure yet how I will deal with that.
The tool is going to be available in a few weeks.
Auf einer Wanderung heute ist mir aufgefallen, dass hier am bayerischen Alpenrand abgesehen von vielen Wanderwegen auch noch einige Ortsbezeichnungen fehlen, und zwar einige Gipfel (natural=peak), ein paar Weiler (place=hamlet) und einige Almen oder andere benannten Plätze (place=locality). Die Einheimischen kennen und verwenden die Namen dieser Orte, als Nicht-Einheimischer ist es aber schwer, sie herauszufinden, da sie nirgends aufgeschrieben sind.