OpenStreetMap
Better way of verifying the connectedness of relations?
Posted by rayKiddy on 12 July 2018 in English.I have an application which I am developing. I am not sure it is of general interest, yet. I track some relations in it. I am interested in cities and counties in California and I am recording their ids and other information about them.
I just implemented code (in java) which reads the relations, and then recursively reads the ways and nodes of it and verifies that the lines therefrom are connected. The obvious implementation is taking a rather long time to do the job. I am hoping there is a better way.
Any suggestions?
I will post more about the application when I have it running on my colo.
thanx - ray
Discussion
Comment from amapanda ᚛ᚐᚋᚐᚅᚇᚐ᚜ 🏳️🌈 on 13 July 2018 at 08:46
I’m not really sure what you’re doing exactly. But from what you’ve described I can’t see anything obviously wrong or slow. So I don’t think there’s a faster way. I’m surprised you say it’s slow, how are you storing/processing/fetching the OSM data? Maybe it could be speeded up there?
Comment from ToniE on 13 July 2018 at 13:03
I have similar code in Perl which checks ways in Public Transport relations (PTv2).
Take the given sequence of the ways in the relation and check
whether a way is connected to its the sucessor or not (count number of gaps),
whether the vehicle (bus, tram, train, …) uses a oneway road in wrong direction w/o oneway:psv=no, oneway:bus=no, …
whether a roundabout is included in the list completely but the vehicle uses only segments,
whether the vehicle is allowed to use the road (no for: hw=construction, =footway, … w/o psv=yes, …)
and so on
The code became pretty complicated and long and maintaining that part of the overall code is hard.
Comment from rayKiddy on 13 July 2018 at 18:14
@rorym The most obvious thing that has occurred to me is that I can split the fetches I am doing into separate threads so that there can be some parallelization. I am fetching the data in calls similar to “curl ‘https://www.openstreetmap.org/api/0.6/relation/3529698’”, then finding the nodes and ways in this, collecting them into a list and then doing the same recursively. At the end, I see if every way endpoint matches up with the end of another way.
It would be most wondeful if something like https://www.openstreetmap.org/api/0.6/way/262738820 would return the nodes with their latitude and longitude included. When I just get a list 100 node ids for a way, one has to do 101 fetches and xml parses. Ick.
@ToniE It sounds as though what you are doing is quite a bit more complicated than what I am trying to do. Do you have your code accessible, such as via github. I would be interested in seeing what you doing. Perhaps some sub-part of that code would suggest something to me.
Comment from ToniE on 13 July 2018 at 18:55
Hi rayKiddy,
yep, the code ist in github
https://github.com/osm-ToniE/analyze-routes
The relevant code starts at line 2850 of
https://github.com/osm-ToniE/analyze-routes/blob/master/bin/analyze-routes.pl
It does what it has to do, after many patches and adding more and more if/else. But I’m afraid 50% of the code is redundant (never change a running system …). A re-design is on the roadmap though.
The tool currently analyzes 27 Public Transport networks in Germany, mostly Bavaria …
An overview (in German) is given here
https://wiki.openstreetmap.org/wiki/User:ToniE/analyze-routes#Analysierte_Verbunde
Examples of the output can be found in the column ‘Name’.
Br Toni
Comment from amapanda ᚛ᚐᚋᚐᚅᚇᚐ᚜ 🏳️🌈 on 13 July 2018 at 21:46
There is a /full API end point for nodes/ways/relations which, I think, will return all the members as well. I think you can also get many objects at once in one API call. That might help. (I haven’t done a lot with the API, so I’m not sure).
Another approach is to download the OSM data and just use that rather than querying the API for every object. You can download a regional extract from Geofabrik, or the use osmium to cut out a part of the planet file (or any other extract). You need to figure out how to store the file. But it’s the same XML format. It would be much faster than an API call for every element. When people do a lot of work on OSM objects this is probably more common than lots of API calls
Comment from mmd on 14 July 2018 at 05:20
i don’t recommend using the Osm api for your project, as it would violate the usage policy: https://operations.osmfoundation.org/policies/api/
Comment from ikonor on 14 July 2018 at 08:57
An alternative for administrative boundaries might be downloading already compiled geometries from the Boundaries Map (not sure if incomplete relations are included).
For ongoing monitoring there is the daily Missing Boundaries report (maybe scraping and filtering for relations of interest).
Comment from ToniE on 14 July 2018 at 11:45
I use OVERPASS API instead of OSM API. This API is designed for those queries.
I download all relevant data in one go into an XML file and let the analysis code parse that file.
Example for querying all public transport data for busses in the the “county” of “Landshut” (all counties worl-wide with that name)
http://overpass-api.de/api/interpreter?data=area[boundary=administrative][admin_level=6][name~’Landshut’];(rel(area)[route~’(bus)’];rel(br);reltype=’route’;)->.routes;(.routes;rel(r.routes);way(r.routes);node(r.routes););out;
https://overpass-turbo.eu/
is a good playground.
Be sure to use the ID of nodes instead of their LAT/LON to check the connectedness of ways.
Comment from rayKiddy on 14 July 2018 at 22:41
@mmd How would it violate the policy? My app would be accessible under the same license as OSM. Or are you referring to something else? Which part of the policy?
Comment from amapanda ᚛ᚐᚋᚐᚅᚇᚐ᚜ 🏳️🌈 on 15 July 2018 at 07:28
If you make loads and loads of API calls and put the sever under undue load, then your app would be banned and people who are maintaining the servers for free in their spare time would be mad at you.
Comment from mmd on 15 July 2018 at 07:31
It’s not a license thing, for sure. As I said, it’s about the usage policy, and the purpose for which you use the API. The policy states: The editing API is provided in order to edit the map data, not for read-only purposes or projects.
From your description it doesn’t sound like you’re editing the map, but it’s rather some pulling OSM data and doing some local analysis thing, which is not permitted according to this policy.
Comment from rayKiddy on 15 July 2018 at 21:22
Well, good thing that I am trying to do this to find relations that need to be edited, for the benefit of the community.
I am trying to do something helpful, something that does not exist now. Is that ok with you guys?
Comment from kucai on 16 July 2018 at 03:36
Piggybacking on this… so how do you include roundabouts in relations? Do you break them up into sections or do you just bypass them and just use the next road segment?
Comment from ToniE on 16 July 2018 at 07:41
@rayKiddy: well, that’s what I also do but I nevertheless use the overpass api instead of OSM api. OSM API is for Editors, … not for read-only QA tools.
@kucai: 1. I usually break them up into sections, including only those which are actually used 2. others include the entire roundabout (if not already split into sections) - also valid 3. do not bypass them, include them
JOSM’s validator allows both 1. and 2. and complains for 3. Same applies for my tool.
Comment from rayKiddy on 16 July 2018 at 18:43
@ToniE Ah! I did not see that distinction. Thanx.
Comment from rayKiddy on 18 July 2018 at 17:07
I am looking at the overpass API and it appears tobe not very difficult to get the information on a relation.
[out:json]; rel(5173237); (._;>;); out body;
That second is … very obvious, no? :- )
I think I still have to read just the top relation node from OSM API because overpass does not include the last edit time for the relation, and this seems important for checking the relation.
I plan on not fetching all the other data if a relation has not changed since the last check.
Comment from mmd on 18 July 2018 at 18:02
There’s really no need for OSM API for your use case. All you have to do is to use out meta; instead of out body; and you should be all set.
[out:json]; rel(5173237); (._;>;); out meta;
If you don’t need to resolve relation members, try rel(5173237);out meta;
Please take a look at the docs for more details.
Comment from rayKiddy on 18 July 2018 at 19:40
@mmd thanx for the info. I have read the docs and I will re-read to try to understand them better. It’s not the most intuitive API I have ever seen. But I know there are reasons for this.