OpenStreetMap

mmd's Diary Comments

Diary Comments added by mmd

Post When Comment
OpenStreetMap NextGen Takes Shape! (screenshots)

can it be run / tested in parallel with current production code (this is huge issue, as Ops want the code to have proven reliability before even considering to look at it. It is also a reason why keeping with same PostgreSQL schema allowed this project to proceed - if NorthCrab were to insist on original noSQL ideas, that would IMHO be insta-killer for the project adoption)

I’d say at this point in time, it’s basically not possible to run the Python code against the same production APIDB instance, like we do for CGImap, or osmdbt for minutely diffs, or the weekly planet dump.

To be fair, as far as i can tell, the idea was always to migrate the current db to the new schema, and run it on a separate database instance. I don’t recall any post or announcement, which claimed to be compatible with the existing schema.

So what has changed?

The 16 database tables currently used for nodes, ways and relations have been replaced by a single “element” table in which each single object version is stored. So one table for everything.

As a result, the code is written in such a way that some concepts are now more generic (e.g. tags behave in the same way regardless of their object type). It remains to be seen how this will affect the performance of our 11 TB database.

Way nodes have been replaced by the member concept already known from relations. That means, a way looks a bit like a relation where every member is simply a node. I recall seeing such ideas on the API 0.7 proposal page, where someone suggested that “everything is a relation”. It’s probably the first time, this idea has made it in any sort of real implementation.

Tags are stored in JSONB fields, and for node coordinates a new dependency on Postgis has been introduced. As far as I can tell, this is used for nodes only, to enable polygon based node queries.

There are many more changes in place which I will skip for now, since I cannot comment on them without some more in depth testing.

Reference: https://github.com/Zaczero/openstreetmap-ng/blob/main/app/models/db/element.py#L23-L34

OpenStreetMap NextGen Takes Shape! (screenshots)

I tried to run the code locally, and took a few screenshots like this one:

image

It’s still a bit bumpy to get this up and running w/o any sort of documentation, but hey, that was kind of expected at this point in time. Nix Shell was a huge pain on my system with issues due to incompatible GLIBC versions and random /nix libraries being injected to LD_LIBRARY_PATH that made some programs fails, while others weren’t working without it.

Obviously, I was more interested to see a bit of API stuff in action. For some reason http://localhost:3000/api/0.6/node/1 would return this nice error message. This looks like a topic for another time.

{"detail":[{"type":"is_instance_of","loc":["path","type"],"msg":"Input should be an instance of StrEnum","input":"node","ctx":{"class":"StrEnum"},"url":"https://errors.pydantic.dev/2.6/v/is_instance_of"}]}
Small Towns in Europe

That took about 3 days 😅

I believe one reason for this long runtime could be your pre-filtering on certain areas (not exactly sure if this how you’ve done it). When using https://overpass-turbo.eu/s/1GQm to analyze all place nodes on a global scale, the runtime should be less than 1.5 hours. It returns about 401k place nodes that would need some additional filtering by location as a post-processing step.

If you like to try this out, I have uploaded the query result here: https://dev.overpass-api.de/misc/test_buildings.xml.gz (file size: 21M)

A minute of facts about the duration of changesets

5 years ago we’ve already discussed to add an optional “close_changeset=true” attribute to the osmChange header. This would, as the name says, close the changeset as part of the upload, without the need to send an additional changeset close message. Unlike the proposed API 0.7 changes, it wouldn’t introduce an incompatible change, since it’s an optional attribute only.

Link: https://github.com/openstreetmap/openstreetmap-website/issues/2201

OAuthtung!

Yes, that’s just a normal OAuth2 Bearer Token, which doesn’t expire, like all other OAuth 2 tokens at the moment. The actual generation happens here: https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/user.rb#L379-L387

Doorkeeper…find_or_create_for is the relevant bit here to trigger the generation on the backend for a given application/user/list of scopes (assuming the token hasn’t been created yet, otherwise the existing token is retrieved).

OAuthtung!

FWIW: The “OpenStreetMap Web Site” OAuth2 application is also officially documented here: https://github.com/openstreetmap/openstreetmap-website/blob/master/CONFIGURE.md#oauth-consumer-keys -> To allow Notes and changeset discussions to work, follow a similar process, this time registering an OAuth 2 application for the web site […] Check boxes for the following Permissions ‘Modify the map’ and ‘Modify notes’.

OAuthtung!

It’s kind of funny that you’ve went the extra mile and blurred the OAuth token on the webpage, then pasting it in plain text and clearly visible in your terminal window. I hope you’ve revoked that token in the meantime ;)

OpenStreetMap Service Availability (2023-12-20 - 2024-01-20)

Yearly database re-indexing was running on the weekend of 01-14, with periods of fairly high load on the database server: https://prometheus.openstreetmap.org/d/Ea3IUVtMz/host-overview?orgId=1&var-instance=snap-01&from=1705130975266&to=1705276768520

This might have impacted some queries to take longer than usual, or even time out.

By the way, the CGImap link points to an outdated mirror. It should be https://github.com/zerebubuth/openstreetmap-cgimap instead.

Future deprecation of HTTP Basic Auth and OAuth 1.0a

So yes, you can do all this in a few lines of shell script, by using only curl and jq, without any external libs, local HTTP server, or anything: https://gist.github.com/mmd-osm/b61956bb4b92e9b37488189379b380c9

Before trying this out, be sure to sign up on the dev instance https://master.apis.dev.openstreetmap.org (you already knew this).

Bonus points for storing the access token in a local file, so you don’t need to go through the osm.org authorization each time you’re running the script. I was too lazy to implement that.

If you’re also too lazy like me, you can also use the access token, and treat it as some kind of Personal Access Token. Line 14 shows you how to use the access token to call an API endpoint.

Disclaimer: This is only meant for personal scripts and local testing. Also, please register you own app and replace client id and client secret with your own values. Use urn:ietf:wg:oauth:2.0:oob as redirect URL.

OpenStreetMap Service Availability (2023-11-20 - 2023-12-20)

12-03 issue was handled in this operations ticket: https://github.com/openstreetmap/operations/issues/1008

IIRC a well known company did some fairly extensive web page and API scraping over the weekend and was blocked subsequently. (Mentioned in another operations issue)

Leveraging PostGIS to Write Your FlatGeobuf Files

Spaces in code blocks work just fine, see https://kramdown.gettalong.org/syntax.html#code-blocks

Example:

def what?
  42
end
Multiple user accounts in JOSM

Switching accounts inside JOSM isn’t exactly a new idea: https://josm.openstreetmap.de/ticket/2710

Multiple user accounts in JOSM

Why don’t you use josm.home? Seems much easier to me to manage multiple profiles/users… https://community.openstreetmap.org/t/mehrere-josm-profile/74386

Future deprecation of HTTP Basic Auth and OAuth 1.0a

My target timeline for the C++ part of the API is no later than Q1/2024, see https://github.com/zerebubuth/openstreetmap-cgimap/issues/286

I don’t know what OWG will eventually come up with. Also sysadmins have the final say on what to deploy at which point in time.

How to Build a Personal Overpass Server on a Tiny Budget

The main purpose of a dispatcher process is to coordinate read and write activities, to make sure that the writing processes (update_from_dir and update_database) don’t overwrite parts of the db files, which are currently being read by other process (such as interpreter or osm3s_query).

Besides witting lots of (mostly very technical) detail information to the log files, the dispatcher doesn’t cause any I/O activity. I have removed the excessive logging in my fork, and it doesn’t really add any value for me.

Also, I wouldn’t simply kill the dispatcher processes, and use the ‘–terminate’ command line option instead.

Regarding downloading clone files, we currently have disk space for around 1-2 days of clone files available on the server. If your download speed is too low, db files might have already been purged from the server before your download has finished! This situation might improve a bit in the next couple of months, though.

Posting your Overpass related questions on “community.osm.org”, and tag them with “overpass” would also work, if you’re not on Slack.

A new home to OSMCha

Maybe that’s a good opportunity to work on the underlying infrastructure as well. Turning spinning wheels into faster responses.

Relation 12907666

I find JOSM’s validator invaluable in such cases - you absolutely ought to be using that to check for problems as you go (even if you’re not using JOSM to edit).

Totally agree here. JOSM reported even some more issues: three relation members should have a member role “inner” instead of “outer”.

Relation 12907666

By the way, after splitting way 1000494463, you should see one more member in https://www.openstreetmap.org/relation/15074887 … overall you shouldn’t have to delete any nodes or ways. Splitting up one way and making sure that the right segments are assigned to each of the two relations should do.

Relation 12907666

To be more specific, JOSM complains about way https://www.openstreetmap.org/way/1000494463, in particular the part of the way starting out southbound with node https://www.openstreetmap.org/node/9235211881 … once I remove that node, and everything further south of it, the closure issue is solved. Once option to achieve this is to split way 1000494463 at https://www.openstreetmap.org/node/9235211880

Relation 12907666

I think there are more issues, like a non-closed geometry up here: https://www.openstreetmap.org/relation/12907666#map=16/43.8049/-122.6301 … You need to re-run the validator, once you have fixed all reported issue, as new issues are likely to show up then.