NorthCrab's Diary

OpenStreetMap NextGen Takes Shape! (screenshots)

Posted by NorthCrab on 18 February 2024 in English.

I am really excited to finally share some tangible progress on OpenStreetMap NextGen! While I have been working on this project almost day-by-day for the past 3 months, just recently I reached the point of launching the website (for now locally) and begun development of its frontend features.

OpenStreetMap-NG has grown immensely in scope since the original announcement. Now the project not only focuses on core performance, usability, and accessibility but also on improving the user-facing interface and experience. I want this project to truly stand by its name, providing NextGen value to all aspects of the website. And so far, it’s on the right track.

So far, the number of improvements is counted in hundreds, including but not limited to various bug-fixes, performance optimizations, privacy, and security enhancements. And the list keeps on growing!

My current primary focus is to wrap up frontend baseline development and prepare an amazing developer documentation so other people can hop in and help. I still need a little more time to prepare the grounds as I want to provide a good contributor experience and ensure everyone’s time is used effectively.

The project is currently funded through individual donations. You can help it grow by donating here, or by leaving a star on the GitHub project itself, or both :-)

An orange crab holding a glowing star, space ship flying in the background

Please note that this project is not affiliated with the OpenStreetMap Foundation. It’s the result of my voluntary work and personal choices. The presented side-by-side comparisons are subject to change as the development is ongoing.

Discussion

Comment from mmd on 18 February 2024 at 13:44

I tried to run the code locally, and took a few screenshots like this one:

It’s still a bit bumpy to get this up and running w/o any sort of documentation, but hey, that was kind of expected at this point in time. Nix Shell was a huge pain on my system with issues due to incompatible GLIBC versions and random /nix libraries being injected to LD_LIBRARY_PATH that made some programs fails, while others weren’t working without it.

Obviously, I was more interested to see a bit of API stuff in action. For some reason http://localhost:3000/api/0.6/node/1 would return this nice error message. This looks like a topic for another time.

{"detail":[{"type":"is_instance_of","loc":["path","type"],"msg":"Input should be an instance of StrEnum","input":"node","ctx":{"class":"StrEnum"},"url":"https://errors.pydantic.dev/2.6/v/is_instance_of"}]}

Comment from NorthCrab on 18 February 2024 at 13:52

Hey! I am currently focusing on frontend development, and I have not yet polished the points you mentioned. Rest assured, they will be fixed for the dev server launch. As mentioned, in the current state, the application is not yet ready for new contributors :-)

Comment from NorthCrab on 18 February 2024 at 16:21

I have gone ahead and investigated the issues today 🙂. I removed the global LD_LIBRARY_PATH override, which is now only applicable to the Python process and nothing else. This should get rid of any potential conflicts you have experienced (you may need to reset your env: rm -r .venv). Regarding the StrEnum issue, this is most likely because FastAPI expects “str, Enum” type instead of the one that is currently used. I will fix it whenever I work on that portion of the code. Nonetheless, I would recommend waiting for the official documentation to avoid unforeseen issues and unnecessary stress — for both of us 😃.

Comment from GeeMaps! on 18 February 2024 at 22:19

For your shown example directions, could you please possibly have them from and to the same places, which would let us see differences better?

Comment from Matija Nalis on 19 February 2024 at 00:09

While I agree that improving usability should be one of the goals down te road, I’d highly suggest that you initially focus only on replicating existing functionality with same UI and not “also on improving the user-facing interface and experience”. The reasons being:

it will be much harder for people to compare if new code is working equivalently to old code if they display UI differently, which will lead to more doubt and less acceptance
not everyone will agree that new UI is better (for a lots of reasons: some prefer compactness to visual fluff, some just have mechanical memory of using old UI, some might disagree on what looks better etc.) - so trying to force modified UI in addition to changed codebase will inevitably result in less buy-in: so people will vote not only on technical merits (speed improvements, improved code readability and maintainability, allowing much wider developer base) but on their UI preference, and that will always be less than 100% on any change, no matter how good it is.

Once the new code (which looks and works just like the old code) is accepted to be new mainstream; then you should start on changing UI to be more usable.

Comment from Friendly_Ghost on 19 February 2024 at 03:33

I welcome NorthCrab’s progressive take on UI development of osm.org. We don’t have to agree with every design decision right now, because that can all be tweaked later. And with new development one should expect some front end enhancements; it would be a shame if after all the work we still end up with a website that looks like a time machine to a distant past. So keep it up, NorthCrab!

Comment from PCarew_EG on 19 February 2024 at 18:18

Great work, keep it up! The front-end enhancements look great, I’d expect to see them given the work you’re undertaking.

Comment from NorthCrab on 20 February 2024 at 15:05

“We don’t have to agree with every design decision right now, because that can all be tweaked later.” Exactly this! I am trying to optimize my time by implementing the ideas that make the most sense to me from the get-go - as a Proof of Concept for later discussion, rather than coding the website one-to-one and then working on improvements. My goal for this project is to: do first, talk second. 🦀

Comment from PlayzinhoAgro on 20 February 2024 at 18:24

I don’t believe that users will complain about some design changes on the site, many want changes, we can’t have a site that goes against any UX conversion. Thoughts like these are a delay for our community but any drastic change to the site has to be based on user research and not just endless discussions on forums.

Comment from PlayzinhoAgro on 20 February 2024 at 18:29

your project is amazing, finally someone making changes happen

Comment from Matija Nalis on 20 February 2024 at 19:29

Well, I hope you guys are right. I still think UI changes at this point (instead of doing them several months down the road) are unnecessary risk taking, as the project needs to get buy-in not just from progressives but also from conservatives too (and they might dislike changing their habits), and there is no guarantee that this new code will actually get used by OSM (once feature parity is accomplished and code published; that will be hugest decision in OSM since OdBL-license schism since its inception, and opinions on that already seem polarized).

I’m in support of python version, if baby steps of walking before running reduce the changes of rejection, I’d consider them.

To me, the main points are:

is the new python code easier to maintain then older ruby on rails? That is IMHO the main breaking point, and it would depend mostly on OSM Ops team decision, and pool of new wannabe-ops-volunteers that it might produce (one of OSM main bottlenecks).
does the new design allow more features to be more easily implemented, and are more people willing to volunteer their time more to do so? This is what most of the community is rooting for, and why this project is a big deal (obviously the language choice, but also to underlying design choices. And adrenaline rush of new developer willing to add new stuff is big in itself!)
can it be run / tested in parallel with current production code (this is huge issue, as Ops want the code to have proven reliability before even considering to look at it. It is also a reason why keeping with same PostgreSQL schema allowed this project to proceed - if NorthCrab were to insist on original noSQL ideas, that would IMHO be insta-killer for the project adoption)
is the the new code faster / uses less resources? Obviously, this is easily shown by benchmarks, and more objective than most, so everybody will like that (provided it delivers on those promises)
UI design changes - meh. You could do such UI design changes on current backend too. It is non-issue why someone would walk this project (instead of just much more simple PR doing proposed HTML/CSS UI changes to current Ruby code). But as any design change, it can be disliked by some, which would prolong (or even reject in extreme cases) time for adoption of whole project. And this high-risk / low-gain is the issue I’ve tried to warn about. (although I might be wrong about my estimates, of course – I’d actually love if they were wrong)

Comment from YjM on 20 February 2024 at 20:24

Awesome work! The presented UI improvements are only nuances to the status quo, I can’t see any drastic changes. Please, don’t be shy to touch it and keep up the great work!

Comment from mmd on 20 February 2024 at 22:11

can it be run / tested in parallel with current production code (this is huge issue, as Ops want the code to have proven reliability before even considering to look at it. It is also a reason why keeping with same PostgreSQL schema allowed this project to proceed - if NorthCrab were to insist on original noSQL ideas, that would IMHO be insta-killer for the project adoption)

I’d say at this point in time, it’s basically not possible to run the Python code against the same production APIDB instance, like we do for CGImap, or osmdbt for minutely diffs, or the weekly planet dump.

To be fair, as far as i can tell, the idea was always to migrate the current db to the new schema, and run it on a separate database instance. I don’t recall any post or announcement, which claimed to be compatible with the existing schema.

So what has changed?

The 16 database tables currently used for nodes, ways and relations have been replaced by a single “element” table in which each single object version is stored. So one table for everything.

As a result, the code is written in such a way that some concepts are now more generic (e.g. tags behave in the same way regardless of their object type). It remains to be seen how this will affect the performance of our 11 TB database.

Way nodes have been replaced by the member concept already known from relations. That means, a way looks a bit like a relation where every member is simply a node. I recall seeing such ideas on the API 0.7 proposal page, where someone suggested that “everything is a relation”. It’s probably the first time, this idea has made it in any sort of real implementation.

Tags are stored in JSONB fields, and for node coordinates a new dependency on Postgis has been introduced. As far as I can tell, this is used for nodes only, to enable polygon based node queries.

There are many more changes in place which I will skip for now, since I cannot comment on them without some more in depth testing.

Reference: https://github.com/Zaczero/openstreetmap-ng/blob/main/app/models/db/element.py#L23-L34

Comment from 快乐的老鼠宝宝 on 25 February 2024 at 12:20

First of all, thank you for the current work, it looks really good. It is obvious that you have put more effort into the interface design to make it user-friendly!

I am both a conservative and a progressive person, so I welcome it very much, but I have some questions:

And: 1. Python is generally considered to be weak in performance. Have you conducted stress tests under heavy traffic so far?

The 16 database tables currently used for nodes, ways and relations have been replaced by a single “element” table in which each single object version is stored. So one table for everything.

Especially when mmd pointed out this issue above, blurring the type of element will inevitably bring additional computational overhead to each query.

When can we expect to see a read-only (not necessarily real-time sync) demo site? This would help other users and developers test their software’s compatibility with OSMNG.
(About a minor UI change) It seems that the feature allowing users to customize scale and short link when sharing has not been implemented?

NorthCrab's Diary

OpenStreetMap NextGen Takes Shape! (screenshots)

Discussion

Log in to leave a comment