OpenStreetMap logo OpenStreetMap

NorthCrab's Diary

Recent diary entries

Welcome to the fourteenth OpenStreetMap NextGen development diary.

🔖 You can read other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source — join us today:
https://github.com/openstreetmap-ng/openstreetmap-ng

GitHub Stars

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

In case you missed it…

Recently, we showcased Search 2.0: a comprehensive overhaul of the search functionality. For the first time, OSM search is now simple and intuitive to use. It’s a must-see (there’s a video demo too)!

OpenStreetMap-NG Repository Moved

This week we have moved the openstreetmap-ng code repository from Zaczero/openstreetmap-ng to openstreetmap-ng/openstreetmap-ng. All stars, forks, and issues have been transferred automatically. All previous URLs are also redirecting to the new address. If you have previously interacted with the project, there’s nothing you need to do.

Why? First of all, since this is a community project, it doesn’t make sense to keep it hosted under an individual account. The project should be hosted under an organizational account where more than one person can manage it. Secondly, by having a project hosted under an organization, we can better manage contributors’ permissions. And lastly, contributors now have the option to display a project badge on their personal GitHub profiles. That’s one more way for us to say thank you!

The organization is free to join to everyone. Check out this wiki page on how to do so.

Finished Directions Implementation

We have also finished the directions page implementation. When searching for a place in the directions, you will now use our improved search engine backend. We have also fixed various small bugs that we experienced on the current website, and made the map segments hoverable and clickable. For comparison, in the current Ruby implementation, only the sidebar handles events. This change makes the directions page more convenient and consistent to use.

Codebase Improvements

Lots of this week’s development time went into improving the codebase. We have now integrated mypy into our workflow, which can detect a wide range of type-related bugs, providing a more stable and predictable development. This integration required more than 300 code parts to be changed.

Additionally, we have dropped the anyio dependency in favor of Python’s standard asyncio. This is one less dependency that new contributors will have to learn. Asyncio provides better performance than anyio (which is an abstraction layer around asyncio) and is more compatible with the Python typing system—especially important now that we use mypy.

Migrated History Feeds

Last but not least, we have finished the migration of the /history/feed and /user/name/history/feed Atom feeds. This brings us one step closer to feature parity with the Ruby implementation.

While working on this migration, we spotted and corrected a 12-year-old bug. Simply, the history feeds advertised their contents as being licensed under CC-BY-SA 2.0, which was changed to ODbL in 2012. The license now points to the OpenStreetMap copyright page, which is a more informative and accurate resource.

Sponsors

This week’s work was sponsored by 17 people!
8 donors on Liberapay, and 9 on GitHub Sponsors.

We can do this and we will do this. Thank you for supporting this project!

Please consider supporting the OpenStreetMap-NG development with any amount. As a thank you, you will be eligible for the unique OpenStreetMap-NG Founder profile badge. You will also help to push the project forward 🏋.

Welcome to the thirteenth OpenStreetMap NextGen development diary.

🔖 You can read other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source — join us today!
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

In case you missed it…

Last week, we showcased Search 2.0: a comprehensive overhaul of the search functionality. For the first time, OSM search is now simple and intuitive to use. It’s a must-see (there’s a video too)!

Default Editor Switching

This week, OpenStreetMap-NG introduced a convenient way to change your default editor. By exposing this functionality, previously hidden in settings, we prevent editor lock-in and allow users to easily experiment with other editors. A small icon now indicates which editor is currently set as default.

Faster and Slimmer GPX Traces

We completed a major overhaul of the GPX trace database design. Here are the results:

  • Storing traces is now much more efficient, with an 80% reduction in data size. What previously required 1TB of storage now only needs less than 200GB.
  • Uploading and downloading traces is noticeably faster.
  • There is no performance impact on querying individual tracepoints.

While further storage reduction is possible, it would slightly degrade the performance of querying individual tracepoints. Given the substantial 80% reduction already achieved, we decided not to pursue further optimization at this time.

Refreshed Data Alerts UI

We recently migrated and refreshed the map data alerts. Below are side-by-side comparisons with brief commentary.

When loading a large amount of data, users are now presented with a much simpler message and a clear decision: show the data or hide it? The previous message included unnecessary details and a small grammatical mistake (“Are sure you”). The (X) button was also unintuitive, making its function unclear.

When deciding to load even more data, the server may silently abort the request. Previously, users were left without any indication of what was happening. With OpenStreetMap-NG, users now see a new alert with instructions on what to do next.

Let’s wrap up this week’s highlights with something visually pleasing. We now have a new logo, which will be used for the project’s and soon the organization’s branding.

And here’s how it looks in the project’s README:

Sponsors

This week’s work was sponsored by 15 people!
8 donors on Liberapay, and 7 on GitHub Sponsors.

Your support makes all the difference! 🔥

Please consider supporting the OpenStreetMap-NG development with any amount. As a thank you, you will be eligible for the unique OpenStreetMap-NG Founder profile badge. You will also help to push the project forward 🏋.

Welcome to the twelfth OpenStreetMap NextGen development diary.

This week brought some exciting updates and major news!

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

In case you missed it…

We are now welcoming new contributors! To show our appreciation, we offer contributor incentives. If you’re familiar with Python, join us – it’s easy to get started!

Continue reading

Search 2.0 (with video showcase) 🎉

This week, we made significant strides towards our NextGen vision. The new search functionality is nearly complete, and I’ve recorded a short video comparing OSM Ruby and OSM-NG side-by-side.

⬇ Click below to play ⬇

Video thumbnail

or click here: https://peertube.monicz.dev/w/c7Qwif6S4MpvuBBDu7x8LM

The search now toggles between global and local modes. By analyzing Nominatim responses, it determines the most relevant results for the user. We’ve also introduced a refreshed interface and new functionality.

Welcoming 4 New Contributors

A warm welcome to our four new contributors! Your valuable feedback has already led to notable improvements. It’s great to have you on board – let’s keep building! 👋

Trace Visualizations on Map

GPS traces are now rendered on an actual map, providing a useful point of reference and enhancing the user experience.

Fixed White Grid Lines

We’ve fixed a major issue: the white grid lines bug. This bug has been a significant problem for many users, affecting the overall OpenStreetMap experience by making it feel unfinished and unprofessional. This week, we found and implemented a solution that completely resolves this issue. Better now than never!

And more…

My development diaries highlight only a fraction of our work. For full transparency, you can always track our progress on the project’s GitHub repository. Again, thank you to the entire team for your dedication and hard work!

Sponsors

This week’s work was sponsored by 15 people!
8 donors on Liberapay, and 7 on GitHub Sponsors.

Thank you for your support! You are the foundation of our work. 💪

Please consider supporting the OpenStreetMap-NG development with any amount. As a thank you, you will be eligible for the unique OpenStreetMap-NG Founder profile badge. You will also help to push the project forward 🏋.

Donate using Liberapay

Welcome to the eleventh OpenStreetMap NextGen development diary.

This week has been focused on the design and implementation of new user profile pages. Today’s profile pages are boring and quite limited — let’s fix that! 🧑

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

Last Week’s Recap

Last week the project was opened for new contributors. The codebase is now stable enough to allow for greater cooperation. There are also contributor incentives to ensure your work is always appreciated.

Continue reading

New Profiles

» View in fullscreen

In summary:

  • Profile pages feel more friendly and personal
  • Fresh accounts have a new user badge
  • Profiles don’t look odd when no user description is provided
  • There are previews of recent contributions, allowing quicker navigation
  • Last activity also showcases notes, traces, and diaries (vs. only changesets)
  • Changesets and notes comments are now possible to filter
  • Added last year’s activity summary
  • Added groups component
  • Friends have been renamed to follows and help text was added

Design Process

When working on new designs, I prefer to draw them on paper first. It allows for quick iterations and greater flexibility. Only then do I transfer the design from paper into code. This technique, despite being a backend developer, allows me to create nice-looking and intuitive sites.

Development Experience Improvements

This week we further improved our developer documentation. There is now dedicated page on testing the software, and the database preloading instructions now include information on how much disk space is required (requested in #33).

We have also fixed a bug which caused the JS/SCSS/locale recompilation to not work on some systems. Resolved by #37 and #39.

Sponsors

This week’s work has been sponsored by 15 people!
8 donors on Liberapay, and 7 on GitHub Sponsors.

Thank you for your trust and continuous support! 🐈

Please consider supporting the OpenStreetMap-NG development with any amount. As a thank you, you will be eligible for the unique OpenStreetMap-NG Founder profile badge. You will also help to push the project forward 🏋.

Donate using Liberapay

Welcome to the tenth OpenStreetMap NextGen development diary.

This week the project becomes open for new contributors. Guides are ready, base code is stable, and more is on its way. Let’s get started 😎.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

Developers Discord

If you are a potential contributor, we have launched a Discord server for NextGen developers. There, we will provide you with quick, personalized support and guidance. We will also use it as the primary internal communication channel for the project.

Getting started can be difficult, but we are here to help.

Discord Developers Chat

Is This for Me?

We support Linux, macOS, and Windows (WSL2) operating systems. To contribute to the frontend, you will need to know HTML, CSS, and JavaScript (without any frameworks). To contribute to the backend, you will need to know Python. We strive for simplicity and accessibility.

In the mid-term future, we will also seek help from translators, graphic designers, and software testers. Stay connected if that’s your area of expertise!

Contributor Incentives: We Pay You

We provide incentives for contributors who help with the project. We will even pay you for your time and effort. Click here, to learn more.

Getting Started Contributing

The recommended first step is to follow the Contributing Guide. It will guide you through the process of setting up your development environment and getting started with the project. There are also many other useful wiki pages! If you are looking for ideas to work on, check out the Issues page.

Introductory Meetings

We have planned four introductory meeting this week, with different hours and dates to accommodate your schedule. Interested, or just want to say hi? Join the events on Discord.

Events Discord Screenshot

Something is Broken?

Surely there are some things that we have missed. Please report any development problems on Discord or by opening an issue on GitHub. We will resolve them quickly!

Short-Term Goals

The project is still heavily focused on reaching feature parity with the original OpenStreetMap website. After that, we will provide a publicly available testing server.

Sponsors 🏅

This week’s work has been sponsored by 14 people!
Four private and four public donors on Liberapay, and six public on GitHub Sponsors.

T-H-A-N-K Y-O-U

Please consider supporting the OpenStreetMap-NG development with any amount. As a thank you, you will be eligible for the unique OpenStreetMap-NG Founder profile badge. You will also help to push the project forward 🏋.

Donate using Liberapay

Welcome to the ninth OpenStreetMap NextGen development diary.

This week was focused on providing final touches to the core system. Within the few days I’ll publish details on how to get started contributing! Keep an eye for a dedicated diary entry 🙂/.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

Let’s jump into this week’s highlights.

macOS and ARM support

OpenStreetMap-NG development environment has received MacOS support. Meaning that it can now be developed on Linux, MacOS, and Windows through WSL2. The installation instructions are the same for all platforms and contain just a single step: install and run nix.

Changesets History Refresh

The changesets history sidebar has received a design refresh. It is now consistent with the overall style language of the OSM NextGen. It now also includes infinite-scroll functionality! The data loads as you scroll, no more clicking the “load more” button.

GPS Trace Animations on Hover

The next generation trace animations are now also displayed in the list view. Simply hover over the entry to see it in action. Previously, the animations were part of the individual trace details page.

Rendering a complete animation in the browser takes about 0.5 ms. It’s really quick and requires no additional data download.

General Refactoring & Fixes

With the first development release approaching quickly, this week I focused on doing finishing touches to many of the core components. It’s important to perform such edits now, to avoid conflicts with other contributors work in the future.

Project Sponsors 🏅

I want to thank all the project patrons. Thank you for your help in building the next generation maps!

This week work has been sponsored by 13 people!
Four private and four public donors on Liberapay, and five public on GitHub Sponsors.

If you can, please consider supporting the OpenStreetMap-NG development.

Donate using Liberapay

OpenStreetMap NextGen Development Diary #8

Posted by NorthCrab on 22 May 2024 in English. Last updated on 23 May 2024.

Welcome to the eighth OpenStreetMap NextGen development diary.

This time the diary is one day early due to how much progress has been made.
There is more than enough for me to prepare a showcase 🏋.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

⭐ This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

Notes UI Gets a Fresh Look

The most noticeable change this week is a refreshed user interface for notes. It’s now more in line with how changeset discussions look, making the whole platform feel smoother and easier to navigate. Future work will include a discussion pagination support.

And here’s how it looks for those who aren’t registered:

A quick rundown of what’s changed:

  • Note discussions subscriptions
  • New social component with profile pics
  • Color-coded badges for resolved/unresolved notes
  • Easier to see notice when a note disappears from the map

Notes Layer Loads Way Faster

This week I’ve been optimizing the notes layer to make it snappier. The new protocol uses about 6-8x less memory, and is more lightweight on both user devices and the server. It’s all about sending only the bare minimum data needed for the notes layer to show up, and it’s built on the recently introduced new web rendering engine.

Here’s a peek at the data format:

» Open in full screen

The location link has a slightly new look that fits in better with its surroundings.

Feature Icons + Taginfo Integration

One of the more interesting things this week is the new taginfo integration. Feature icons now utilize tagging popularity information. This will improve the icon conflict resolution for complex tagging. For example, if a node is tagged as both a building and a cafe, the algorithm will now always choose the cafe icon because it’s less common.

Click here to browse the data file (if you are curious).

Prepping Up for the Development Release

I’m still working on getting everything ready for the first development release, which should be happening in early June. I’ve been putting together wiki documentation and setting up automated tests and code coverage reports.

Project Sponsors 🏅

Thank you all for your support of the project and my work! You are a real motivating factor that makes this project a reality. ✨

Currently, the project is sponsored by 12 people!
Four private and four public donors on Liberapay, and four public on GitHub Sponsors.

If you can, please consider supporting the OpenStreetMap-NG development 🦀:

Donate using Liberapay

Cat

Welcome to the seventh OpenStreetMap NextGen development diary.

This week, I focused on delivering core website functionalities and website and migration script optimizations 🚤.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

📖 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This initiative is not affiliated with the OpenStreetMap Foundation.

In Case You Missed It…

OpenStreetMap-NG is planned to have its first development release at the end of this month, May. After this milestone, the project will be open for new contributors! My current work focuses on delivering on that promise, finishing the core functionalities, and stabilizing the code.

Originally posted in diary #5.5.

New Fast Rendering Engine

One of the highlights of OpenStreetMap-NG is its brand new, super fast object rendering engine. It’s a significant improvement over the current solution.

For example, rendering all of Poland (about 1500 ways) takes a mere 34ms.

To put that in perspective, the same relation on the current OpenStreetMap website takes about 1.8 seconds. That’s over 50 times faster! And the best part: zero compromises on quality - it’s the same data, just way faster - badum tss.

Here’s the visualization we’re talking about:

Support For Historical States

The new rendering pipeline also lets you see how things used to look. Want to check out a street from 5 years ago? Just hover over the version in the history view. Easy.

This feature is going to make it way easier to understand how things have changed over time. It also opens up new opportunities for rich changeset diffs visualizations in the future!

Relations Members Icons

OpenStreetMap-NG now has a bunch of icons for relation members. This makes it much easier to tell what kind of relation you’re dealing with.

These icons are from the iD project.

Page Timeout Fixes

If you’re a frequent OpenStreetMap user, you might have run into errors like this:

OSM-NG has fixed a ton of these timeout issues with improved pagination. Now everything loads smoothly, even the history of a big relation like Poland.

Refreshed Element Sidebar UI

The element sidebar has received a makeover. The new look and feel are designed for regular users, not just developers. You’ll also find it much easier to use the element versioning system.

Project Sponsors 🏅

Time for the weekly appreciation of the current project patrons. Thank you, everyone, for supporting the project, including those who starred it on GitHub! There’s no way back now 🚂.

Currently, the project is sponsored by 13 people!
Four private and four public donors on Liberapay, and five public on GitHub Sponsors.

If you can, please consider supporting the OpenStreetMap-NG development 🦀:

Donate using Liberapay

Welcome to the sixth OpenStreetMap NextGen development diary.

This week, I continued the preparation of the project for the first development release scheduled for the end of this month 🔨.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

📖 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

🛈 This independent initiative is not affiliated with the OpenStreetMap Foundation.

In Case You Missed It…

OpenStreetMap-NG is planned to have its first development release at the end of this month, May. After this milestone, the project will be open for new contributors! My current work focuses on delivering on that promise, finishing the core functionalities, and stabilizing the code.

Originally posted in diary #5.5.

Finishing Up the Elements Sidebar

The elements sidebar has been mostly finished. The “Part of” and “Members/Nodes” sections are now more consistent in design and have received pagination support. The orange-colored map visualization is also now working.

Preliminary API 0.7 /map Implementation

The data layer is now working and uses the new API 0.7 for map requests. It also utilizes the new rendering algorithm, which will provide much better performance than the current implementation, handling more elements in view. The new map view padding optimization also reduces the number of API requests when panning around, providing a smoother user experience.

Public Feature Ideas

I have also publicized some collected feature ideas on the GitHub Issues tracker. These currently include only medium and long-term ideas. Short-term ideas will be added closer to this month’s deadline. Some features embedded within the code will also be added to this list soon - currently it’s not exhaustive as it covers only my personal notes.

General Code Cleanup

This week, the code has undergone cleanup. The database usage for private structures has also been reduced. By switching from UUID to Snowflake ID-inspired type, the type now takes just 64 bits of space instead of 128 bits.

Project Sponsors 🏅

Time for the weekly appreciation of the current project patrons. Thank you, everyone, for supporting the project, including those who starred it on GitHub! We are making it real 😎.

Currently, the project is sponsored by 14 people!
Five private and four public donors on Liberapay, and five public on GitHub Sponsors.

If you can, please consider supporting the OpenStreetMap-NG development 🦀:

Donate using Liberapay

Welcome to my fifth (and a half) OpenStreetMap NextGen development diary.
Tomorrow, I’m returning home and I’ll be able to resume work at full speed 🔥.

This is a short edition of the development diary.

🐙 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

Intro

For the past 13 days I have been on a journey of finding a new place to rent. Without my home office, I wasn’t able to become productive. The place I’m staying at doesn’t have a good office spot and being on my laptop doesn’t help. However, I am now very motivated to get back to work and push even harder!

May Will Be Big

At the end of May, OpenStreetMap-NG will include necessary functionality to run on a testing server, as well as to invite new contributors into the project. Starting with 6th of May, I won’t have any time-consuming plans for this month so I’ll do my best to wrap everything up. What’s exactly left has been described in Diary #5 Short-Term Development Plan. I have already started to prepare the All-in-One Contributor Guide which will also be finished up (it currently lacks backend/frontend-specific guides). This is going to be the first major milestone of the project!

Project Sponsors 🏅

I was happily surprised to see new faces even during my lower activity period. I will do everything to deliver the promised results. As always, thank you for supporting the project, monetarily, and with staring the project on GitHub!

Currently, the project is sponsored by 13 people!
Five private and four public donors on Liberapay, and four public on GitHub Sponsors.

Disclaimer

This project is not affiliated with the OpenStreetMap Foundation. It’s an independent and community-sponsored initiative.

I am currently on a visit to Ireland 🇮🇪 and a lack of proper office space makes it difficult to stay productive. I will try to prepare something cool to show off this week. Sorry for keeping you waiting!

🍟

Location: Murphystown, Leopardstown Rise, Glencullen Electoral Division, Sandyford, Dún Laoghaire-Rathdown, County Dublin, Leinster, D18 CV48, Ireland

Welcome to my fifth OpenStreetMap NextGen development diary.
This week has been mostly focused on GPS Traces 🛰.

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

🐙 This project is open-source and publicly available:
https://github.com/Zaczero/openstreetmap-ng

GPS Traces Demo

It’s best to experience the refreshed traces in a video form, so I prepared a short demo (no audio):
⏯️ https://files.monicz.dev/osm/traces-demo.mp4

For comparison, here’s how the same trace looks on the current website:
https://www.openstreetmap.org/user/bielebog/traces/11326871

You will quickly notice the super-fast upload speed and the new trace animations. If there’s something wrong with the file you attached, you will receive instant feedback on the upload page. One new feature is possibility to edit trace name. Previously, this feature has been hidden behind API 0.6.

One more planned feature is rendering the map behind the trace animation. Now that the system works on individual coordinates, it will be fairly easy to implement. Traces without a human-understandable point of reference are not as useful as they could be.

Unified Traces URLs

Let’s start with discussing the current URL routes.

  • If you want to access a way, you visit /way/<ID>
  • If you want to access a note, you visit /note/<ID>
  • If you want to access a trace, you visit /user/<USER>/traces/<ID>

Which is not consistent. OpenStreetMap-NG unifies this experience by introducing a new URL route: /trace/<ID>. All existing URLs remain backwards compatible and are automatically redirected.

Short-Term Development Plan

There’s just a few things left before reaching the core feature parity with the existing website. Those are the things I want to finish before inviting new contributors ツ.

  • Elements sidebar (50% work in progress)
  • User Diaries
  • User Profiles
  • Applications (OAuth) settings

Contributor Benefits

Last week I hinted towards the announcement of a contributor benefit. Today, I will talk shortly on 1 of 2 currently planned ideas, that will help the project grow and stay strong.

Firstly, who are the “contributors”? Those are the people who help the OpenStreetMap-NG project. For example: by testing the website, donating, contributing code, helping with localization, graphics and interface design, etc.. The scope is broadly defined, as people can contribute in many different ways!

Contributors joining before the project is officially accepted as the main OpenStreetMap website, will be able to become a member of the NextGen Founders invite-only community and receive a small badge on their user profiles. This feature is a part of the original announcement (under the name “Community Profiles”).

This is a time-limited benefit, that provides a unique thank-you to all people that help (and will help) making this project a reality. The 2nd benefit will be announced in some time in the future.

Project Sponsors 🏅

Here’s my weekly appreciation to the current project patrons. Thank you for believing and helping me do what I love :-)

Currently, the project is sponsored by 11 people!
Five private and three public donors on Liberapay, and three public on GitHub Sponsors.

If you can, please consider supporting the NG development:

Donate using Liberapay

Disclaimer

This project is not affiliated with the OpenStreetMap Foundation.

Welcome to my fourth OpenStreetMap NextGen development diary.
Sorry for being a day late! I wanted to finish up one of the new features which caused this small delay. ✨

🔖 You can read my other development diaries here:
https://www.openstreetmap.org/user/NorthCrab/diary/

🐙 My work is available publicly on this GitHub repository:
https://github.com/Zaczero/openstreetmap-ng

GitHub stars counter

Let’s summarize the last week’s work:

Client-side Trace Images

While migrating the traces functionality, I came up with an amazing and seemingly obvious idea. Why not make trace images SVGs and render them client-sided? This feature has few significant advantages: even faster trace uploading, no additional disk usage, unlimited customization, infinite resolution, faster page loading. And so here it is:

Comparison screenshot of the new client-side generated trace images (SVGs)

The application can freely adjust the quality of generated images. The code can also be reused for implementing proper trace-on-map rendering, which is one of the new upcoming features.

SVG supports animations too!

Animated trace using SVG

Refreshed Traces UI

Screenshot showcasing refreshed public traces UI

» Open in full screen

Last week I have also worked on refreshing the traces UI, focusing on making it more open and friendly. If you have been following my previous diaries, you may recognize some of the new style language.

Deployment Scripts

I also wrote and successfully tested server-deployment scripts for the application. They are currently a part of the openaedmap-backend project but will soon be copied over to the openstreetmap-ng. Both projects share many similarities in how they are run.

Final Words

Previous development diary #3 was packed with lots of new stuff. I took this week a little slower to catch a breath. Meantime, I contributed to other projects (openaedmap, starlette) and also helped OSM-PL with server migration process. I was also away for a short time for some BBQ🌞!

Project Sponsors 🏅

Thank you to this week’s project patrons! I truly appreciate your every contribution!

Currently, the project is sponsored by 11 people!
Five private and three public donors on Liberapay, and three public on GitHub Sponsors.

If you can, please consider supporting the NG development. Any amount helps me focus on making high-quality free software for everyone to enjoy! (shh… next week I will announce a new supporter benefit) 😋

Donate using Liberapay

Disclaimer

This project is not affiliated with the OpenStreetMap Foundation.

OpenStreetMap NextGen Development Diary #3 - Packed with Goodies

Posted by NorthCrab on 31 March 2024 in English. Last updated on 1 April 2024.

Welcome to my third OpenStreetMap NextGen development diary.
This week has been super busy and I can’t wait to show you the progress 🧑‍🍳!

You can subscribe to my diary updates on RSS here:
https://www.openstreetmap.org/user/NorthCrab/diary/rss

Video Presentation 🎉

This week I was focused on integrating features together, as we get closer and closer to the first development release. I am super happy with how things are progressing and I decided to record a short video presentation (3 minutes).

⬇ Click below to play ⬇

Video thumbnail

or click here: https://peertube.monicz.dev/w/kGnomi7LTveXZNaaQtEwH6

Refreshed Changeset UI

Changeset sidebar comparison screenshot, showcasing simplified and refreshed UI

OpenStreetMap-NG now features a refreshed changeset sidebar. Featuring, profile pictures and de-emphasized discussion subscription button. I’ll collect feedback on this design decision (and others) after the first development release.

Working Rapid & iD editors

OpenStreetMap-NG delivers on its promise and features Rapid editor as an alternative to iD.

Navbar screenshot, highlighting Rapid editor option

A lot of effort was put in into securing against potentially malicious JavaScript code making requests as a user to the website (not API). Uncompressed Rapid editor weighs about 5MB, with iD being around 4MB. A library of such size is impossible for a human to review properly, making it an easy target for abuse. With OpenStreetMap-NG, hosted editors can’t physically use user cookies, mitigating all the risk.

New Icons System

Elements view, showcasing deleted nodes with defibrillator and bench icons

There are plenty more feature icons and many original ones are now in higher resolution. Many of the icons have been borrowed from OpenLevelUp editor as they closely resemble the existing icon style and blend together nicely. There are currently 286 features with icons.

There is also this neat new feature, icons and feature names are now displayed correctly for deleted elements. This should make browsing through changesets easier.

Elements Pagination 2.0

Screenshot of pagination control under the elements list

Navigating through element pages is now instant and does not require a full sidebar reload. Your browser only loads necessary resources to display the current page of elements. It’s fast, it’s efficient, it’s simple — and most importantly, it’s not annoying to use!

(check out the video presentation for a short demo 😉)

Support for Multiple Value Tags

OpenStreetMap-NG introduces native support for key prefixes/suffixes and multi-value tags. The system can be easily extended with support for new tagging schemas with the use of formatting plugins. Keys are separated by : and values are separated by ;.

And More…

All of my work is publicly accessible on this GitHub repository. While my development diaries focus on selected highlights, there are many more details to read through. You can always browse the commit history with day-by-day updates.

Project Sponsors 🦀

I am super grateful for all this week’s project patrons. Your growing support makes me work at 110% capacity! Everything is possible if you just believe it. Thank you 🫰.

Currently, the project is sponsored by 10 people!
Five private and two public donors on Liberapay, and three public on GitHub Sponsors.

This week’s donations go directly towards the development of OpenStreetMap-NG.

Donate using Liberapay

Disclaimer

This project is not affiliated with the OpenStreetMap Foundation.

Welcome to my second OpenStreetMap NextGen development diary. I am ready to highlight this week’s progress — and there was a lot 👷!

You can subscribe to my diary updates on RSS here:
https://www.openstreetmap.org/user/NorthCrab/diary/rss

Finished Settings Page

Last week I showcased initial progress on the new settings page for OpenStreetMap. This week, I’ve finished it. Here’s what’s new. Keep in mind that if you believe some features should be done differently, there will be a period of public testing where I’ll collect this kind of feedback 😉.

New settings page screenshot

Click here to view it in full screen.

Aside from the improved layout, I’ve introduced two notable changes. First, the application language selection has been redesigned; it’s now a simple dropdown selector. I believe the current system (where you input locale names by hand) is difficult for less technical people to understand, and I want to make the website more accessible to everyone!

New privacy controls screenshot

Secondly, OpenStreetMap finally gains account privacy controls. Want to opt out of activity tracking but still send crash reports? No problem! Today’s opt-out implementation relies solely on the Do-Not-Track browser setting. Even when opted out, you’ll still connect to analytics servers (for some reason). The NextGen codebase supports Do-Not-Track headers, the new Global Privacy Control, and does not connect to analytics servers when opted out. Account configuration takes precedence over browser configuration.

Seamless 180th Meridian Handling

This week I’ve invested time in smoothing out handling of the 180th meridian in both the frontend and backend. When browsing the map around the 180th meridian, there are no more visual glitches when crossing it. Additionally, the map and notes layers will no longer throw exceptions around it. The new position handling script is also significantly smaller compared to the existing Ruby codebase. The backend API now seamlessly handles 180th meridian requests.

The current API integration when requesting an area from 160° to 200°:

  • Send 1st request from 160° to 180°
  • Send 2nd request from -180° to -160°
  • Merge results

The new integration:

  • Send 1 request from 160° to 200° (or from -200° to -160°)

Splitting, wrapping, and merging are done under the hood by the API, significantly reducing code complexity in many consumer applications. Less complexity means fewer bugs and easier integration with OpenStreetMap.

System Apps and Test Users

As I actively work towards creating the best developer experience, this week I focused on two new features: system apps and test users.

System apps are OAuth2 applications managed by the application itself, not by users. Currently, there are two system apps: iD and Rapid — the supported web editors.

INFO:     | 2024-03-23 23:43:45 | Registering system app 'iD'
INFO:     | 2024-03-23 23:43:45 | Registering system app 'Rapid'

In the current Ruby codebase, you need to follow all these steps just to get your own iD instance running. OpenStreetMap-NextGen does this automatically on application launch for you — you’re welcome 🙂.

Debug login panel screenshot

Test users are just that: users for testing. I like when applications provide developers with multiple users ready to be used, rather than requiring them to go through the registration process. Contributors can now log in to any of these users with a single button press!

Progress on Notes and Changesets API 0.6

Changeset read API response screenshot

I’m actively working on the migration of existing API features. This week, I successfully tested CRUD (Create, Read, Update, Delete) operations on notes and changesets. There were a few issues that needed correction, but nothing notable.

And More…

All of my work is publicly accessible on this GitHub repository. While my development diaries focus on selected highlights, there are many more details to read through. You can browse the commits history.

Project Sponsors 🦀

I want to take a moment to personally thank this week’s project patrons. It was super motivating to see new supporters! I really appreciate all of you! 🫰

Currently, my work is sponsored by 5 patrons on Liberapay, including three private donors. I have one public donor on GitHub Sponsors and have also received a private donation via PayPal.

~1847430, gileri, 快乐的老鼠宝宝/LaoshuBaby

If you’d like to join my development sponsors, you can find me on Liberapay or GitHub Sponsors. Currently, all contributions go directly towards the development of OpenStreetMap NextGen 🚀.

Donate using Liberapay

Disclaimer

Please note that this project is not affiliated with the OpenStreetMap Foundation. It’s the result of my voluntary work and personal choices.

Today I decided to introduce a new format for sharing OpenStreetMap-NextGen development progress with the community. I’ll post weekly/bi-weekly updates highlighting changes and the current project status. Since this is the first update, I’ll cover some recent highlights.

You can subscribe to my diary updates on RSS: link.

New Settings Page (⭐ Highlight)

New settings page screenshot

I’ve begun migrating the settings/preferences section. My goal is to streamline this experience, as I’ve found the current system a bit complex. Surprisingly, many users don’t know it’s possible to change the default editor — I want to make this more obvious.

A new menu on the left of the screenshot (hidden, not yet finished) will provide clear navigation between general, 2FA, OAuth, and other settings.

This page is still work in progress. I intend to add a help text explaining how to contribute to translations and that the translations are made by the community.

Image optimization logging output, console screenshot

This screenshot highlights a new image optimization algorithm, which uses a binary search-like algorithm to find the perfect image optimization configuration in limited amount of steps.

Last Week’s Progress

I am heavily focused on migrating the HTML templates and pages. I believe it is a critical step towards opening up the NextGen codebase to new contributors. Without those (mostly) functional pages, it is difficult to add new features or improvements.

The following templates have been worked on:

  • /welcome - finished
  • /fixthemap - finished
  • email base template - finished
  • email signup confirm - finished
  • /settings - work in progress

I have additionally addressed issues that lead to some issues with API endpoints, as well as worked on frontend and backend optimizations. You can see the full breakdown in the repository commits log. I keep my work completely transparent.

OpenStreetMap Website Vulnerability Report

I finally published my OpenStreetMap website vulnerability report. I conducted this security audit while studying the website source code, which was a mandatory step to preserve backwards compatibility.

Some of the highlight findings is a security flaw allowing an attacker to blindly reply to any private message as anybody. Another surprising finding is that the Ruby website stores user authentication tokens in plain text. If an attacker had gained access to the server where these tokens were stored (with just read access), they could have potentially compromised a large number of accounts.

All of the vulnerabilities have already been fixed or are being fixed in the NextGen implementation.

OpenStreetMap NextGen Benchmark 1 of 4: Static and unauthenticated requests

I have recently published the first benchmark of the OpenStreetMap-NG. It focuses on measuring static and unauthenticated requests as this code is fairly stable unlikely to be changed. Future benchmarks will include more realistic scenarios.

I compared the results with the current Ruby website implementation. I faced issues with reproducing deployment scenario on my local machine due to outdated documentation (and since I am a Ruby-noob, I couldn’t fix it myself).

Despite the imperfect benchmarks, I believe the obtained numbers hint at the potential performance gains of NextGen’s codebase.

🦀 Project Sponsors

In my development diaries, I want to include a dedicated section thanking my current project patrons. It’s through their support that I’m able to work full-time on OpenStreetMap-NextGen. Rather than focusing on the amount donated, I want to highlight the individuals themselves — it’s the gesture that is the primary driving factor.

Currently, my work is sponsored by 2 patrons on Liberapay, including one private donor, and one public donor with the mysterious looking username ~1847430.

Thank you to both of you, you made me smile 😋.

If you’d like to join my development sponsors, you can find me on Liberapay or GitHub Sponsors. Currently, all contributions go directly towards the development of OpenStreetMap NextGen.

Donate using Liberapay

Disclaimer

Please note that this project is not affiliated with the OpenStreetMap Foundation. It’s the result of my voluntary work and personal choices.

OpenStreetMap Website Vulnerability Report

Posted by NorthCrab on 15 March 2024 in English. Last updated on 18 March 2024.

As part of my commitment to OpenStreetMap-NextGen migration, I undertook a comprehensive security review of the existing OpenStreetMap website. I followed the principles of coordinated vulnerability disclosure, working directly with the maintainers to responsibly report my findings. Today, I’m making the details of this process public and verifying the status of the fixes.

Disclosure Timeline

  • 2023-11-04 - Contacted Ruby security maintainers & disclosed the timeline publicly
  • 2023-11-08 - Maintainers acknowledged the report
  • 2023-12-04 - Reported additional vulnerabilities with a 3-month deadline
  • 2023-12-06 - Maintainers acknowledged the additional report
  • 2024-03-02 - Publicize vulnerability details
  • 2024-03-15 - Publicize vulnerability details

1. Plain-Text Authentication Token Storage

Authorization tokens (oauth, session, user tokens) are stored in plain text. Read access to the database allows full impersonation of any user.

Status as of 2024-03-15: Partially vulnerable (oauth still depends on plain text storage)

NextGen Codebase Status: Fixed (all authorization tokens and credentials are hashed or encrypted for secure storage)

2. Insecure Email Reply Address Tokenization

The tokens used to ensure the authenticity of email replies are too short (24-bit), making them susceptible to brute-force attacks. An attacker could potentially guess the token and impersonate a legitimate user in email conversations. This vulnerability is especially dangerous if the attacker already has access to a conversation’s metadata (allowing for complete security bypass).

Status as of 2024-03-15: Fixed (now with 48-bit security)

NextGen Codebase Status: Fixed (128 or 256-bit security; undecided)

3. Unbounded GPX Extraction Denial of Service

The OpenStreetMap website does not impose limits on the number of files that can be extracted from a GPX archive. Attackers can exploit this by uploading specially crafted archives containing a massive number of files, potentially overwhelming the system and causing it to crash.

Status as of 2024-03-15: Mostly vulnerable (a partial fix limits points within trace files but does not fully address the issue)

NextGen Codebase Status: Fixed

4. Notes Search Query Denial of Service

The /api/0.6/notes/search endpoint is poorly optimized. This allows attackers to craft specific search queries that are extremely resource-intensive, effectively using the search feature to launch a denial of service attack.

Status as of 2024-03-15: Vulnerable

NextGen Codebase Status: Not yet tested

5. String Comparison Timing Attack

The way the code compares strings can, under certain circumstances, leak timing information. When attackers can carefully measure these timing differences, they might be able to extract secret information, such as security keys and tokens.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Fixed

6. User Block and Rate Limit Bypass

A loophole exists where a malicious user can repeatedly delete and recreate their account (using the same credentials) to circumvent security measures like user blocks and rate limits.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Not yet tested

7. Application Preference Leakage

OpenStreetMap allows different applications to store user preferences. A flaw exists where any application can access preferences created by a different application. This could leak information, and might be abused to attack other vulnerable applications that mistakenly trust data stored in user preferences.

Status as of 2024-03-15: It’s intended behavior

NextGen Codebase Status: Fixed (in API 0.7; via optional preference partitioning)

8. Unrestricted Email Checking via Password Recovery

The password recovery feature can be abused to check if any arbitrary email address is associated with an OpenStreetMap account.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Fixed

9. Forced Downgrade from OAuth1.0a to OAuth1.0

Under specific conditions, the authentication protocol can be downgraded from the more secure OAuth1.0a to the less secure OAuth1.0. This downgrade would omit important security checks.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Fixed

10. Insufficient User Token Integrity Checks

Tokens, used for things like password resets or confirming email changes, aren’t properly differentiated. An attacker might be able to use a password reset token to change an email address, or vice-versa.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Fixed

11. Inadequate Randomness Source in Token Generation

The system generating security tokens relies on a weak source of randomness. Predictable tokens are much easier for attackers to guess or manipulate.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Fixed

12. Unicode Normalization Flaw Allowing Email/Display Name Duplication

The lack of Unicode normalization means a user can register with an email address or display name that is identical to an existing one (but uses different Unicode characters). This could be used to trick users or bypass database restrictions.

Status as of 2024-03-15: Fixed

NextGen Codebase Status: Not yet tested


Support the NextGen development

I work on the OpenStreetMap-NextGen full-time. If you find my work valuable, please consider supporting the development on Liberapay or GitHub. Every contribution helps push us closer to that first stable release. Thank you! 🦀

And a huge thank you to those who have already supported me!

Submit a correction

If you find any inaccuracies or have suggestions to improve this diary, please feel free to contact me via https://monicz.dev/#get-in-touch.

Disclaimer

Please note that this project is not affiliated with the OpenStreetMap Foundation. It’s the result of my voluntary work and personal choices.

Today marks a milestone in the development of OpenStreetMap NextGen. After months of rigorous development, I conducted the 1st OpenStreetMap NextGen performance benchmark, a crucial step towards realizing the vision of a more robust, efficient, and user-friendly OpenStreetMap.

The focus of today’s benchmark was on evaluating static and unauthenticated requests. Since this core functionality is unlikely to change significantly during future development, it’s the perfect time to test it.

Future benchmarks will focus on timing authenticated requests as well as API 0.6.

What was measured

The benchmark analyzed request processing speed, excluding network and client latency. Both osm-ruby and osm-ng support the X-Runtime response header, which tracks how long it takes to process a request and generate a response.

X-Runtime header in browser inspect tools

Here’s a general breakdown of a typical static request processing:

  • Grabbing configuration settings
  • Checking for authorization (cookies, oauth, etc.)
  • Configuring translations
  • Rendering the HTML template

The setup

The benchmarking setup consisted of local machine testing, as well as official production and development websites.

I initially planned to run the benchmark solely on my local machine, following the official Docker instructions. However, I quickly discovered that the production deployment instructions were outdated and required some Ruby knowledge to fix, which I lacked. In particular, the instructions for replacing the Rails server with Phusion Passenger had been redirected to a generic support page.

osm-ng was launched in production mode with all Python and Cython optimizations enabled. Since we were only dealing with static requests, both local Postgres databases remained empty.

The benchmarking script

I created a basic HTTP benchmarking script that first warmed things up with a few requests before launching into the actual test. It then measured runtime times for a series of HTTP requests, and I repeated the benchmark multiple times for consistency.

A note before the results

It’s important to remember that OpenStreetMap NextGen processes static requests in a similar way to osm-ruby, and it does not (currently) introduce any new caching logic for templates, especially since that would significantly impact the benchmark results (and some people would consider it cheating). osm-ng remains completely backwards compatible with the existing OpenStreetMap platform. Additionally, it’s important to emphasize that the X-Runtime header used for benchmarking is agnostic to network latency, meaning it only measures the processing time on the server itself.

And the winner is…

Here’s a detailed breakdown of the results:

Environment Minimum Runtime (s) Median Runtime (s)
Ruby (local) 0.04264 0.04521
Ruby (official) 0.01892 0.02921
Ruby (test) 0.00913 0.01725
Python 0.00314 0.00325

As you can see, osm-ng consistently outperformed osm-ruby in all test scenarios. The fastest Ruby deployment had the minimum runtime of 0.00913 seconds, while osm-ng achieved the blazing-fast time of 0.00314 seconds, a remarkable 290% performance improvement.

Support the NextGen revolution

I’m truly convinced that OpenStreetMap NextGen will be a game-changer for OpenStreetMap, not just in terms of performance, but also in privacy, security, usability, and overall openness.

If you believe in this project as much as I do, please consider donating so I can keep working on it full-time! 🙏 Every contribution helps push us closer to that first stable release.

And a huge thank you to those who have already supported me!

Today, we benchmarked not just a system, but the future. And the future is bright.

I am really excited to finally share some tangible progress on OpenStreetMap NextGen! While I have been working on this project almost day-by-day for the past 3 months, just recently I reached the point of launching the website (for now locally) and begun development of its frontend features.

OpenStreetMap-NG has grown immensely in scope since the original announcement. Now the project not only focuses on core performance, usability, and accessibility but also on improving the user-facing interface and experience. I want this project to truly stand by its name, providing NextGen value to all aspects of the website. And so far, it’s on the right track.

Directions side-by-side comparison.

So far, the number of improvements is counted in hundreds, including but not limited to various bug-fixes, performance optimizations, privacy, and security enhancements. And the list keeps on growing!

Share panel side-by-side comparison.

My current primary focus is to wrap up frontend baseline development and prepare an amazing developer documentation so other people can hop in and help. I still need a little more time to prepare the grounds as I want to provide a good contributor experience and ensure everyone’s time is used effectively.

Data export side-by-side comparison.

The project is currently funded through individual donations. You can help it grow by donating here, or by leaving a star on the GitHub project itself, or both :-)

An orange crab holding a glowing star, space ship flying in the background

Please note that this project is not affiliated with the OpenStreetMap Foundation. It’s the result of my voluntary work and personal choices. The presented side-by-side comparisons are subject to change as the development is ongoing.