OpenStreetMap logo OpenStreetMap

rphyrin's Diary

Recent diary entries

Severe floods and landslides devastated Sukabumi Regency, West Java, after two days of intense rainfall, with over 100 mm of rain falling in a short period, according to the Meteorology, Climatology, and Geophysics Agency (BMKG). On Wednesday, December 4, the overflow of the Cikaso and Cibening Rivers inundated numerous areas, displacing residents and severing access roads. The hardest-hit areas included Palabuhanratu, Sagaranten, and Pabuaran. Floodwaters reached heights of 80–90 cm, submerging homes and halting transportation.

In Sagaranten, neighborhoods like Kampung Rangcabungur faced dramatic rescue operations as narrow alleys flooded waist-deep. Rescue teams, battling strong currents, evacuated several infants and their mothers. “We successfully rescued two to three babies. It was a tense process, but thankfully, everyone was saved,” said the Head of the Sukabumi Police’s Samapta Unit.

Flooding impacted at least 27 villages across 20 districts, with seven districts, including Ciemas and Gegerbitung, reporting widespread inundation. Landslides struck 14 locations, particularly in Simpenan, Warungkiara, and Palabuhanratu, while soil movements were reported in Cikembar and Bantargadung. The collapse of a 12-meter-long bridge on the Loji-Puncak Darma road added to the region’s challenges.

In Kampung Parungseah, Desa Curugluhur, and neighboring areas, the Cikaso River’s overflow swept away vehicles and flooded dozens of homes. Officials from the Sukabumi Regional Disaster Management Agency (BPBD) identified six districts as flood zones and seven districts as landslide hotspots.

Emergency response teams, including 61 personnel from the National Police’s SAR unit, were deployed to assist in rescue and recovery efforts. Local authorities have begun repairing infrastructure and monitoring weather conditions as heavy rains continue to threaten the region.

Upon learning about this news, I used OSM_PANTAU! to explore the humanitarian mapping activities by the Indonesian community in the affected region.

The circles are color-coded to indicate the recency of activity: red represents more recent updates, while blue indicates older ones.

… and wait 1 or 2 days

Posted by rphyrin on 1 December 2024 in English.

On the first day of every month, I run a manual monthly statistics script to calculate which Wikipedia article is the most active.

Today, while analyzing this month’s statistics logs, I came across an interesting article.

That article discusses the efforts to integrate public transportation in the Jakarta metropolitan area.

This article also included a map that displays the current extent of the integration effort.

But I noticed something was missing, and now I’m itching to fix it.

So, I immediately hit the ‘edit source’ button.

Later, I found out that this map is generated using the MediaWiki Maplink extension.

But I don’t see any line coordinates here. All I see is just the Wikidata item ID.

So, I checked that Wikidata item.

Still, no line coordinates.

Where are the line coordinates?

I need to find them if I want to add a new one!


The next few minutes turned into a hide-and-seek game with these elusive line coordinates.

There’s nothing left to uncover on Wikidata. They could be stored on Commons (but linked through Wikidata), or as template data on the local Wikipedia (also linked through Wikidata). I checked all the possibilities, but still, no luck.

So, I turned to the Maplink extension documentation.

That’s when I had some revelations.

The line coordinates are extracted directly from OpenStreetMap.

So, all we need to do is link the Wikidata item to its corresponding OSM node, way, or relation.

Alright, let’s do this.


Create a new Wikidata item, link it to the OSM relation, and use that Wikidata item ID in the Maplink syntax.

Then, submit and refresh the page.

But… nothing happened.

Oh no.

I did everything right, but still, nothing happened!

So, I started over from the very beginning. Maybe I missed something at the start? Maybe it’s not on OSM? Maybe it’s stored on Commons or in the template data? Oh, no no no.


…..

…..

…..

While re-reading the Maplink extension documentation, I came across something shocking.

Apparently, the OSM-to-Wikipedia extraction process can take days.

So, I just have to wait…

Hmm…

….

Alright then.

Case closed, I hope.

Let’s just wait a few days and see what happens.

OpenStreetMap - Debian 12

Posted by rphyrin on 30 November 2024 in English.

Concept : I will read OSM-related news, summarize passages from all the web links related to that news, and then post them here.

My main intention is to learn about it in depth.


January 29, 2023

On running sudo apt update, users will get a notification similar to this, stating that several packages from the universe repository have security updates that require Ubuntu Pro:

The following security updates require Ubuntu Pro with ‘esm-apps’ enabled: imagemagick libopenexr25 libmagick++-6.q16-8 libmagickcore-6.q16-6-extra libmagickwand-6.q16-6 imagemagick-6.q16 libmagickcore-6.q16-6 imagemagick-6-common

February 10, 2023

Currently we run a combination of Ubuntu 20.04 LTS and Ubuntu 22.04 LTS across our server estate.

Ubuntu 20.04 has recently moved away from the established 5 years of maintenance updates and now requires a paid Ubuntu Pro plan to receive some package security updates.

We’ve also had issues with Ubuntu not fully supporting the packages they source from Debian (e.g. : Apache2 which required us to backport a fixed version to workaround an issue affecting mpm event scaling).

Ubuntu has also caused us issue with some packages now only being distributed as snap packages (e.g. : Firefox).

February 20, 2023

Your Ubuntu LTS is still secured in exactly the same way it has always been, with five years of free security updates for the ‘main’ packages in the distribution, and best-effort security coverage for everything else. This has been the promise of Ubuntu since our first LTS in 2006, and remains exactly the same. In fact, thanks to our expanded security team, your LTS is better secured today than ever before, even without Ubuntu Pro.

Ubuntu Pro is an additional stream of security updates and packages that meet compliance requirements such as FIPS (Federal Information Processing Standards) or HIPAA (Health Insurance Portability and Accountability Act), on top of an Ubuntu LTS.

As part of our global mission to amplify the impact of free software, we offer a free personal subscription to Ubuntu Pro that covers up to 5 machines.

November 24, 2023

Servers fume and grisu are now running Debian 12.

  • fume.openstreetmap.org (discourse server) : HPE ProLiant DL360 Gen10, 2 x 20 core Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, memory 378GB Total (12 x 32GB 2400 MT/s DDR4 DIMM), 2 x 4TB Micron Technology Inc 7450 PRO NVMe SSD, Debian GNU/Linux 12 (bookworm), Equinix Dublin.
  • grisu.openstreetmap.org (overpass server) : HPE ProLiant DL360 Gen10, 2 x 20 core Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, memory 378GB Total (12 x 32GB 2400 MT/s DDR4 DIMM), 2 x 4TB Micron Technology Inc 7450 PRO NVMe SSD, Debian GNU/Linux 12 (bookworm), Equinix Dublin

August 30, 2024

“After 18 years on Ubuntu, we’ve upgraded the @openstreetmap servers to Debian 12 (Bookworm)”

November 27, 2024

There is a large overlap between OpenStreetMap mappers and the Debian community.

Part of our reason to move to Debian was to get closer to the maintainers of the packages that we depend on. The Debian package maintainers do an excellent job of maintaining their packages - e.g.: osm2pgsql, osmium-tool etc.


OpenStreetMap.org is primarily run on actual physical hardware that our team manages. We attempt to squeeze as much performance from our systems as possible, with some services being particularly I/O bound.

We ran into some severe I/O performance issues with Linux kernel versions ~6.0 to < ~6.6 on systems with NVMe storage.

Non-Volatile Memory Express, a high-speed storage technology that connects directly to the PCIe bus, offering much better performance than older SATA-based drives; The issue could stem from changes in how the kernel handles NVMe drives, such as queueing, scheduling, or power management.

This pushed us onto newer mainline kernels, which led us toward Debian. On Debian 12 we could simply install the backport kernel and the performance issues were solved.

A backport kernel refers to a newer version of the Linux kernel that has been made compatible with an older or current stable release of a Linux distribution. Debian Stable is known for prioritizing stability over cutting-edge features. Its default kernel is often several versions behind the latest mainline Linux kernel. However, users who need newer kernel features can install a backported kernel. The backport kernel is a newer kernel version (from Debian Testing or Unstable) that has been adapted for use in the current stable release.


Thankfully we manage our server setup nearly completely with code.

We use Chef to automated the configuration of all of our servers.


“It looks to me that the open source version of chef is free to use, however if I want a web based GUI, I’d have to pay for Chef Automate. Does anyone have experience running just the command line tools? Are there any gotchas I should be aware of? I’m interested in hearing about what people’s production chef environments look like when they’re not paying for anything.”

“We use Open Source Chef server and have reproduced most of the knife commands as Jenkins jobs which execute API commands. It is good for automation and speed.”


“My situation is as follows: I have a few computers running Windows (fewer than 10) and several virtual machines (some on Windows, some on Linux). I maintain them manually, but this has become increasingly difficult over time. I looked into automation options like Chef, Puppet, and Ansible, and I believe Chef is the best fit for my needs. However, I’m unsure where to start if I want to use only the free components of the system. When I contacted support via chat, they directed me toward a paid plan. Can you recommend any tutorials that explain how to set up and use the free components of Chef infrastructure? Do you use Chef in its free version? If so, what are the key differences or limitations compared to the paid plan?”

“The free version should probably suffice for your use case. You might want to consider a paid version if the number of nodes increases or you want a dashboard interface (Chef Automate) to manage them.”

“Chef is open source but by default you can’t really use the code without re-branding and building it yourself.”


We also use Test Kitchen with inspec to test this infrastructure code.

Test Kitchen : It creates temporary environments (e.g., virtual machines or containers) where configuration changes can be tested safely before applying them to production servers.

InSpec : A framework for writing and running automated compliance and infrastructure tests.

Tests run locally using Podman or Docker containers, but also run as part of our git code pipeline.

Manage containers, pods, and images with Podman. Compatible with other OCI (Open Container Initiative) compliant container formats including Docker. Run your legacy Docker containers (including docker-compose files) on Podman.

It creates temporary environments (e.g., virtual machines or containers) where configuration changes can be tested safely before applying them to production servers. Containers simulate the servers, allowing infrastructure code to be tested without affecting actual production systems.

A code pipeline (e.g., CI/CD system like Jenkins, GitHub Actions, GitLab CI) automatically runs tests whenever changes are made to the codebase. This ensures that : (1) Errors are caught early; and (2) Only tested, reliable configurations are deployed.

We added Debian as a test target platform and fixed up the infrastructure code until all the tests passed. The changes required were relatively small, simple package name or config filename changes mostly.


In August 2024 we moved the www.openstreetmap.org Ruby on Rails servers across to Debian. We haven’t yet finished moving everything across to Debian, but we will upgrade the rest when it makes sense.

Some systems may wait until the next hardware upgrade cycle.

Our focus is to build a stable and reliable platform for OpenStreetMap mappers.


We are still in the process of fully migrating between Linux distributions, but we can share that we recently moved our frontend servers to Debian 12 (from Ubuntu 22.04) which bumped the Ruby version from 3.0 to 3.1 which allowed us to also upgrade the version of Ruby on Rails that we use for www.openstreetmap.org.

We also changed our chef code for managing the network interfaces from using netplan (default in Ubuntu, made by Canonical) to directly using systemd-networkd to manage the network interfaces, to allow commonality between how we manage the interfaces in Ubuntu and our upcoming Debian systems.

Over the years we’ve standardised our networking setup to use 802.3ad bonded interfaces for redundancy, with VLANs to segment traffic; this setup worked well with systemd-networkd.

IEEE 802.3ad Link Aggregation Control Protocol (LACP). Allows combining multiple physical network interfaces into a single logical interface (a “bonded” interface). If one physical link fails, the others continue to operate, ensuring uninterrupted network connectivity. By aggregating multiple links, it increases the total throughput available to the system.

VLANs (Virtual Local Area Networks) : A way to logically separate network traffic on the same physical network infrastructure. Each VLAN is assigned a unique identifier (e.g., VLAN ID), and traffic is tagged to belong to a specific VLAN. Traffic segmentation increases security by isolating different types of traffic (e.g., separating user traffic, server traffic, and management traffic).

systemd-networkd : A component of the Linux systemd suite that handles network configuration.

We use netboot.xyz for PXE networking booting OS installers for our systems and use IPMI for the out-of-band management.

netbootxyz : A tool that consolidates various operating system installers into a unified network-bootable interface. Allows administrators to boot directly into OS installers or utilities over the network without needing physical media (like USB drives or DVDs).

PXE (Preboot Execution Environment) : A protocol that allows a computer to boot from a network interface before the operating system is loaded. Enables automated installation or maintenance of operating systems across many systems without local setup.

IPMI (Intelligent Platform Management Interface) : A protocol for managing and monitoring servers remotely, even if they are powered off or unresponsive. Provides out-of-band management, meaning it operates independently of the main operating system. Capabilities : (1) Remote power control (e.g., rebooting or powering on/off); (2) Monitoring hardware health (e.g., temperatures, fan speeds, or voltage levels); (3) Accessing the server’s console (like using a keyboard and monitor physically connected to the server).


In some cases we do run pre-release or custom patches of OpenStreetMap software.

With Ubuntu we used launchpad.net’s Personal Package Archives (PPA) to build and host deb repositories for these custom packages.

We were initially perplexed by the myriad of options in Debian, but received some helpful guidance from a Debian contributor and we now manage our own deb repository using aptly.

For the moment we’re currently building deb packages locally and pushing to aptly; ideally we’d like to replace this with a git driven pipeline for building the custom packages in the future.

Sources

  1. skorupa12 (June 23, 2021) “How to use Chef Open Source” r/chef_opscode
  2. Artur Meinild (January 29, 2023) “What are ESM Apps, and how do they relate to Ubuntu Pro?” Ask Ubuntu
  3. Firefishy (February 10, 2023) “Switching away from Ubuntu LTS to Debian as our preferred distribution” OpenStreetMap Operations
  4. Lech (February 20, 2023) “Ubuntu Pro - FAQ” discourse.ubuntu.com
  5. Donald Norwood, et al. (November 27, 2024) “OpenStreetMap migrates to Debian 12” debian.org