There are about 450,000 open notes on OpenStreetMap. Each one is a message from someone who saw a problem on the map. But to know what a note needs, you have to open it and read it, one by one. That takes a lot of time.

So I trained a machine learning model to do this first read for us. I also made a simple website to show the results:

→ https://osm-notes.rub21.com

What does the model do?

The model reads the first comment of each note and gives it two labels:

• Category: what the mapper should do. For example: missing place, place closed or changed, wrong name, missing or wrong road, other wrong data, spam, empty, or not mappable (opinions, questions).
• Theme: what the note is about. For example: food, shops, health, roads, transport.

How I built it

1. The notes. Andrés (AngocA) built OSM-Notes-Ingestion. It loads all OSM notes into a database and keeps them up to date every minute. On top of it, I built a small pipeline that:
   • loads the full notes dump from the planet file,
   • takes the first comment of each note (the one that explains the problem),
   • detects the language of each note.
2. Training data. This was the hardest part, and it took several rounds. The AI helped me a lot, but I reviewed many of its labels:
   • I sent about 10,000 notes to three popular AI models: Claude, GPT and Gemini. When all three agreed, I kept the label, and I also reviewed many of them.
   • When they did not agree, I read the note and chose the label myself. I labeled about 1,300 notes this way.
   • After the first training, I looked at the notes where the model failed. In many of them the label was wrong, not the model. So I fixed many labels and trained again, until the results were good enough.

   In the end, I had about 11,700 labeled notes in Spanish and English. I did not train with other languages. The model is multilingual, so I hope it can also work in other languages, but I have not measured it yet.

3. The model. I fine-tuned XLM-RoBERTa large, a multilingual model, with these notes. Category macro-F1 is 0.78 and theme macro-F1 is 0.83.

The model is not perfect. It still makes mistakes, mostly with “other wrong data” and “not mappable”.

For ML people: model and results

• Input: the model reads the full text of the first comment. I only remove things that do not help, like URLs, dates, hashtags and app versions. About 31% of the notes come from apps (StreetComplete, Organic Maps, MAPS.ME…). The rest come from the OSM website.
• Models: I trained one model for category and one for theme. I compared three options: a simple TF-IDF + logistic regression baseline, mDeBERTa-v3 base, and XLM-RoBERTa large (XLM-R).
• Split: 72% for training, 8% for validation (to pick the best checkpoint and stop early), and 20% for testing, stratified by class. I only used the test notes at the end. The main score is macro-F1, so rare classes like spam count as much as common ones.

Labeled notes per class (click a class to see it on the map):

Category	Meaning	Labeled notes
missing_place	A place exists but is not on the map	3,706
wrong_road	A road or path is missing or wrong	1,933
other_data	Other wrong data: hours, address, type…	1,843
place_changed	A place closed, moved or changed	1,367
wrong_name	Wrong or missing name	994
not_mappable	Opinions, questions, things not for OSM	785
spam	Ads or unrelated links	200
empty	Says nothing: “test”, “ok”	192
personal_data	Personal data (moderators only)	148

Theme	Labeled notes	Theme	Labeled notes
roads	3,712	shop	952
none	1,379	food	579
services	1,285	nature	524
leisure	1,279	transport	329
buildings	1,224	education	301
		health	194

The classes are not balanced. missing_place has 3,706 notes, but personal_data has only 148. I also labeled 590 notes as unsure, and I did not use them for category.

XLM-R large gave the best results: category macro-F1 0.78 and theme macro-F1 0.83. For category, it works a bit better in English (0.80) than in Spanish (0.74).

The confidence score helps. If we keep only predictions with 90% confidence or more, category accuracy goes up from 82% to 90%, and we still keep 81% of the notes.

One thing to keep in mind: most test labels come from AI models. So these scores show how well the model agrees with the AI labels, not with people.

A website to see the results and close notes

• Running the model without a GPU

I did not want to pay for a GPU server. So after training, I exported both models to ONNX with 8-bit weights. Now they are 4 times smaller and run on a normal CPU. They give the same answer as the GPU model for 96% of the test notes.

• The website

The website is simple and looks a bit like OSMCha. You can list notes by category, theme and confidence, and filter by language and app. I hope this makes it easier to find notes you can fix and close. You can also export the notes as GeoJSON for MapRoulette.

You can log in with your OSM account. Then you can tell the model if a prediction is right or wrong (👍 or 👎), and help label notes.

Some notes have personal data, like “this is my house” or a phone number. I did not want to show them on a public map. So the public map only shows how many there are, and only OSM moderators can see them after they log in.

Next step

• Use closed notes. The closing comment often says what the mapper did, like “added the shop” or “spam”. This gives a label confirmed by a real person. The model will still read only the first comment, but these labels can help me train and test it in more languages.

• Use the work of Mateusz Konieczny - bot. He used an LLM to find notes that can be closed (spam, junk, personal data, duplicates) and closed many of them. These notes were checked by a person, so they are good training data, especially for rare classes like spam and personal_data.

Feedback welcome

I hope this helps people solve notes faster. I would love to hear from you:

• Are these categories useful? Would you change any?
• Does the map help in your area?
• You can also help: log in and press 👍 or 👎 on a note, or label notes at https://osm-notes.rub21.com/label.

Note: this runs on my own small server, so the website may be slow sometimes.