Looks like my diary here is going to turn into a development blog for my note viewer, which I’m going to call note-viewer in this post. I haven’t came up with a fancier name because I wasn’t sure what this thing was supposed to do other than being an alternative to user’s notes pages. Now I’m adding features that are not strictly for viewing notes. After all, you don’t want to just view notes, you want to do something about them.

Editing the notes along with the data

One of the things that you can’t currently do from my note viewer is to manipulate notes, that is you can’t comment or close them. You have to do those things either on osm note pages or in editors. What you can do is to go to a note page by clicking its id in the table, then you can do anything with the note that the osm website permits. It may not be very convenient though because you have to work with that note in isolation, without seeing related notes and other osm data. This lack of note editing functionality is there because I wanted first to implement a replacement for user’s notes pages. You can’t edit notes from those pages either.

But of course you want to edit the notes, usually along with related data. A note may tell about an error in the data, you edit the data, then you close the note. After that you move to another related note. Related notes are the ones you selected with note-viewer based on some criteria. That could be your own notes for some period of time made about some kind of objects. An example of such kind of use is described in my earlier diary entry. Another kind of note manipulation would be mass commenting/closing/reopening of selected notes, but the tool is not yet ready for that and we won’t look at this usage here.

So we want to edit selected notes alongside the data. For this we of course need an editor. We want to load a bunch of notes in our editor to make them visible and editable. Also we don’t want other unrelated notes to be shown in the editor because we don’t want to be distracted by them. It means that we can’t just download all notes in some area, if that is good enough we wouldn’t need note-viewer to select the notes. The editor has to be able to receive the notes from note-viewer. And the editor is better be able to edit the notes too because we want to close them after editing the data.

Luckily we have such editor. It’s JOSM of course. It has a remote control feature that can be used to receive data from other applications and note-viewer makes use of that. To do this kind of data-and-note editing first you need to enable remote control in JOSM’s options. Then you need to load notes in note-viewer and select the ones you want to edit by ticking their checkboxes. You can tick a range of checkboxes by clicking while holding Shift. Then in the tools panel below the table press RC > Load selected notes button. Now you can switch to JOSM, load the data around the notes (without loading along more notes), edit the data, close the notes and upload the results. Don’t forget to upload the note changes, in JOSM this is done separately, with a button in the notes window.

That was JOSM, the proper editor. But what if we’re using a computer without JOSM or any other editor installed? Note-viewer doesn’t require installation and works fully in a browser with a web server required only to serve static files. (Actually you can kind of install it by copying the files to your local filesystem and opening index.html with a browser) I’m talking about being able to use iD of course. Can we load selected notes in iD? Yes, to some extent. In the rest of this diary post I’ll go over my attempts to make it work.

Making notes visible

First of all, how do we make notes visible at all? On the OSM website there’s a note layer enabled by Layers > Map Notes checkbox. It adds &layers=N to the current url. So if we want to make an osm url for a map with notes visible, this one will do the job. We won’t have any control over the age of notes after being closed but let’s ignore that for now. Since iD is integrated into the website we can hope that iD also shows notes when launched from a map with notes. But it doesn’t work: iD launches without notes being visible. To make notes visible we have to enable Map Data > Data Layers > OpenStreetMap Notes in the right-side menu. That’s not a big deal, we can tick a checkbox once. But that of course is not our end goal.

Can we make one specific note visible? Under the hood, the remote control feature passes notes to JOSM one by one, so that might be our way to successful note loading implementation. On the website you can have a link and a webpage to a element like this: https://www.openstreetmap.org/node/id. When you’re on the webpage, the edit button contains a reference to the element like this: https://www.openstreetmap.org/edit?node=id. (All /edit links in this post assume that you have iD set as a default editor in your osm preferences.) Opening this link launches iD and selects the element for editing. Maybe there’s a similar link for notes? Unfortunately, as you can see, going to a note page (https://www.openstreetmap.org/note/id) doesn’t make the edit button to contain any reference to the note. But we’ll try anyway by constructing the link ourselves: https://www.openstreetmap.org/edit?note=id. And we’ll find out that the note is not going to get shown but the map inside iD pans to the note’s location. If we enable the notes layer after opening the link, we’ll see the note in the center of the screen. This works even if the link doesn’t contain the #map= part referencing the location. But none of this helps us much because the note is not highlighted in any way inside iD and we can’t add another note to a would-be highlight. We want to highlight several notes at once.

While we’re poking edit urls, it’s worth noting that there’s an API documentation for iD. It doesn’t mention notes though, so we made a discovery of an undocumented feature. It didn’t help us and now we need to look for something else. If there’s no way to tell iD from outside to display a selection of notes, maybe we can tell it to display something else at the locations of the notes? There’s a gpx parameter, how about we put gpx waypoints in place of notes and make iD show us the gpx data? We can put links to notes inside the waypoints or we can just display the note layer and pay attention only to the notes coinciding with the waypoints.

The gpx parameter

Let’s try to open some gpx data that already exists. There’s a link on the Notes/Advanced_use wiki page that lets us download notes inside some area as gpx. We’ll urlencode it and put it in the gpx parameter like this. What do we get? 500 Internal Server Error: Application error… Maybe the data is wrong? We can go to Map Data > Data Layers > Custom Map Data in iD, click ... to edit custom data settings and enter the url there. Now the waypoints-in-place-of-notes are loaded. We can load the gpx data from a file too, so if everything else fails, we’ll have to export the selected notes from note-viewer into a file and load this file in iD.

Can we pass anything to the gpx parameter without the site crashing? Empty value https://www.openstreetmap.org/edit?gpx= crashes, adding editor=id like in the documentation still crashes, not urlencoding the gpx url like in the documentation again crashes. If we can’t open iD with a gpx parameter, we may try to automate away the save-to-file-open-file bit in some other way. But we’re probably using the gpx parameter wrong. Remember the undocumented note parameter? It’s actually an an osm website feature, implemented here. And we see the gpx parameter nearby. Doesn’t it accept a track id like note accepts a note id? Yes, it does, it’s an id of a track uploaded to osm like the ones you can find on Public GPS Traces page. That’s not what we want, we’re not going to upload our fake track just to see it in iD. So gpx parameter won’t help us.

That was osm’s gpx parameter. However, standalone iD has its own gpx parameter. We found out that osm’s gpx is not doing what the iD’s API documentation says. Maybe we should use the iD’s one, if we can get to it. Standalone gpx parameter should be passed through the hash part of the url. Let’s try passing something simpler through the hash, for example let’s pre-fill the changeset comment like this. If you make an edit and press upload, you’ll see our “testing123123” comment. So passing parameters through hash works. Can we pass our original gpx link through the hash? Yes, we can.

Now that we figured out how to launch iD with gpx data displayed, we can try to avoid exporting notes to a file. We’d like to open gpx in iD like we open notes in JOSM with one click of a button. For this we need to generate an url with gpx data that iD will accept. We can construct a data url, data:text/xml;… url with gpx data encoded. That of course is not going to work because security. Same would happen if we paste the url into Custom Map Data dialog. Can we try a blob url constructed with URL.createObjectURL() that I use for gpx export to file? Same result.

Did we lose the fight with the gpx parameter? Do we have to import our note track from a file? If we examine iD’s code, we’ll see that any url we pass eventually goes to fetch(), and if fetch() rejects the url, there’s nothing we can do about it. The fetch call is buried inside npm-style ten-line import which is mostly there to throw an exception when the fetch response is not successful. (Excuse me, it’s actually an eight-line import. Ok, it’s part of a bigger library used for other things.) The exception is then ignored by iD. I actually skipped part of the story above where I passed a damaged link to #gpx and was wondering why nothing happened. You have to look at your browser console for error messages. There you’ll find that iD wants “only URLs with network schemes”. Note-viewer is not a web server, so that’s out of the question. Looks like the gpx parameter won’t help us unless we find a way around Content Security Policy.

Now let’s see what iD accepts as custom data. We tried gpx first because it was the name of the related url parameter. But iD accepts other formats too, and osm api can produce other formats. Let’s go over api formats following the example in the wiki for /api/0.6/notes. Which ones are supported?

• notes or notes.xml - the default xml note format - no
• notes.gpx - yes
• notes.json - yes
• notes.rss - no

We got gpx and json, and, as you can see, they are displayed slightly differently. When displaying json, iD doesn’t render any labels for points. This is likely because geojson feature’s properties object has no standard definition and iD doesn’t know what to render. If we generate our own json, we can put there something that iD likes better. But for now we’ll stick with gpx. Then of course you’ll notice that labels in gpx as produced by osm api are ugly, containing html tags. Again, if we generate our own gpx, we’ll put something nicer there. The difference is that we know where exacly that there is, because gpx waypoints have well-defined name and description tags. There’s also kml support in iD, but osm api doesn’t support it.

Why do we care about osm api support for these formats? If we’re generating our own file with notes, we don’t have to care about the api. These api urls are just for testing. However, if we could convince the api to output exactly the same set of notes that we selected in note-viewer, we could feed that output to iD through #gpx, because it comes from an acceptable protocol and source. We wouldn’t have to generate a file then. Can we do it? The examples above that output note data weren’t very selective. They output all notes that fit some criteria, like being inside a bounding box. But we need a set of notes selected by the user. Basically we need an api call that would accept a list of note ids and output their gpx waypoints. There are only two api calls that can output gpx for a set of notes: bounding box and search. There’s also a feed call, but it’s rss and has no advantages over a bounding box. Neither bounding box nor search accept note id parameters. Which calls do? The usual read one. But it only accepts one id. We’d need something like multi fetch for elements but there isn’t one for notes. We can open only one note like this. For it to be useful we’d need to convince iD to display multiple custom data layers, one for each note. I guess it can’t do that. Looks like yes, we lost our fight with #gpx. See also this question on help.openstreetmap.org.

Drag and drop

What else do we have besides going to Custom Data’s open file dialog? We have drag and drop. We can drag a gpx file and drop it into iD’s tab and iD will render the data. We still have to create a file and then find it in our filesystem with a file manager that supports drag and drop, then actually drag it into iD. That’s slower than one press of a button to transfer notes with remote control. Can we use drag and drop with less ui fiddling? Knowing that there’s HTML Drag and Drop API that lets us transfer various data types by dragging html elements, we may think that this is the way. We won’t need to create any file, we’ll just make some element draggable, put a dragstart event listener on it and set our notes data in the listener with whatever type iD wants to receive: text/xml or maybe application/gpx+xml. It’s implemented in note-viewer now on gpx export button. In addition to pressing it you can drag and drop it. Problem solved?

Nope. For this thing to work, iD needs to receive the data in its expected format. And setting the type won’t help us because iD expects files. If your drag and drop data doesn’t contain a file, it’s ignored. That makes the draggable button almost useless for now. Maybe later iD developers will change their mind and implement receiving draggable data without files, so I left this feature in. One thing you can do is select text/plain as a type, then note-viewer will pretend that it’s sending plaintext, although it’s still going to be xml. But this makes more apps being able to receive the data. For example, you can drag it into html text inputs like <textarea>. That a very limited use, often not even a text editor is able to receive such data. I guess we failed again?

Not quite. We still can drag stuff to iD without leaving our browser for a file manager. Dragging the export button may not help us, we’ll have to press it to create a file. But then usually some download manager pops up. If file is downloaded we can drag it from there. It will also go to our download folder which we’ll have to clean up later. Dragging is easier to do if we have iD opened in another browser window. If iD is in the same window as note-viewer, we have to switch to iD’s tab in the middle of the drag operation. In Firefox we can drag the data over to tab title of iD tab, but don’t drop it, hold it over the title until the browser switches to the tab, then drop it inside iD. This may work differently on other browsers and maybe its easier to go through the Custom Data file dialog. Let’s suppose that we can open our gpx file in some way.

Finally getting to see our notes in iD

Our workflow with iD starts like this: download notes with note-viewer, select some of them, export them to a gpx file, open the file in iD as Custom Map Data. We still don’t see the note pins in iD, but that’s hopefully easy to fix. We can actually do the same thing and load a file in JOSM instead of using its remote control. We won’t see the note pins either but we’ll see buttons next to waypoint markers that let us open associated links. Those links lead to osm note pages. However we won’t see similar links in iD. iD apparently doesn’t know what to do with <link> elements in gpx. Therefore we can’t see the actual notes yet. Instead, we can enable all notes, like we did earlier with Map Data > Data Layers > OpenStreetMap Notes. Now for each note of interest we can see a pink circle marker from gpx Custom Map Data and a note pin inside it from OpenStreetMap Notes. Pins that are not inside circles correspond to notes we didn’t select, which we can ignore. We can click a pin to select its note for commenting or closing. To make sure that a pin corresponds to a selected note we can compare the text label of the pink circle marker with the note comments. By default only the first comment is displayed next to the circle marker. That’s because only the first comment is saved as a gpx waypoint description. We can change that during the gpx export in note-viewer, picking all comments instead of first comment, but the labels may become too long.

What else can we change during the export? There’s a connections option. Since we’re exporting notes to gpx, we can take advantage of the format and pretend that this is some kind of a track. Then we’ll have a line connecting circle markers. Maybe it’s going to be easier for us to go to the next note by following this line. To get the line, change without connections to connected by route or connected by track. Route and track are different connections allowed by the gpx spec. Route actually makes more sense but may be less supported by applications or less convenient to work with. For example, JOSM keeps routes in separate layers, making three layers per file when saved as connected by route. Saving as connected by track will result in two layers. From iD we may not care about this.

What we do care about is seeing all of the note pins. Here lies another problem mentioned earlier: the age of the notes. You know that old closed notes disappear from the map. They don’t actually disappear but most of note viewing tools show resolved notes that were closed no earlier than a week ago. 7 days is the default value for the closed parameter in osm api calls. One exception to this is user’s notes pages https://www.openstreetmap.org/user/username/notes. There all notes are shown. Since note-viewer was developed to be an alternative to these page, it also defaults to showing all notes. You can of course choose not to fetch old notes or not select them for viewing in the editor, but what if you do? Well, you won’t see them, you’ll only see their gpx circle markers, not their note pins. Therefore, you won’t be able to comment or reopen them from iD. For JOSM this doesn’t matter. When JOSM is told to open a note with remote control, the note is going to be displayed no matter what it’s age. But JOSM also has an option to control the age limit when downloading all notes inside an area. You can find it in Edit > Preferences > OSM Server > Max age for closed notes. Is there a similar option in iD? There’s one deep inside iD’s code, but it seems to be unused. With this and with lack of support for waypoint links, its going to be difficult to use iD for reviewing closed notes. Hovewer, that’s likely not the most common use of notes, and when you need it, you can use JOSM instead.

This is an answer to our very first question of this post about iD: yes, we can use note-viewer with iD, but it’s going to be less convenient, especially when working with old notes.