Recent diary entries
Four years ago on May the 25th 2018 the, then new, EU-wide General Data Protection Regulations (GDPR) went in to force. It is therefore quite fitting that nearly to the day 4 years later, the OSMF has given up any pretense of ever fulfilling the legal requirements arising from the regulation and, perhaps more important, undertaking the ethically and morally indicated steps to protect the privacy of our contributors.
It isn’t as if the OSMF was caught by surprise in May 2018. The topic was by far the best researched, documented and planned change in its history. And it isn’t as if the quality of the work or the conclusions were in doubt, completely independent and unaware of what had been prepared by the LWG three years earlier, at last years SOTM there was a talk by Robert Riemann that came essentially to the same conclusions.
But, four years later, with the sole exception of new users having to accept the OSMF terms of service on sign up (added by yours truly), none of the required changes to the APIs and data access have even been started on OSMF properties.
To make the situation worse, OSM isn’t simply static, new services, third party and OSMF operated are being added, existing operations are changed and rarely are the data protection impacts and consequences considered.
Ironically some organizations in OSM-space have taken required steps, for example Geofabrik, OSMcha and Pascal Neis’s HDYC, that the OSMF has not.
Why have we ended in this fraught with legal and financial danger place? On the one hand there are no brownie points to be won with championing the changes to the OSM website, the API and data distribution. Since Frederik has left the board no director has been willing to show any support for the matter.
On the other hand, the technical community is full of data protection flat earthers. Some just believe that data privacy isn’t and shouldn’t be a thing, others believe that OSM has a get out of jail free card in such matters. As a consequence there is both passive resistance to making the changes and at the same time no volunteer developers that are going to code them.
All things given, I don’t believe that without making the matter a priority of the board and tasking an outside organisation any progress is going to be made. But I’m not holding my breath for anything to happen without the police banging on the door.
GDPR on the wiki has links to more material.
As every quarter I’ve updated the contributor statistics in the OSM wiki. The most notable effect is the drop off in new contributors in the 2nd half of 2021, which can at least partly be attributed to the demise of maps.me as a source of new mappers.
This is more obvious when you consider just the new users from this source
Note that the numbers include new users using organic maps, but even that doesn’t change the trend.
Just realized that I didn’t notice that on Tuesday (the 5th) I sent out the 10’000ths welcome message to a contributor.
I’ve been doing this for SOSM since December 2013, nearly 8 years ago, averaging a bit over 100 messages per month.
I’ve discussed our expectations (or rather the absence of them) before see https://www.openstreetmap.org/user/SimonPoole/diary/40071
PS: The happy “winner” was https://www.openstreetmap.org/user/JFischli
As a rule of thumb, the primary name would be the most obvious name of the feature, the one that end users expect data consumers to expose in a label or other interface element.
From the OSM wiki Key:name
The NSI was conceived back in 2013 by Aaron Lidman as a list of canonical name spellings for chains of stores, restaurants and other similar facilities, most notably for use in iD. Vespucci has supported use of the NSI nearly since day one, see a diary post from 2014 (this works quite differently in current Vespucci versions). Since then it has morphed to a, supposedly, authoritative source of tagging for a very wide range of objects. Some would say that the current version has expanded its reach far beyond what is actually useful (flagpoles ffs), but that is not the topic of this post.
Prior to the release of iD 2.20 the NSI had been in a long period of stasis with updates being made to the data, but these were not actually being deployed and given the larger format changes I had all but forgotten about it and the data in Vespucci was really old. However over the last couple of months some colleagues pointed out some weird behavior in Vespucci and iD when using NSI generated presets for tagging. In particular name tags are being added to things like excrement bag dispensers and automated postal package depots.
Now which objects to add a name tag to has a slightly controversial history in OSM, and this conflict predates the NSI. Particularly when tagging facilities (shops, restaurants etc) belonging to chains it can be argued that they shouldn’t have individual names and the brand tag should suffice. On the other hand I don’t think larger parts of the community are actually uncomfortable with adding a name tag to a McDonald’s and I would speculate that this is what in common usage would be considered its name.
However I believe there are numerous things that are commonly not named and that should generally not have name tags automatically added, particularly when they are the same as the brand. Examples would be ATMs, vending machines and similar small mechanical facilities. To determine the scope of the issue I ran a quick analysis on the current NSI contents and this is what came out:
While there are a couple of weird entries, landuse=residential?, that warrant closer inspection, most seem to be inline with with current tagging practice. The exceptions would seem to be the already mentioned ATMs and vending machines, and then for example payment terminals, charging stations, bicycle parking and so on, that commonly shouldn’t have names. Note that the total is one less than the current number of entries in the NSI for reasons that could justify their own post.
But my take on this might just be too strongly influenced by my mainly Western European cultural background and I would be interested in hearing thoughts from all on this matter.
PS: Vespucci 16 programmatically removes some of the name tags before use, but obviously it would be better to have the source data reflect actual best practice.
Today we’ve started rolling out the 1st beta release of 16.0. If you intend to install this, or at a later date the release version, please read the release notes http://vespucci.io/help/en/16.0.0%20Release%20notes/ before installing.
Due to the preparation for targeting Android 11 there have been some changes in where files are stored that you need to know about.
The first beta release of Vespucci 15.2 has just become available on googles play store, alternatively you can download it from releases in Vespucci repository.
While it would always be nice if as many users as possible could test this as fast as possible, this year due to events outside of our control (bintray demise, mapillary sunsetting its current API at extremely short notice, not to mention the usual multiple man weeks of work due to googles requirements), we need to get this release out of the way asap, any help with that would be appreciated.
Dedicated address mode and address handling improvements
When we introduced the “simple mode” in Vespucci 12 in 2019 there was no way to directly add address nodes without switching to the old element creation mode via long press (you could however add a node and then predict the address in the property editor). To make this a bit easier 15.2 introduces a dedicated “Address” mode that switches the “plus”-button to allow adding
- an address node (including automatically adding an entrance tag if the node is on a building outline)
- an address interpolation
- a map note
Further improvements include support for addr:block, addr:conscriptionnumber and addr:block_number in the address tags preference, and the use of information from geocontext project for region specific default address tags.
Support mode specific launcher shortcuts
After adding a short cut for the “help” system in 15.1, we’ve added further short cuts for specific modes: Indoor, Tag-only and Address.
In launchers that support the functionality short cuts can be accessed by a long press on the app icon, and dragged to create short cut links for direct access.
Support merging closed ways/polygons
Basic support for merging two closed ways/polygons has been added to the multi-select mode. If their are no common nodes, or if the function dedects inner rings this will create a multi-polygon, otherwise a single closed way.
If a multi-polygon is created common tags from the outers will be moved to the multi-polygon relation, this is done in a separate undo changeset and can be reverted without undoing the polygon merge.
Better detection and handling of empty relations
Creating empty relation objects is a harmless, but sometimes annoying side effect of other operations. Vespucci has detected these in a number of places prior to this release, 15.2 expands this to cover everywhere this can happen indirectly. Contrary to other editing apps behaviour we do not delete empty relations automatically (for example you may have just emptied one for re-use in a next step), but will prompt you for how to treat them.
To prevent accidental upload of empty relations they are now highlighted by the validator in the upload dialog. Additionally we’ve added an extension to our object search support that allows searching for relations with a specific number of members, see JOSM filter expression parser project.
Support for creating OSM objects from GeoJson Features
Individual geometries in a GeoJSON layer can now be directly converted to OSM elements in the OSM data layer. Attributes of the geometries will be mapped to OSM tags.
Don’t forget that any source used in this way needs to have compatible licence terms and, if you are planning on importing larger number of objects, you need to follow the OSM import guidelines.
Support node icons and flexible labeling in data styling
Previously the icon used to display a tagged node was retrieved from the best matching preset (if any), in 15.2 this can be overridden in the data style, just as the key that is used to label an object (default is the name key). This allows more flexible icon and label selection using the data style hierarchical matching. Any other styling parameters for nodes is ignored outside of the internal node style values that can be set in the style. See Vespucci data styling.
- Show a disambiguation menu when multiple ways could be appended to
- Add preference UI for taginfo server URL
- Add support for ELI meta field and tests
- Add proper multi-polygon support for ELI
- Add support for selecting MultiLineString and GeometryCollection features in the GeoJSON layer
- Support multiple parse error messages from opening hours parser
- Support regular expressions in object search (JOSM filter expressions)
- Support import of JOSM .osn and osmand .ocs files with Notes
- Improve task dialogs behaviour (Notes, OSMOSE etc)
- Keep way selected after rotation
- Suppress display of pre-element-creation state in info dialog
I’ve just updated the contributor statistics for the whole year 2020.
As maps.me has hit the news again, this time in a more negative way, I thought it might interesting to give the influence of the app on new contributors another look.
The original blog post on maps.me can be found here: 2017 blog post.
As can be seen from the data the importance of maps.me as a source of new editors has continued to decrease since we visited the subject the last time. Further noteworthy is that the growth of active contributors on an annual base has recovered from the dip in 2019 and is looking positive again:
Vespucci 15.1 has been available for a couple of days now from googles play store marking the third larger release (14.1, 15.0 and 15.1) this year and our traditional release for the holiday season.
Details on what has changed can be found further down, but first let me point out a few important points from the past year and things to be aware about that will be happening in 2021.
The OSMF and iD
2020 marked the OSMF becoming a participant in the OSM editor market place, you might say what does that concern me? Well, even if not all the developers are competing for funding (like iD, Rapid, StreetComplete and many others), all are competing for relevance. Or put differently: mapper market share. How exactly this will play out going forward isn’t clear yet, but it already has consequences in development, for example I’ve put all tablet related improvements on hold. The required effort just doesn’t makes sense with the OSMF expanding iDs use outside of the classic desktop on to tablets..
More https/SSL woes
A couple of weeks back Let’s Encrypt dropped a bomb. Let’s Encrypt is by far the most popular provider of free encryption keys for https use, and are widely used throughout open source and data, and the OpenStreetMap ecosystems. For example Let’s Encrypt certificates are used on all OSM infrastructure that has so fully migrated to https that use without is no longer possible.
Android 7.1 as the oldest version that will no longer work with Let’s Encyrpt is fairly young, and lots of users are “still”* using it. An estimated 33% of the 2.5 billion Android devices will be effected. So we can’t simply ignore this happening.
On the OSM operations side Tom Hughes has thankfully made some changes that will push back the deadline from January to September 2021. My intention is to have a workaround available that will work for imagery sources and similar in release 15.2 that will be available in January or as soon as we can properly test the change. What is unclear is if there is an actual longer term solution for OAuth (re-)authentication with the OSM API (this uses a system component), however it is still too early to tell.
* just the amount of e-waste that this change will potentially cause would seem to make it far too early to contemplate, however that doesn’t seem to have been a consideration.
Vespucci 15.1 Highlights
The full text can be found on our website 15.1 Release notes.
Note that the version of 15.1 currently available from F-Droid was the first Beta release of 15.1 and can’t be recommended for production use. Signed APKs can however also be found on our github repo
User interface changes and additions
Action modes that require multiple actions are now exited by touching on a floating action button with a check icon instead of via the back button. This effects the way creation mode, creating a relations members, editing a relation and the new create and add to route modes.
All these modes will now further retain state over the Android activity lifecycle.
Way creation mode
All additions are now encapsulated in one undo checkpoint, and can be completely undone by one undo operation once the way is created. The undo button visible in the mode will, as previously, remove the last added node, this will however not add extra checkpoints. Aborting the mode will currently retain the way and any created nodes.
Create relation mode / Add to relation mode
Available from the element (node, way, relation) selection and multi-select modes. Create relation now asks you to select a relation type before proceeding and Add to relation asks you to select a relation type. Elements can be added and removed. Clickable elements are filtered, when available, by preset role filters. If a role filter matches exactly the role will be set for the member.
NEW Add / remove member
Available in the Relation selected mode allows adding and removing members for the currently selected relation.
NEW Create and add to route mode
Available from the Way selected mode. These are specialized modes for building route relations and support auto-splitting of ways. They currently do not support removing route members, that needs to be done via the Relation selected mode. Aborting the modes will not create or modify the relation, however ways that have been split will remain split (as this is done in per way undo checkpoints, individual way splits can be undone if necessary).
Alternative tagging display
We’ve added a facility to display alternative tagging for an object that matches a specific preset with such information. Currently this is available via a menu entry in the property editor.
Removed Help launcher icon
Instead of a launcher icon we now provide a shortcut (long press on the icon) to directly start the help viewer (available on Android 8 and higher.
Better reporting and display of tag issues when merging and splitting ways
Issues occurring when a ways are merged or a way is split are now displayed in a dialog in detail, with the option to investigate them via the element information dialog and potentially fix them in the property editor.
Feedback form behaviour changes
As github no longer supports logins with user name / password this has been removed as an option for the feedback form. If you have a github account you can change a preference to allow issue creation via the normal github web UI. This will be used automatically if you have the github app installed, however note that while the github app catches the URL it doesn’t support documented github API for passing parameters (issue template to use and submitted information). Unluckily there is no reasonable way to submit bug reports to github to get this issue addressed.
I pointed out some of my concerns with the change to the OSMF articles of association and changes to who is to become eligible to a member on the the OSMF mailing list a couple of weeks back, see https://lists.openstreetmap.org/pipermail/osmf-talk/2020-October/007231.html
Those comments were just on the general concepts that were being proposed not the actual text, which we have now received as part of the formal general meeting announcement.
Unluckily my concerns not only persist, they have increased with respect to the proposed change to the OSMFs articles that will allow the board to create committees that include non-board members.
Currently a large part of OSMFs business is carried out by working groups, in particular the OWG running the infrastructure, the CWG communicating on behalf of the OSMF and the LWG handling licence and other intellectual property matters.
The working groups go back to a OSMF board meeting in November 2008, https://docs.google.com/document/d/1OcoOqas1NnOLyhiyFZk12K_BpjhWeXvVU7Nl4MDSGeU/edit?usp=sharing and it is clear that these were established to have topic specific groups in which the board could work on OSMF business with additional participants without having to have the full board present. With other words, working groups have always been board committees with additional members, that this practice wasn’t quite inline with the foundations articles was and has been just a bit of fuzziness among a lot of other issues that the OSMF has had.
Bringing the articles more in line with actual practice is naturally a good thing however it hasn’t been framed that way by the OSMF board, instead the reasoning is that the board needs to have groups that can actually work on behalf of the OSMF, and by extension commit the foundation, for example on human resources related matters. This naturally pulls the proverbial rug out from under the working groups, calling in to doubt every single action and statement they have made over the last dozen years.
Now back in October when I discussed this with Allan, I let the matter lie because there was an obvious quick fix the OSMF could have applied once it had realised what it had done. It could have been made clear that the working groups are such board committees and the AoA change was just an adaption to previous practice. Unluckily since then the actual text of the change includes that a board member needs to be the chair of such a committee, which would upend all the current working group chairs and is incompatible with how that working groups have operated since roughly 2012.
The rule that the chairperson cannot be a board members was created out of the experience of misuse of the WGs and, while it existed, the management board. Essentially by creating a working group and being its representative on the management board, a board member could multiply their weight in decision making by having multiple votes on a matter, when a run of the mill board member would only have one.
My recommendation would be to reject the change as is, and have the board, as this seems to be a high priority item for it, arrange for a vote early 2021 on a correctly framed change with the requirement for a board member to be chair removed.
On to the 2nd controversial item. The board is asking for a mandate to investigate restrictions on who is eligible for OSMF membership. Obviously there is no harm in investigating this, however again I believe this is framed in the wrong fashion. Membership eligibility should follow from the organisations goals, and as a concretization of that its mission statement.
The OSMFs mission statement https://wiki.osmfoundation.org/wiki/Mission_Statement clearly states
The OSMF membership is open to all who want to support the project and participate in the OSMF’s democratic process.
At the time it was a conscious decision to not limit membership only to active participants, but far more leave this as open as possible. Even though this would likely lead to the OSMF being a “Friends of the opera” organisation and not the “Union of the opera singers”*.
The membership can and probably should breach this subject again, but it shouldn’t be framed as a mere administrative detail inline with providing free membership to active contributors and other similar measures, it is a discussion about the fundamentals of the organisation and should be treated as such.
* had to sneak one of my dreaded analogies in there :-)
After a large number of delays, mainly to due to late breaking issues with undocumented behaviour changes and other issues due to the androidx migration, I’ve tagged the 15.0.0 release. Installable APKs should be available from google play and the Vespucci github repository in the next couple of days (not from f-droid which is lagging a quarter of a year currently and is unlikely to fix its build problems any time soon).
More information on the future of the app will be announced later.
End of support for pre Android 4.0 devices
Version 15 no longer supports devices with Android versions older than 4.0 / API 14. We’ve now migrated the Vespucci code base to use the “androidx” libraries to provide backwards compatibility over multiple Android versions and there is no feasible way to continue to use the old support libraries that would be necessary for older Android versions.
Import note: even though the app will work on Android 4.0 devices, devices with Android version prior to 4.1 do not have TLS 1.2 support that is required for accessing the OpenStreetMap API since August 2nd 2020. If you are experiencing authorization issues on Android versions between 4.1 and 4.4 see Can’t (re-)authenticate - TLS 1.0 / 1.1 issues.
Support for array like semantics for multi-selects
In some circumstances, tags with multiple objects have array-like semantics (that is a fixed number of elements) contrary to being variable length lists. A typical example are lane related tags, which contain the number of elements in the lanes (or lanes:forward and lanes:backward) tag. There is no practical difference for tags that need to have a value for every element, but in the case of tags that have free form, potentially empty values, for example destination:lanes, there was previously no way of indicating this in the preset.
The current support will add empty fields if the number of elements is lower than the required number, and highlight surplus ones if there are more than the relevant tag requires.
Switch to JOSM imagery configuration as default
As you may know, Vespucci has used the “Editor Layer Index” as the source of imagery configuration since version 0.9.4 released in March 2014. ELI was one of the few, if not the only, successful, cross editor projects that reduced the amount of work duplicated across editor development, not to mention reducing the effort by community members to contribute. It was used directly by a number of projects and there was even a degree of synchronisation with JOSM.
Unluckily the iD developers have forked their own branch of ELI potentially making ELI longer term untenable. However JOSMs imagery configuration has come a long way since 2014 and now has feature parity with ELI and the JOSM developers have been kind enough to provide their data in an ELI compatible format.
Starting with version 15 we now use JOSMs background layer configuration as the default, manual updates can be pulled from the JOSM repository or you can continue use ELI. As updates replace the whole configuration with the exception of custom layers, this means you can effectively switch back to using ELI if necessary.
Improved layer support
The internal layer support has been rewritten to allow multiple layers of the same kind for layers that support this functionality (background and overlay imagery layers, geojson layer). Layers can be moved up and down in the layer stack. It should be pointed out that adding imagery layers uses a lot of resources and should be used sparingly.
A Mapillary layer can now be added to the layer configuration. Clicking on a Mapillary image marker will open the sequence at that marker in a viewer, the forward - backward buttons will navigate along the sequence. On devices with Android 7.0 the viewer starts in a separate activity that can be shown at the same time as the map display if you add them to a multi/split window view.
Mapillary images are cached on device, the size of the cache (default 100MB) can be set in the advanced preferences.
Support for WMS endpoints
Vespucci now has support for querying WMS servers and adding layers as custom imagery (similar to how this works in JOSM). Additional WMS servers over the pre-configured ones can be added manually. The functionality can be accessed via the layer dialog menus for the background and overlay layers.
- WMS support continues to be limited to services that provide their layers in either EPSG:3957 or EPSG:4326 projections.
- You need to make your own determination, just as with custom imagery sources, if the layer you are using is licensed on terms that are compatible with OpenStreetMap.
Support for barometric sensor and improved elevation support (experimental)
Many Android phones support a pressure sensor that can be used instead of GPS data for determining the current elevation. Support can be turned on in the Advanced preferences / Location settings. This will enable using the barometric height in recorded tracks and when creating nodes from the current GPS location.
Android reports elevation as the height about the WGS94 ellipsoid, this is very different (up to many dozens of meters) from what you typically would expect, that is a height above mean sea level. For this reason starting with this release we no longer record the default Android height information in GPX tracks, to get roughly correct data you can either:
- switch to using Android NMEA output (can be selected in the_Advanced preferences / Location settings_, unluckily this has the tendency to be broken and to be more expensive to process,
- or download correction model data to your device (Install EGM from the tools menu),
in both cases elevation data will then be included in your GPX tracks.
Show current position information
The current WGS84 and related information can now be displayed from the “GPS” menu, the displayed information will automatically update once a second as long as the dialog is shown. As a replacement for the old “create node at current location” functionality such a node can now be created from the dialog.
Styling for validation errors
Every Vespucci user has seen the violet highlighting that indicates that an object may have an issue. Starting with this release the highlighting style can be configured in a limited fashion, see https://github.com/MarcusWolschon/osmeditor4android/blob/master/documentation/docs/tutorials/data_styling.md The default styles use this to highlight potentially out of date objects in orange.
Upload selection and update data functions
It is now possible to upload only the currently selected objects, useful for example if you want to save part of an ongoing larger editing session. Further you can now update already downloaded data in place without effecting unsaved edits.
Improvements built-in photo viewer
On Android 7.0 and later devices the view is run as a separate Android activity, this allows it to be used in split screen or popup window mode together with the main app without having to use an external photo viewer.
- Allow copy/pasting over data loads
- Support for translated data style names
- More consistent keyboard support
- Show number of pending edits to upload on menu button
- Don’t use full screen mode if Android 10 gestures are enabled
- Show warning when so much data has been loaded that the app is unlikely to work correctly
- The snackbar shown on an erroneous long press in simple mode has been replaced by a one time tip display and a beep.
Upgrading from previous versions
- The format of the saved state including data has changed, you should upload any changes before updating.
- Default tile size for some WMS servers has changed, this will lead to distorted imagery for any such tile source for which you already have tiles in the cache. Flush the corresponding tile cache to fix this.
- If you previously had Beta 1 of 15.0 installed you may experience crashes on startup after installing the release version, removing the app specific data via your devices app management user interface should fix this.
- The documentation is out of date.
- Creating issues on github from the app with login / password no longer works. use the pseudo-anonymous issue submission feature.
- For known problems with this build please see our issue tracker
Those that follow the Vespucci twitter account @vespucci_editor will already know, on Sunday the main had TLS 1. and 1.1 support OSM API turned off. This is of a bit wider relevance as older Android devices do not support TLS 1.2 out of the box or at all, and obviously this may effect other apps as well and even such simple things as retrieving map tiles may no longer work when tile caches are updated.
What is older in this context? Well there is no support in any form for devices older than 4.1, introduced in 2012 and no out of the box support, before 4.4.something (likely 4.4.3). 4.4 was released just 6 years ago, so is not that old. Vespucci will enable TLS 1.2 on devices in the range 4.1 - 4.4.? and will work just fine with the OSM API, with two caveats:
- it is possible that there is no support for TLS 1.2 even though there should be, sorry nothing we can do about that.
- while everything for which the network connections are directly initiated by the app should work, components that open their own connections may fail.
Off the top of my head there are only two situations in which the later applies, one not so important is opening issues on github, the other more relevant is authorizing the app via OAuth to use the OSM API. If that happens on your device you should follow the instructions on the Vespucci documentation site and switch to using login and password. Yes, the irony of this is not lost on me.
The deployment of upgraded operating system versions without TLS 1.0 and 1.1 can be followed on Dropping TLSv1 and TLSv1.1 support. If you have any questions or issues not covered above please don’t hesitate to ask.
Obviously version 15 is quite far along at this point and major parts of the work have already been completed. In particular the migration to the AndroidX libraries and re-targeting for Android API 29 have been finished and, as a consequence, support for Android versions before 4.0 has now been completely dropped.
I’m not planning on adding major new functionality in this release, after major internal changes a shake-down period tends to be required due to device specific issues that are only exposed when the app gets actual use. Changing too much just compounds that. There are however a number of small changes (for example indicating the number of pending changes for upload on the main menu bar) and performance improvements, that will make the user experience more pleasant.
Work on the release can be followed in the Vespucci github repository.
While keeping a quarterly minor or major release schedule is a bit of an uphill battle, I believe the work on the next major version of Vespucci has progressed far enough that the first beta will be made available later this month (snapshot development builds are available now).
The fall releases always tend to suffer from porting to the next mandatory Android API/SDK version and this one is not different. I assume that this typically consumes a man-month of development work, this time around it doesn’t seem to be quite so bad however there are still a few issues I haven’t resolved that need to be fixed before release.
The bad news is that this will be the last release that will support an Android 2.3-3.0 compatible build. The reason is that to be able to continue to follow googles minimum target API/SDK rules we will have to switch to a new packaging of the Android libraries that provide backward support for older versions of Android (this implies that the estimated one man month for following google changes for next year is very optimistic). As all the package names have changed it is simply not feasible to use an old support library as we have done up to now for the legacy build with minimal addition version specific code. Undoubtedly it is a side effect of the packaging change that google is quite happy with.
My intent is to revive the legacy build for 4.0-4.3 devices when google drops support for them, which is likely not far away.
Changes in download menu
We’ve now removed the “Download at other location” as the functionality can just as well be obtained by using the find function and then downloading. The Download current view function now merges data, to replace the data in memory you will need to selec Clear and download current view.
These changes makes the behaviour more similar to JOSM and reduces some unnecessary clutter.
Automatically apply preset when property editor is invoked
This is a significant behaviour change relative to earlier versions were you had to apply the preset manually. When the property editor is invoked the best preset will be applied (without optional tags), with other words fields will be shown with empty values for such tags. Buttons to apply the preset without optional tags have been added too.
Automatically applying the preset has some dangers as, depending on the preset, existing values could be overwritten, if that happens a warning will be shown.
The feature can be turned off in the Advanced preferences.
Pan and zoom auto downloading
This release adds support for downloading data and tasks in a more conventional way by retrieving the data covered by the current screen. To keep the amount of data in memory in check a manual facility to prune the data in storage to the screen contents (plus a tolerance) is available via the layer dialog entries for the data and task layers. Additionally the data layer will try to “auto-prune” the stored data once it has reached a configurable number of Nodes in memory Advanced preferences.
The auto-download speed limit preferences will be observed both for data and tasks.
IMPORTANT: this facility should only be used with caution as it has the potential to severely tax both the OpenStreetMap APIs and your device. The former can be reduced by using an offline data source.
Internal photo viewer
The internal viewer is used be default in place of trying to start an external on device gallery or viewer app, it allows deleting and “sharing” of photos directly from Vespucci. When started via a click on a photo icon it will load references to all photos currently in view.
In the current version the photo is shown in a dialog, however we intend to integrate it in a split window view in the future.
The feature can be turned off in the Advanced preferences.
Incremental preset search
THe preset search will now search while you are typing and display the results directly in the preset display. As we integrated the name suggestion index in to the same search in 13.1, the separately available name search dialog in non-simple mode has been removed.
On-the-fly custom presets
A custom preset can be built from existing tags in the “Details” view in the property editor. Select the tags you want to include, then select “Create custom preset” from the menu and then enter a name when you are prompted. The new preset can then be found in the “Auto-preset” group.
The function tries to do the “right thing” by not including the values for tags that have “name” semantics, and setting the current value for combo and multi-select fields as the default value.
“Tip” display function
A function has been added to display tips on certain functions. Every tip will be displayed only once, optional ones can be suppressed by setting an option in the display itself, non-optional ones will always be displayed once.
Support for multi-valued but editable tags
Some tags, for example phone numbers, lane values, allow multiple values that can’t be or are difficult to edit with the previously available forms. 14.0 adds a view for phone numbers and for editable multi-select fields that will display multiple vertically arranged text fields that can be individually edited, and in the case of multi-select fields will have individual drop downs.
Automatic phone number formatting
This is an experimental feature.
When entering a new phone number the number is automatically formated correctly for the country the object is in. Further the same formating is applied when invoking the property editor, a warning will be displayed if this actually happens.
Previously unjoining a way required selecting a node at where you wanted to unjoin ways and resulted in all ways except one receiving new nodes. This makes sense for example when you are trying to rewire a junction, however not so much if you want to “unglue” landuse joined to a road and similar constructs. While this wasn’t a classical use case for Vespucci, but as the app gets used more and more for nearly everything, we’ve now added support for more classical unjoin options.
There are two option when you have selected a way that has shared nodes with other ways:
- Unjoin this will simply replace all nodes in the way with copies of the original nodes.
- Unjoin dissimilar this will replace nodes as above, but additionally replace nodes in similar ways if necessary to maintain connections.
An example of the later function: assume you have landuse glued to a way from both sides and additionally other roads connecting to to it. If you use the basic Unjoin option the way will be disconnected form the landuse but also from all the roads connecting to it, if you use Unjoin dissimilar the connections to the roads will be maintained.
- Selection of OAM imagery has been moved to the layer dialog and imagery configuration removed the preferences with the exception of adding custom imagery.
- Allow Vespucci to consume intents to show photos from other apps. Note: due to the ongoing changes in the way Android controls access to files this may or not may work reasonably.
- If you have left a changeset open after an upload, you can close it before the next upload with an option in the upload dialog.
- Undo and redo checkpoint lists have been moved to separate tabs in the display.
- Go back to last edit function added, this will pan and zoom to the bounding box of the top most undo checkpoint.
- “Squaring” / “Straightening” threshold is now adjustable in the Advanced preferences. The process itself is now done asynchronously and doesn’t lock up the device anymore.
- The preferences can now be changed from within the property editor.
- Multi-select now support copying and cutting multiple objects to the clipboard, and pasting of multiple objects.
- Merging a node in to another node or way will now show a context menu if there are multiple candidate target elements and offer the option of merging with all candidates.
- Lots of stability improvements.
This is a minor release mainly containing UI improvements. You can install the beta from the google play store (after enabling the “beta” releases setting), or obtain it from the Vespucci repository.
Updated opening hours editor
The opening hours editor (a separate project found here) has received a number of updates, including
- tapping the time bars will start a large time picker, this was previously only available via the menu.
- ability to load and save templates to on device files.
- support for region and object type specific templates.
Use of SAF file picker for Android KitKat and later
Previously, to give a consistent look and feel over as many different Android version as possible, we were using a third party file picker/selector for those operations that require a file to be selected for input or output. Unluckily in recent Android versions removable “SD card” storage has become unavailable to the third party library which is rather annoying as this is the location where you typically want to save larger files.
To work around this we are now using the system SAF (Storage Access Framework) file picker for Android KitKat (19) and later, in older Android version the third party file picker should work fine. Using the SAF file picker does have the additional advantage that cloud-based storage location are usable for operations that don’t need direct file access, so you can save and load files from your favourite cloud storage provider if you want to.
Download offline data directly with automatic configuration
To make using offline OSM data in MSF format easier I’ve added a small UI that will display the available files from mapsplit.poole.ch, download them with the Android download manager and configure a new API entry if necessary.
Yes, I know we should be using lavish amounts of corporate newspeak to generate “excitement” among the more gullible here, but frankly I can’t be bothered. We’ve mentioned the work on complete offline support more than enough as is.
Support for multi-line comments and source in the upload dialog, and multi-line text fields in the form tag editing UI
Comment and source field are now multi-line making adding longer texts far easier. Further multi-line input fields are supported in the form based tag editor, depending on the “length” attribute in the presets.
Improvements JS support
The layout of the JS console has been improved and scripts from preset fields are now evaluated after all fields have been set, allowing for order independent consistent behaviour. Further the key - preset mapping is now visible to scripts.
GPX and GeoJson files can be loaded via Intent
Clicking a GPS or GeoJson/Json file in a file manager should now show Vespucci as an app that can read the files.
Go to coordinates function and OLC support
The “GPS” menu now features an item “Go to coordinates” that will display an input field that will accept coordinates in a wide range of formats and Open Location Codes. Further, the element information dialog will display the OLC equivalent of the coordinates for Nodes.
We do make an attempt to resolve shortened OLC codes, but this requires network connectivity and is not guaranteed to give the same results as a google based implementation or even remain stable over time.
Improvements JOSM style remote control
We now support imagery configuration and draft comment and source tags, see Controlling Vespucci from other apps.
- Data style configuration provided that doesn’t render multi-polygons, this may help performance on slow devices.
- Lots of stability improvements.
- Support reading Overpass API generated non-standard OSM XML data from files.
If you remember in Going completely offline with Vespucci last December I gave some insight in to the improved offline support in Vespucci 13.0. Now that version 13 is available, I need to loose a couple of words on how to generate the files.
The original mapsplit application was created back in 2011 by well known OSMer Peter Barth, at the time it was used to prepare input for OSM2World. The slight rework that I started late last year adds support for producing output in MBTile format (instead of copying a couple of 100’000 or so files to your mobile :-)), higher and variable maximum zoom level for the tiles and an optimisation pass.
When generated from OSM source data that has referentially complete ways, the individual tiles are referentially complete too. That means a tile will contain all nodes referenced by ways included in the tile, and all relations that have members in the tile (but not all members of those relations). If input data contains metadata fields (that is OSM id, version and date) the resulting files can be used by OSM editors just as data from the API. Naturally the data will be at least a bit stale and that needs to be taken in to account when using it in an editor.
While you can simply download the current release and generate files yourself, to make life easier for user that just want to quickly try things out, I’m generating some files for some regions on a daily basis on https://mapsplit.poole.ch/. Within reason I can add more if there is interest.
To configure MapSplit files as a data source in Vespucci, see the 13.0 release notes.
Still in the pipeline is support for updating files (best would be on device, but that will need some work).
Some of you may have heard of the upset around the introduction of “scoped storage” in the upcoming Android Q (10) release, see for example androidcentral
So why should you care about this change, after all it promises to enhance users privacy, and it isn’t exactly new that the goog doesn’t really care about burning developers time?*
Well there is a special angle to this new feature for “geospatial” apps, it essentially inhibits direct file access and assumes content can be accessed as a stream of bytes. That is all fine and dandy if you are writing something that just access pictures, videos and other media files, but it breaks down as soon as we are using larger structured files.
Classic example a MBTiles format file containing images or vector tiles or anything else, or a geopackage file and so on. Now there are workarounds of sorts, but they essentially all require copying around potentially multiple GBs of data, potentially duplicating it on device and are not really practical or user friendly.
A further twist is that a user will typically want to persist larger downloads over app de- and reinstalls, data in app private directories doesn’t survive this. For example vespucci creates a separate directory exactly to circumvent this issue.
See https://commonsware.com/blog/2019/03/26/death-external-storage-can-haz-file.html for some more information and https://issuetracker.google.com/issues/128591846#comment50 is a comment from a well known Mapbox developer (in case you are wondering why the later doesn’t turn up when googling, yes, the goog doesn’t actually index its own issue tracker).
Now, after protests, google has essentially pushed back enforcement of this by a year to Android R, but that is just kicking the can down the road a bit.
*In the case of paid devs, well they are paid, the others are simply expected to spend a couple of $1000 per year equivalent amount of time to accommodate whatever google has decided to break this time around,
Yes, 10 years ago was the first documented public commit to Vespucci
This seems to have marked public availability including the wiki a couple of days later.
10 years is methusalem age for Android apps given that the first commercial Android device became available at the end of 2008, Naturally both functionality and code base has developed a lot since the beginnings.
In any case thank you to all the developers that have contributed, lots of them long before I became involved and special thanks to Marcus for his contributions and for continuing to provide the access to the google play store.
Back to current times, version 13 is nearly ready for a beta release that I expect in a couple of days.
Some of the highlights:
Support for multi-polygon, improved area and turn restriction rendering and improved styling configuration
Vespucci will now render multipolygons, the standard style for this and some common area types is to have a thick border on the inside of the polygons similar to other editors. In principle it is possible to use a fill style, however particularly large multipolygons will typically have some non-downloaded members which will lead to rendering artifacts.
Turn-restrictions will be indicated with an icon near the via node or way.
The style configuration has been substantially revamped to allow cascading styles (and by that requiring less typing) and matching on arbitrary tags or tag combinations, further some previously hard wired constants are now exposed for configuration, for example the maximum zoom level at which icons are displayed.
The style format currently remains undocumented outside of the source code and the configuration files themselves
Support for applying osmChange format files
osmChange format files can now be applied to loaded data. This makes the functionality to save such files much more useful, for example potential problems can be inspected and fixed in the saved files and then re-applied without having to use JOSM.
Support for MapSplit tiled OSM data
MapSplit sources contain original OSM data in PBF format tiles. These files can be configured as read-only data sources and be used for editing on the go when there is no expectation of network coverage or there is only slow Internet connectivity available. While it was previously possible to simply load larger areas directly in to memory, this has a larger impact on performance that is avoided by being able to load data on demand.
See this blog post for more information.
Support for reading PBF format OSM data files
Vespucci can now read OSM data from files in PBF format, the most popular binary format for OSM data.
I’ve just updated the contributor statistics for the whole year 2018.
As I pointed out here 2017 update blog post some the graphs look a bit odd due to the one-time effect of an initial large number maps.me users contributing as the editing feature was introduced.
This effect continued to decrease during 2018, but for completeness sake here are the updated graphs from the blog post for 2017 that continue to show continuing growth even without maps.me.
I suspect that the slight decrease in unique active contributors 2018 is due to the decrease in new mappers from maps.me too, in any case the number of mappers that edited at least in one previous year continues to grow, and as that might be the most important metric overall the numbers continue to look positive.
Now that Vespucci 12 is already nearing release (if you are helping with the translations, please have a look at them asap, as they are the major hold up), it’s time to think of what is coming next.
Vespucci has supported unconnected mapping since day one (nearly a decade ago) and that has improved over the years with addition of support for reading and saving files in normal OSM and JOSM xml format (the JOSM format stores information on local changes).
In version 10.1, early this year, we added support for MBTiles files for local on device background imagery sources, supporting building an imagery source while you are online on your desktop and avoiding having to download to Vespucci imagery cache manually which tended to be rather tiresome.
But despite all of this, the main problem remained that you need to keep OSM data files on device reasonably small because the contents would be read in to memory and while such areas can be quite large on a modern phone, things tends do to get slow. And as we all know when you make a selection in advance, Murphy comes in to play and you are surely are going to miss exactly the area that you suddenly stumbled in to.
To get around all of that I’ve been investigating a compact, indexed in one way of the other, on device format for raw OSM data for quite a while, and now have a solution that works even better than what I originally envisioned.
The format is simply a MBTiles format sqlite database containing tiled OSM data in PBF format, nothing special. Some aspects of the generated files are not even particularly refined, for example there is currently only one tile size supported which make it a bit inefficient for empty areas. However it already works well as is, so no need to rush to improve that.
Having all of Switzerland (not exactly an empty country in OSM) will use 433MB, which compares quite well to the original PBF file it was generated from (295MB). Some of the overhead is due to having to duplicate ways crossing tile boundaries and including relations in every tile that has a member element and tiles with little content not compressing well, but overall the size increase is not big enough to cause concern.
To update the tiles it is currently necessary to re-tile the data (after either downloading a new extract or keeping the extract up to date from the diffs). This is not a big deal, generating the tiles for Switzerland from scratch only takes a couple of minutes, but naturally the holy grail is to update in situ on device, doing away with the need for a desktop completely. There are some hurdles that need to be taken before we are there, but it seems to be completely possible. Even if that turns out to be not practical there are some potentially interesting intermediate steps that I’m exploring, for example updating the tiles with the local edits (after they have been uploaded and received definitive IDs).
Some of this functionality is likely to be available in 12.1 which I expect to be available in Q1 2019.
The fist beta/test version of Vespucci 12.0 is now available from the play store (together with the monthly update for the current release, 11.2.3).
Simple action mode
The current way (pre version 12 that is) of adding new objects with a long press goes back to the work done by Jan Shejbal during GSOC 2012 (a couple of years before I had anything to do with the project). This has the advantage that it doesn’t need an extra menu, you can inspect where you are placing the object before actually creating it, and in general that it is fast. The downside is that it can be a bit fiddly at times and users need to be told about it in advance and as we all know RTFM is not very popular.
Simple action mode replaces the long click action on the screen with a menu driven way of creating new objects. Long clicks are disabled as long as the mode is active.
Tapping the large green floating button will show a menu. After you’ve selected one of the items, you will be asked to tap the screen at the location where you want to create the object, pan and zoom continues to work if you need to adjust the map view.
The mode can be toggled on and off via an item in the main menu. See a comparision of the two modes for more information.,
Support for MapRoulette
MapRoulette tasks are now supported in the task overlay. While mostly MapRoulette tends to be revolve around aspects of mapping that can be done remotely, in line with the support for OSM Notes, Osmose bugs and custom tasks, it does seem to advantageous to be able to view and resolve such issues when you are actually on the ground at the location.
To be able to change a tasks status you will need to set you API key. If you have not set it, you will be asked on first upload, preferably you should set the key before making any changes via the “Tools” menu. This is a first cur at support, and may evolve over time.
Notes are movable
Newly created OSM Notes that haven’t been uploaded yet can be moved by selecting the Note and then dragging. Behaviour for downloaded Notes remains unchanged.
Most recently used support for OSM keys, values and roles
This adds support for storing, retrieving and prioritized selection display of keys, values and roles that have been recently used. When possible the mru values are associated with a preset.
Persistent storage is provided by an XML format file (“mrutags.xml”) in the publicly accessible “Vespucci” directory on device. The contents of the file can be inspected, and for example used to indicate what is missing from the presets.
In-app function to provide developer feedback
You now can, by using the “Provide feedback” function in the main menu, directly open a new issue on our github issue tracker with the most important information about your device and app version automatically included. If you have a github account you can simply login with that, if you don’t, you can submit an issue pseudo-anonymously with your OSM display name included. If you are using the later and haven’t yet authorized Vespucci with the OpenStreetMap API, you will be asked to do so.
The anonymous issue creation is only available in the official builds or if you are building the app yourself if you have configured a github personal access key during the build process.
- more layer configuration, specifically for the tasks and grid layers is now accessible from the layers dialog.
- using google login during the authentication process should work again.
- alert behaviour for notification groups is now configurable.
- improved UI and error behaviour when merging objects.
- preset items are now alphabetically sorted by default in the displays, this can be changed in the preset itself.
- Nominatim searches can now be limited to a bounding box.
- in multi-select mode we now show a total element count in the header of the tag editor.
- the build process now uses proguard which results in a roughly 20% smaller APK file.