OpenStreetMap

Diary Entries in English

Recent diary entries

Border Crossings and National Highways

Posted by apm-wa on 18 November 2018 in English (English)

Turkmenistan aspires to renewed status as the "hub" or "crossroads" of Central Asia, harkening back to the days of the Great Silk Road that connected Xian in China to Istanbul, Jerusalem, and Antioch, and from there to Rome and beyond. I thus took it upon myself to check out and ensure proper mapping of as much of Turkmenistan's national highway system as I could, and then to check out its border crossings with neighboring countries. They are now all easily identifiable, both national highways and border crossings, I think properly tagged, and in the case of highways, every gas station I could see has been added to the map (still have a few FIXMEs out there, and there are undoubtedly some gas stations I haven't found yet, but...) So far other mappers and I have mapped 156 gas stations in Turkmenistan, up from the very few on the map in 2015. Turkmenistan is ready for higher volumes of motor traffic and has a better road atlas on line to help navigate!

Maptime Salzburg - November 2018

Posted by bufferclip on 17 November 2018 in English (English)

We started a new meet-up about OpenStreetMap and other geo-related topics. Check our blog post: http://maptime.io/salzburg/event/2018/11/13/Stammtisch/

Location: Lehen, Salzburg, 5020, Austria

The most surreal and memorable OSMF board meeting yet

Posted by imagico on 16 November 2018 in English (English)

Yesterday was the last OSMF board meeting before this year's Annual General Meeting. And like it is already kind of tradition this last meeting had more visitors listening in than any on the previous meetings and IIRC even more than last years pre-election meeting making it probably the largest board meeting in OSMF history.

And this despite the guests having to wait for half an hour because the board started the meeting with a closed session.

And here is where the surreal part started. The closed part was - as you can read on the meeting agenda - meant to determine if two of the board members (Frederik and Heather) had a conflict of interest regarding the subject of the OSMF organized editing policy or guideline as it is now called. This is because apparently two other board members (Mikel and Martijn) were already considered to have a conflict of interest (and after me asking at the end of the meeting confirmed that they recused themselves). Now we don't know the details yet - no word was uttered by the board in the public part of the meeting about the outcome of the closed part of the meeting (did i already mention it was surreal?). We know that apparently Frederik and Heather were not determined to have a conflict of interest because they participated in the vote later. But we don't know who decided this and we don't know who decided that those who had a conflict of interest were allowed to participate in the discussion none the less.

The most amazing thing however is that this discussion and decision if there is a CoI for the various board members happened literally just minutes before the decision on the matter and not years ago when the whole process to develop an OSMF policy was started. And as if to rub in everyone's face that to fully participate in and influence the whole process from within the board despite an obvious conflict of interest is apparently no problem in today's OSMF Mikel essentially kicked off the public part of the meeting (after approval of the minutes from the last one and a few minor other things) by criticizing about the new organized editing policy draft (that i strongly criticized as being whitewashed and essentially mostly free of any hard requirements) that (and i obviously paraphrase here) the data working group should not have the right to sanction corporations for not doing what the policy requires them to do - specifically using as an example the requirement to document their organized activities on permanently available OSMF platforms (the wiki) rather than channels they control and can delete as they like (like github). The audacity of this is simply mind-boggling - both in substance of criticizing the possibility that a mostly toothless policy can potentially be enforced but also in the way of at least from my perspective dropping any pretense to act in the interest of the OpenStreetMap community en large on this matter, which we happen to know through among other things the survey the OSMF made, is to a significant part concerned about organized activities.

Never in the history of the OSMF board meetings - and i have listened in to quite a few of them - have i regretted so much that there is no verbatim transcript of the meetings. As a piece of real-life comedy this was just priceless.

But i am getting carried away. What inevitably followed was the vote on the weakened policy or guideline (it was approved). The punch line was a comment Peter gave with his vote critizing that apparently very recently yet another policy draft was proposed to the board that was apparently written by a mapping company and was discussed by the board. My request after the meeting that this draft be made available to the OSMF members was rejected since the board does not give out details of their internal conversation or documents they receive. But the fact that on a matter of policy making concerning the OSM community the OSMF board discusses specific drafts without making them available to the members or the OSM community is fairly remarkable. The board can obviously not control who sends them drafts for policy documents but they can surely refuse to look at and discuss drafts that cannot be made available to the OSMF members and the OSMF community. That this apparently did not happen is to me a major issue.

I kind of wonder what would happen if someone leaked this ominous draft. Because the only handle agaist making this public would be copyright and this would obviously require someone to claim authorship of said document. And i kind of assume whoever wrote and pitched this to the board does not really want to be associated with it...

After this topic was through everyone was clearly exhausted from the whole performance. The only major other thing discussed was the membership fee waiver program - which was considered to be urgent due to the potential influence of members eligible for voting in the upcoming elections. The discussion of this was quite reasonable and the result fairly acceptable although in total i think the concept of waiving membership fees upon request is not a solution for the overall problem of the lack of proportional representation of the OSM community in the OSMF.

That more or less closed this memorable OSMF board meeting.

Like Heather did in the meeting (she kind of managed to beat me on that) I would like to express thanks to Martijn and Peter for their work on the board - neither of who apparently re-runs in the upcoming elections.

KPNE NE Airport details

Posted by That NE Philly Guy on 15 November 2018 in English (English)

Added details to Phila. NE Airport KPNE from FAA airport chart 00528AD

Location: Ashton Wooden Bridge, Philadelphia, Philadelphia County, Pennsylvania, 19114, USA

Problems with level

Posted by Anton Khorev on 15 November 2018 in English (English)

There are problems with level=* tag stemming from the fact that not all mappers agree on what it means. This problem comes up more often in some geographic areas than in others where different floor numbering traditions exist. In this sense it is like name=* tag where certain parts of a name stop looking like parts of a name after being translated into a different language.

So what are exactly the problems with level=*? Some people think that the value of this tag should be whatever floor number that is used in place. You can usually but not always read it off some plaque, floor plan or elevator button. Other people think that the value should be a number such that the first floor at or above the ground level gets number 0, the floor above it gets 1 and so on. Those two different ways of assigning a value could be exactly the same, especially for English-speaking areas where they have a "ground floor" as level=0 and a "first floor" as level=1. If you are from those places, you may not even understand that there's a problem. If you are from a place like Russia or the US, it's evident for you that those ways are different. I'll be using Russia as an example because it's where I map.

But there should be no debate about the exact definition of level=* tag because we have a wiki where it's perfectly documented, right? Not quite. First of all, we don't have one wiki, we have multiple wikis in different languages. Those languages may correspond to areas with different floor numbering traditions, which in turn affects the wiki contents. Again, this is like name=*, where local wiki editors put their preferred spin on what should and what should not be part of a name.

Even if we stick to English wiki, we still won't avoid the problems with its current contents. You may get different impression on what level=* actually means depending on where you start and where you end reading the page, and also whether you follow the links to certain other pages or not. In fact, since I mentioned other pages, how do you know which wiki page you have to read in the first place? You may think that it's obviously Key:level, but that page disagrees with you, sending you elsewhere for further information. You may also remember building:min_level=* tag which also takes on level as a value. Or is it a different level? Is there a definition of level that exists independently of particular tags?

What can those disagreements lead to? We can look at this from the point of view of a data user and of a data editor. For a user, let's say you want to get to a POI which is inside a building. You look it up in some app, for example OsmAnd, and the app would tell you which floor the POI is on. So you see among other things about the POI something like "Floor: 1". Actually, OsmAnd will say "Level: 1", but that's going to happen if you have English interface. In Russian it's going to be "Этаж" which is literally a floor. At this point we can remember what happens to the wikis in different languages and look at this as an attempt to bolster a particular interpretation of level=* through localization. I'm talking about level tag because the number you see in the app comes from this tag. As a user, you may not know anything about tags and possible differences in their interpretations. Your most likely interpretation of the number you see is going to be that it's a floor number, a designation of a floor inside the building, which is either evident from local traditions or directly written somewhere. You are unlikely to attempt to determine level zero and then count the specified number of levels upwards. In fact, this counting operation may seem unnecessarily complicated and even nonsensical, which looks like a good argument to just use the self-evident floor number as a value of level tag. With this assumption you'll only get the correct floor number if the last editor of level tag on the particular POI shares this opinion. If not, in Russia you'll most likely end one level below the POI. Sometimes it'll be two levels. It's going to happen when "first floor" is slightly below the ground. In some cases the level will coincide with the floor number even under this "counting" interpretation, but you wouldn't know that in advance.

What are the problems for editors? Obviously, different people may want different values for the same tag on the same object. This may lead to edit wars. You may then propose to keep whatever level numbering scheme already exists in a certain building and make everyone follow this scheme. Unfortunately this is not always possible. In order to know what scheme is in place you have to find stuff that is already mapped with level tag and figure out what scheme it corresponds to. There's a chance that no single scheme would fit because different things were mapped by people with different opinions. This is likely to happen if there are many elements tagged with level. On the other hand there could be few and you wouldn't notice them. All this scheme guessing has to be done because nobody indicates what scheme is to be followed at a particular place. Nobody indicates that because according to most mappers there's only one right scheme and other schemes shouldn't exist.

Even if you know the scheme, you can find yourself in a situation where you have to change already existing level values. For example, you may tag all of the POIs along a street in one manner and eventually you get to a place that was mapped differently. Do you stop and change your way starting with this place? Do you go back and retag everything you already have entered? If you have entered a lot and there's only a few things mapped differently, it makes more sense to retag those few things.

Of course having two different interpretations of one tag is inconvenient. We are better off with just one that makes more sense and is easier to use. And we already know which one it is. Just write the local traditional floor number, you can always check it on place, you don't need to figure out where's **level zero, you don't need to count floors, you don't need to follow the rules made up by foreigners who don't know how things work in your country. That's how people who prefer this particular scheme think, however there are reasons to think otherwise.

That other way of thinking goes as follows. We had the definition of level that has 0 assigned to the first level above the ground level for at least seven years. This definition is used in Simple 3D buildings (S3DB) for about as long. All of the software that uses S3DB data relies on this definition. It would be inconvenient to come up with another "indoor" level numbering scheme. It may not even be a numbering because floors can be designated by letters and then you wouldn't know the vertical order of levels without extra data. Even some of software that mainly concerned with indoor visualizations like OpenLevelUp assumes 0-based numbering. So there is only one scheme that works, it's been around for a long time, and it's how it defined in the wiki. That last statement about the wiki is a bit of a wishful thinking done by the proponents of this scheme.

Those were the ideas behind the two major level numbering schemes: locally-defined labeling (it's may not be strictly numbering) and 0-based numbering. As you can see, the proponents of these schemes can be pretty convinced that their scheme is the correct one and they want the level tag for themselves. In addition you can come up with many more schemes. For example, you may want 1-based numbering where you say that those who think that ground floor and first floor are two different floors got their traditions wrong and have to adapt to 1-based counting because all normal people count from one.

One noteworthy scheme lies between the two major ones. A scheme that we'll call locally-based numbering attempts to fix some issues with locally-defined labeling. It uses only numbers that make the vertical level sequence well-defined. For this scheme you replace all non-numeric level labels with numbers. For example, if a building had level K below level 0, like described in this comment, you replace K with -1. Usually it's the same as matching the number sequence ..., -2, -1, 0, 1, 2, ... as close as possible with all numeric labels. In this case the result is usually going to be the same as picking either 0-based or 1-based numbering, whatever fits best.

There's one extra quirk that is skipped levels. Imagine a 100-storey building without 13th floor. The best match with the sequence would involve setting level=2 for 1st floor, level=3 for 2nd and so on until 12th floor. This is not what people typically want to do so it's fixed by extra hack. This hack can be added to any of the numbering schemes mentioned above. It consists of declaring some of levels as non existent and their numbers are thrown out of the sequence before it's matched with numeric labels. It also makes level numbers incompatible with S3DB levels, in case you use it with 0-based numbering.

For places like Russia, no matter what scheme you pick, you lose either compatibility with existing tools or traditional floor numbers, maybe both. Losing traditional numbers is particularly annoying because you have to use numbers that are almost always one off and it's easy to make mistakes because of that. I think that compatibility with tools is the main divide here. Those who use S3DB want to see the visualizations with tools that assume 0-based numbering. After a while they get somewhat comfortable with that scheme and are more likely to stick with it for other things. Those who don't use S3DB see 0-based numbering as something alien.

How prevalent are the problems described here, are they worth your attention? In the place I map, Saint Petersburg, Russia, both main schemes are in use. That means you can't rely on levels displayed by apps like OsmAnd. I've seen levels changed from one scheme to another when next user comes to edit elements with existing level tags. I had to do it myself too, in the situations I described above. Now I mostly avoid the buildings such as malls that are mapped using the other scheme, but eventually I'll have to do something about them. I haven't yet seen full-blown edit wars over level values, however currently there's nothing that would stop one from happening.

What's to be done about it? You can do something about the editing apps, the end-user apps and the tagging conventions with their documentation. That last thing is going to be the subject of another post, but I can tell right away what can be done with the editors even if we don't agree on the exact meaning of level tag. This is for editors with indoor mode such as Vespucci. In this mode you don't have to enter the level values directly, the editor sets them for you. You only have to select the level to be displayed. The level selector can be modified to display some other number instead of the actual level value, or display both actual and altered values alongside each other. The simplest way to specify how to alter the value is to have the level offset option that takes numeric values. For example, with the +1 offset, level=0 would be displayed as 1. A more elaborate options would be required to skip (or insert) certain numbers and replace numbers with letters (or letters with numbers), but the simple offset would solve a lot of problems.

I tried to write this post from neutral point of view without preferring any particular scheme. You can read about my opinion on which scheme is better and which one do I use in my previous post (in Russian). It's quite long, but you only need sections Вертикальная геометрия and Как я маплю. The next thing to be discussed about levels is what exactly the wiki says. I'm going to leave it for another post.

24 hours to rebalance the OSM Foundation's governance

Posted by Stereo on 14 November 2018 in English (English)

The OpenStreetMap project is organized on an international level by the OpenStreetMap Foundation, under British law.

Some companies are paying employees to join, and there are rumors that they give voting instructions.

In recent years, the Board has gradually lost its balance. Over-representation of some areas and groups makes it difficult to maintain a balanced and diversity of government action and raises some conflict of interest issues. Therefore, it is important that all kinds of mappers join in to balance things out.

On December 15 there are board elections, and everyone who was a member before November 15 can vote.

489 ballots were cast during the last election. By way of comparison, in the last few days, 17 employees from a single US company have joined. This might not be anything sinister, but it's worrying especially shortly before the board election, and getting the largest amount of OSMF members is the best safeguard of balance.

https://join.osmfoundation.org/normal-membership is the form. It costs £15 (about €17 or $20), it takes 4 minutes, it's important and it's too late in 24 hours. I am a candidate for the board election, but of course you distribute your votes however you want.

Location: Geesseknäppchen, Hollerich, Luxembourg, Canton Luxembourg, 1265, Luxembourg

Two small Potlatch 2 updates

Posted by Richard on 14 November 2018 in English (English)

Last week there was a small change to the API which means some GPS traces are now returned unordered whereas previously they may have been ordered. This obviously has implications for editors' GPS display. Potlatch 2 now copes with this by doing some basic proximity testing in an attempt to 'reorder' the unordered points, which restores the GPS layer to some degree of sanity.

At the same time, I took the opportunity to fix something that's been bugging me for months. Historically, Potlatch 2 draws all the features crossing the current viewport, and then removes the sprites when they're no longer visible. This is fine except when you pan across a 2000km power line, which takes ages to draw as a dotted line at z19, and bogs down Flash's display performance for evermore. For such large objects, P2 now only draws the on-screen part, which makes it a whole bunch snappier.

Details added to NE Philly Fox Chase section

Posted by That NE Philly Guy on 14 November 2018 in English (English)

I spent some time clarifying the Fox Chase-Newtown branch of the old Reading Company, currently the Fox Chase SEPTA commuter rail branch. It starts at Shady Lane and continues to Country Line Rd. as two separate agreements between SEPTA and Montgomery County. Anticipating a 2019 agreement between SEPTA and the City of Phila. for the section between Rhawn St. and Shady Lane.

Tracing the northern section of the P-9 highway

Posted by apm-wa on 13 November 2018 in English (English)

Ann and I celebrated the Monday holiday of Veterans Day, November 12, by driving out to Hauz Han about three hours east of Ashgabat, heading north on the P-9, then turning east on it to Mary. I had never been on that route and we collected two gas stations plus took lots of ground-level imagery using Mapillary. I will upload that imagery plus imagery collected on the P-25-to-P1 drive when we next come stateside and have access to high speed internet. I have already uploaded the GPX files Mapillary generates, however, so can tweak the highway itself as needed.

I used MAPS.ME to collect some POIs along the way, including marking some new U-turns on the M37. Turkmen Auto Roads State Concern is putting up guardrails to separate new dual carriageways, so mapping the U-turns is an ongoing effort. Some of the village and farmers association names have changed so I have more editing to do, updating names of municipalities to match the signage. Gulanly is now called Gokhan, for example, renamed in honor of one of the sons of the mythical Oguz Han, progenitor of the Turkic peoples. I tried to capture the signage on Mapillary.

We stopped for lunch at the Lebap Cafe in Hauz Han, to eat fresh fish (grass carp) deep fried in cottonseed oil. If you are in the neighborhood, I recommend the Lebap Cafe!

Location: Ahal Region, Turkmenistan

A tool to display Wikimedia Commons categories with coordinates on the OSM map

Posted by Alex-7 on 12 November 2018 in English (English)

This tool can display on the OSM map as geo-markers:

  • Wikipedia articles which have coordinates for all language versions of Wikipedia

  • wikidata, wikipedia, and wikimedia tags from the OSM map database

  • Wikidata items which have coordinates

And now I added the possibility to display also the Wikimedia Commons categories which have got the geographical coordinates.

The tool is available via this link: http://ausleuchtung.ch/geo_wiki/

I created it for my personal needs, for my surveys, but, perhaps, it could be useful to others.

The tool is as simple as it could possibly be. Just click on the map and the geo-markers will appear on the map up to 10 km around the click.

In areas with many categories it makes sense to reduce the radius of the search, because the maximum number of displayed categories I limited to 100.

P.S. Only categories which are connected to a Wikidata item or Wikipedia article are displayed. I do not know how to display absolutely all categories with coordinates via MediaWiki API, or if it is possible at all. On the other hand, the categories which are not shown are usually the sub-categories of those main ones which are displayed.

But it is possible to add the tag wikimedia_commons=Category:* to an OSM object, and then it will be displayed via search in the OSM database (the 2nd option).

MapRoulette 3.10 Release Notes

Posted by mvexel on 12 November 2018 in English (English)

MapRoulette 3.10 is now available on maproulette.org. Here's what's new!

For Mappers

We made changes to the map layer selection. You can select more and more relevant background map layers in MapRoulette. The available layers are retrieved from the OSM Editor Layer Index. For aerial imagery layers, you can also select the Mapbox road overlay.

When browsing for Challenges, you can now sort the results different ways: by age, by popularity and 'smart' which also takes into account Featured Challenges and Challenges you saved. The old sorting by name is also still available.

Smaller improvements include support for Level0 as a preferred editor, more keyboard shortcuts and seeing the age of tasks when browsing Challenges.

For Challenge Owners

You can now use mustache tags in your Challenge description. These are placeholders for values that are unique for each task. For example, if you have a value lanes in your source task GeoJSON, you can add something like "We think this road has {{lanes}} lanes. Can you verify this using imagery and update OSM as needed?". When a task is shown to the mapper, the tag will be replaced with the appropriate value.

You can now organize the Challenge and Project administration screens the way you like them. All components are widgets you can drag, remove, and add. You can even create different views to see different sets of information at a glance.

We also support a new streamable GeoJSON format and give the option to ignore GeoJSON errors.

For the full list of features, bug fixes and changes see the release notes on Github.

Happy mapping!

Opensnowmap style reload

Posted by yvecai on 12 November 2018 in English (English)

I've figured out recently that opensnowmap.org base map (topo) renders underground water pipes like plain rivers. It will need a database reload to fix this. Certainly I can correct other glitches at the same time, so have you spotted or can you find something else really wrong with this style? I'm mainly interested in mountainous places, of course, don't expect fixes in Manhattan subway lines ;-) The style is derived from OSM-bright. Thanks in advance, Yves

So how does the Facebook's AI Assisted Road Import Process work?

Posted by Jeff Underwood on 12 November 2018 in English (English)

Although the team has shared the process in depth at various conferences and discussion post and the wiki, we have gotten feedback that we could clear up some of the processes by providing more details. So here it is.

At Facebook, we use our ML (machine learned) data as a starting point but then completely re-edit, fill in the gaps, and correct errors with human editing. Machine learning can do amazing things, but even the best predicted area will always have issues that need a human to fix. An untouched ML file will often have disconnected ways, gaps in the network, road type inconsistencies, and geometry issues. The job of our human editors is to use this ML data as a base and build fully fleshed out, high quality data.

The Machine Learned XML file

This is the base file we work with. Our engineers take a snapshot of OSM data at the time of XML creation. They then merge our predicted roads onto this OSM data to produce the Machine XML file. From this starting point, the editor will cleanup and expand upon the ML data. A limitation that will be immediately apparent to some, is that by necessity, our ML data is merged with a snapshot rather than live OSM. This means that if we take too long to map it, the data could gain conflicts or someone could map out our predicted area. Luckily, our FB iD has some features that reduce the impact of these problems, which I will go into later on below. image_1 A typical Machine file. There are some breaks in generated roads and lints in pink.

Creating a Machine File image_21 A snapshot of OSM data is taken. Here you can see the town has been mostly digitized but many roads to the South are missing.

image_20 Generate a road prediction for the area. All the roads are predicted, even those that exist in OSM already.

image_22 Conflate the machine learned data with live OSM. Roads that already exist will be dropped from the ML output.

image_19 Final Machine File result. predicted roads that do not exist in OSM already are created. Predictions outside of the task bounds are included in a different XML.

Lints

Fresh out of the oven, our ML XMLs will typically contain several automatically applied validation checks that must be resolved and removed before our version of iD will even allow the file to uploaded. This sort of automated error flagging is called linting in software development.

Let's go into excruciating detail!

lint_disconnected When a road or cluster of roads does not connect to the greater road network they will get tagged with this lint. Once one of these roads is connected to the network, the tag is automatically removed from all the ways. Our policy is to either connect or delete as we don't want to create a bunch of roads divorced from the rest of the map.

lint_disconnected A typical lint_disconnected

lint_connectExtend This lint is essentially the "Way end node near other highway" check in JOSM. If one of our generated roads ends near another highway the machine will generate this to let our editors know that a connection might be possible here. These missed connections are often from obscuring trees or buildings and are typically confirmable with alternate imagery. Of course, some times the lack of connection is appropriate and in these instances, the way is evaluated and the tag just deleted.

lint_connectextend A typical lint_connectExtend

lint_crossWaterway To speed up the editing process, the machine file will automatically split and add a bridge tag when one of our roads crosses a waterway. This lint tag prompts the editor to first check the validity of the water feature as many are poorly digitized or difficult to see in satellite, then to adjust the automatically created bridge segment to the actual size of the bridge in imagery.

lint_crosswaterway a typical lint_crossWaterway

SplitPoints and lint_autoconnects When the ML data is being gridded up into tasks, generated roads that travel outside the bounds of a task are automatically split at a node and a tag is applied to tell the editor not to move this particular point. When it comes time to publish the data, if the other half of the split point road has already been uploaded in the neighbor task, our iD will automatically load the adjacent road and connect the two halves of the road together. When this happens, a lint_autoconnect tag is applied which cues the editor to check the connection for correctness and the highway types for consistency.

image_2 The red points at the end of the streets are SplitPoints. Our roads should automatically connect between tasks where these are present.

lint_autoconnect Example of an autoconnect. Notice that the adjoining road is also one of ours.

FB ID

Our engineers have created a more advanced version of iD that has been fully customized for our purposes. The general design philosophy was to add some of the great features JOSM has while retaining the simplicity of iD.

image_3 FB iD

Load Live Data

Despite our XMLs being based on OSM data from the date we generated them, our iD has a feature to automatically pull the latest OSM data and update any preexisting road in the task area. Using this, we avoid experiencing almost all conflicts as our data is fresh each time we open our task area. If any duplicates pop up, like when a road we predicted has been digitized by someone else, these will be loaded and typically caught by our validator with a crossing ways warning.

image_4 The loading live data dialogue

Validation

Our version of iD has a number of handy enhancements built in to make it a much more powerful tool than the official version. Most useful to the community at large is our built in validator. This has many of the same checks that JOSM uses, such as overlapping ways and duplicate nodes, and helps prevent errors or poor quality data from ever reaching the live map. Like JOSM, we show a list of validation checks categorized into errors and warnings with errors blocking data upload until they are resolved. Our lint checks are also picked up by this validator as errors and also will also block submission.

image_5 Users can select and cycle through the errors using the arrow buttons. The way involved is highlighted and the error type is displayed in the corner

Styling

In our iD we display all of our ML roads in green, or pink if there is a lint tag. We restyled most road types from default iD to make them distinct in various ways since color alone was no longer an option.

image_6 The style sheet for ML data.

image_7 The style sheet for pre existing data.

Additionally, we developed two style toggles to aid our editing process. The first will turn roads bright blue if they have been touched by editor. This helps the editor track their progress as they work the task.

g_key Example of “edited roads” style

The second will randomly color our roads in order to show ways that could be better merged or split. Affectionately known as rainbow roads.

rainbow_roads Example of random color or “rainbow roads“ styles.

Also in the realm of styling, we created a grid overlay to break the task down into smaller pieces to also aid in editing. This can be set in whatever size is most comfortable for the user or particular task.

grid Showcase of the various grid options

Our ML prediction layer is also available as a toggle-able overlay. This can be handy while reviewing to ensure that the editor did not trim out any data we consider high value. You might also notice that we can fully turn off the data layer, rather than just wireframes. Another really handy feature.

ml_layer Example showing the ML layer

Workflows

Getting from the Machine file to data onto the map is a multistep process for us. Everything must go through three stages of editing before it makes it gets uploaded and afterward we do an additional round of cleanup. image_8 The workflow stages

Editing Workflow

An editor will typically start in a corner and work their way along a task, going through ML roads one by one. They will check the generated highway types for correctness and change as necessary, improve the geometry or connections of the way, and, of course, evaluate if the road is even worth keeping. Marginal tracks are often discarded due to low impact for the amount of work needed to complete them.

image_9 A task before editing

image_10 and after

As a team, we are very cautious when changing community data as people are understandably protective of their hard work. People on the ground will have more context than we ever could as well. In iD, we actually disable deleting or splitting preexisting roads completely. When we do have to make alterations, an editor will leave a lint_fixme explaining the needed fix, typically road type changes. The reviewer will then evaluate if the change is warranted and make it themselves.

image_11 Community road with a lint_fixme saying it should be upgraded to unclassified

Once the geometry, tagging, and visible lints are cleaned up, the editor will run through the errors and warning on the validation panel for anything they might have missed.

validation_panle A small lint_disconnected was missed in the initial editing but immediately obvious with the panel

Lastly, an editor will zoom out and take a look at the highway tagging structure. Does it look correct? Does it follow a logical hierarchy? Only once they are satisfied do they click save to FB backend and mark the task as done. At this point, no data has been added to the live map.

image_12 Marking done on our internal OSM tasking manager

Reviewing Workflow

Reviewers follow a very similar procedure to editors. They also work grid by grid and check all the work of the editor. Additionally, they will make any changes needed to pre existing data that the editor requested.

review_fixmes The editor requested this community road be upgraded to an unclassified. In this case, the reviewer agreed and made the change.

When they do find issues they drop a lint_review tag and leave a note detailing what is wrong. These can be for incorrect tagging, missed or incorrect connections, or any number of issues.

lint_review The geometry here is not satisfactory so the reviewer left a lint_review with the note improve geometry

If the reviewer is not satisfied with the quality of work, the task is sent back to the editor for additional editing.

image_13 Sending back in our internal OSM Tasking Manager

If everything looks good, then the task is approved and we're on to publishing.

image_14 Approving in our internal OSM Tasking Manager

Publishing Workflow

Once a task has been approved by a reviewer, it is ready to be published onto OSM. The publisher will go through and cleanup any remaining lint_fixmes and make note of what needs fixing in JOSM after submission. They then click "Publish to OSM", add a changeset comment with our hashtag "#nsroadimport #thailand" and a comment saying what types of roads have been added and noting if any pre existing data has been altered.

image_15 a typical changeset comment

Our engineers have built a more robust conflict handler into iD so when conflicts do arise we can handle them in a painless manner. Rather than simply having to choose between my version or theirs we have the option to keep both sets of edits and intelligently merge the changes.

image_16 The enhanced conflict screen. Keep both is almost always the best choice.

Post-submission Cleanup

After uploading the ML data in iD, the publisher will then open the live OSM data in JOSM to do a final validation check and cleanup the borders of a task area. This part happens in JOSM since it has more robust validation tools than even our FB iD and we want to catch everything we can. We make an effort to combine our ways along task edges and to add consistency to our road tagging. Additionally, if there was any further work that had to be done to preexisting data, such as splitting a community road, we do it during this step.

image_17 A task opened in JOSM. We have a custom map paint style that looks similar to our iD. In addition to the data we overlay the project grid on top so that task bounds are clearly defined.

cleanup Showing the before and after in a typical task in JOSM. Notice the yellow intersections disappearing as roads along the border get merged. In the Southeast corner, two roads are extended to complete them.

Full Project Check

After a project is fully uploaded, we do a final cleanup of the submitted data. We load up an entire project grid in JOSM then download the data in the area. This is very similar to the previous JOSM cleanup step but at a grander scale. Some road type decisions are much easier to make once an entire area is present so this is an important step in making our data cohesive and correct. Again, we check task borders for any connections that need to be merged or were missed and run validation on the entire area.

image_19 An entire project loaded up in JOSM

Rinse and Repeat a Few Thousand Times

And that's how we do it. I hope this helps to make our process clearer. Please feel free to reach out if you want anything further clarified. We're happy to share :)

Glossary of terms

  • Machine learning (ML): a field of computer science that uses statistical techniques to give computer systems the ability to "learn" (e.g., progressively improve performance on a specific task) with data, without being explicitly programmed. For our purposes, this means teaching a computer to recognize roads from satellite imagery.
  • ML prediction: This is the output of our machine learning. In its raw form its a black and white image showing where the machine thinks roads exist. Our engineers run scripts to process this and turn it into our Machine XML files.
  • ML roads: OSM roads created from the ML prediction. AKA predicted roads
  • Linting: The process of running a program that will analyse code for potential errors AKA engineering speak for validation
  • XML: The file format OSM data is stored in. Often times saved with the .OSM extension though.
  • Machine Learned (ML) XML: a snapshot of OSM data that we merge our ML predicted roads with
  • JOSM: An advanced OpenStreetMap editing application that we use for validation.
  • FB iD: More accessible OpenStreetMap editing tool (that we have customized and use primarily).
  • OSM Tasking Manager: a tool for gridding out areas for mapping. Our internal version is based on OSMTM 2 while the current HOT OSMTM is the newer version 3
  • Community data: OSM data created by someone outside of Facebook .

Helpful links

Import discussion

Import Wiki

Thailand Discussions

Indonesia Collaboration with Local OSM Community and HOT

OSM community – you're great!

Posted by tyr_asd on 11 November 2018 in English (English)

Normally, I blog here more about technical topics, but today I would like to take the opportunity to honor three small parts of the OSM community, who I happened to have the chance to get to know over the years that I've been active in mapping. All of these groups deserve a shout-out for their great work they do on OpenStreetMap month after month.

  • The OSM community in Graz (Austria) managed to map their city in great detail and keeps it complete and up to date. It was a pleasure to be a part of the monthly meetups back when I studied there. I remember many fruitful discussions over good food an a nice beer or two. Keep up the good work, guys!
  • Geofabrik is a company from Karlsruhe (Germany) which provides OSM related services since many years, some of which are available for free to the OSM community. Thank you for maintaining your OSM data extract download service, and for hosting your iconic hack weekends. You're awesome!
  • The disastermappers heidelberg are a group of students of Heidelberg University who regularly organize mapathons and workshops, and try to raise awareness of the benefits OpenStreetMap data towards a wide audience. Don't stop educating the mappers of tomorrow!

Do you also know groups or individuals that do great work in OSM? Let them know! :)

Update of the P-1 and P-25

Posted by apm-wa on 11 November 2018 in English (English)

Much more of the P-1 national highway in Turkmenistan is now a dual carriageway divided highway, so the last couple of evenings I've been updating it based on GPS traces collected and marker tags inserted via MAPS.ME during travel on November 8. It's done, but there will be more to do over time. Pavers were working, and the right of way of the new parallel lane, long neglected, is back in play on much of the P-1 route. All bridges appear to be up, so it's now a matter of bulldozing, grading, graveling, and paving.

Special bonus: We found a new grocery store near our house in Ashgabat today.

Location: Ahal Region, Turkmenistan

Spam is appearing within Changesets

Posted by alexkemp on 10 November 2018 in English (English)

Hard on the heels of my previous Diary (How to report spam) I've discovered spam within a changeset (map edit) for the first time:–

Changeset#6050808957

 addr:city  Hà Nội
 addr:district  Thanh Xuân
 addr:housenumber   25
 description    Video Bài Hát Việt Nam
 email  <redacted>@gmail.com
 name   Ngoc Anh
 phone  <redacted>
 start_date 10/11/2018
 website    https://www.youtube.com/watch?v=<redacted>

Splendid.

osm tag translate

Posted by mdmahir on 10 November 2018 in English (English)

I have created a proof-of-concept to translate tags using overpass api, wikidata query service queries. you can write your own queries and get the data from both wikidata and osm.

we can extend this translation for ways,streets, and to so many other properties. [Text]https://mohamedmahir.github.io/osm-tag-translate

Adding road names from TIGER using MapRoulette

Posted by mvexel on 9 November 2018 in English (English)

TIGER is the street database from the U.S. Census. An old version of TIGER was used to bootstrap OSM in the U.S. back in 2008. We have come a long way since then (see image below) and OSM is now much better than TIGER in most places.

Image from https://wiki.openstreetmap.org/wiki/TIGER_fixup

New versions of TIGER are released every year and they are still useful. Local governments update it with new roads and street names. If you compare new TIGER data with what's in OSM, you get useful information about where OSM may need improving. If you edit in iD, you get visual cues when roads are missing:

You have to stumble upon them, though. And you only see new roads, not if a street that didn't have a name now has one.

My colleagues in the Telenav map team have run OSM and TIGER 2017 through our Cygnus conflation engine to find those streets in OSM that don't have names yet, but TIGER does have a name. We put them in MapRoulette for a few select areas.

MapRoulette is a micro-task tool that I built originally to clean up the redaction mess after the license change. These days, it is available to anyone who wants help to fix all kinds of small problems in OSM. Scroll through my recent diary entries and you will find blog posts about MapRoulette. Right now I just wanted to walk you through solving these TIGER based name adding challenges.

First I select one of the TIGER Challenges in MapRoulette, for example this one for New Orleans.

The first task I get is this one here:

I have three options on the left, Edit (which takes me to my preferred editor, you can set this in your MapRoulette user profile), Not an Issue for if you can already see on the map that there is no issue, or Skip if for whatever reason you'd rather leave this task to somebody else.

Clicking Edit takes me to my preferred editor, JOSM.

And in fact this street segment has no name. The TIGER roads overlay tells me that this is just a part of Lake Trail Drive. I fix the error and upload.

MapRoulette has already taken care of pre-filling a changeset comment and source, but you're of course welcome to tweak these to your liking.

After the upload completes I am ready to switch to MapRoulette in my browser, click I fixed it and go on to the next task.

Finally, here's a list of all TIGER name challenge we have currently. If you want your city or area included, let me know!

Happy Mapping!

Western Michigan University Mapathon

Posted by Kevin Haynes on 9 November 2018 in English (English)

I am pleased to announce that in celebration of Geography Awareness week AND International Education week. The Western Michigan University geography department has organized a mapathon.

Where: Western Michigan University Classroom E in the Swain Education Library

When: Wednesday, Nov. 14 from 5-10 p.m.

Why: Because points, lines, and polygons will change the world!

Food provided by the Geography Department.

If you are a WMU student, this event is part of the WMU Global Engagement Signature Program.
Link: https://wmich.campuslabs.com/engage/event/2816689 for more information

Contact me if you have any questions.

Location: Oakwood, Kalamazoo County, Michigan, 49008, USA

The P-25 and the P-21 national highways

Posted by apm-wa on 8 November 2018 in English (English)

Just got back from 2-1/2 days on the road, during which inter alia I traversed the P-25 and part of the P-21 national highways in Turkmenistan. I collected roughly 38,000 ground-level images with Mapillary, but will delete some before uploading due to quality issues. Got good GPS traces, however. Tried using MAPS.ME to collect POIs but there are issues due to its limited universe of categories. I ended up using "Attraction" as a default tag just to mark POIs geographically (including U-turns on newly divided highways and such), and now am plowing through them at my desktop computer to correct them to something resembling reality and OSM guidelines. I hope this will not garner too much opprobrium from the OSM community. I'm cleaning up the mess I've created as quickly as I can, but I did want to collect data as I was traveling!