OpenStreetMap

Diary Entries in English

Recent diary entries

What next?

Posted by Kilkenni on 6 December 2018 in English (English)

Our case revealed a serious problem. This problem is not only of ignoring international law, but also a problem of lack of transparency in an open community-driven project. The boil was present for a while now, but this case is where its existence became apparent. Data Working Group now is not what it was when it was created, and not everyone is happy with what it turned into. Currently it is a non-transparent OSM body that answers to no one but has serious influence on decision-making inside OSMF, as three out of seven OSMF Board members are also DWG members.

Our indignation and our actions have flushed out this issue into the open. We must, however, remain calm and patient. We should inform OSM community and draw attention to the issue as it is the only way to see it properly resolved.

This text is not mine but I agree. OSM currently lacks rigid rules or firm agreements concerning community, it lacks some sort of social contract. While this approach has certain advantages, it utterly fails in case of internal conflict in the community, and resolving such a conflict is challenging (to say the least).

(careful, language. I would use a spoiler tag if Markdown had one) Troubleshooting or, for English-speaking people, roughly

"System of assumed defaults" has serious benefits for rule-making in small communities as

  • it is fast to build, and

  • it allows jumping straight to business instead of paperwork

However, when such a system breaks causing a conflict, troubleshooting it is a clusterf--k because you'll go nuts before you figure out all the defaults explicitly and find an error to blame.

State of the Map US 2018: OpenStreetMap Data Analysis Workshop

Posted by Jennings Anderson on 5 December 2018 in English (English)

(This is a description of a workshop Seth Fitzsimmons and I put on at State of the Map US 2018 in Detroit, Michigan. Cross-posting from this repository)

Workshop: October 2018

Workshop Abstract

With an overflowing Birds-of-a-Feather session on “OSM Data Analysis” the past few years at State of the Map US, we’d like to leave the nest as a flock. Many SotM-US attendees build and maintain various OSM data analysis systems, many of which have been and will be presented in independent sessions. Further, better analysis systems have yet to be built, and OSM analysis discussions often end with what is left to be built and how it can be done collaboratively. Our goal is to bring the data-analysis back into the discussion through an interactive workshop. Utilizing web-based interactive computation notebooks such as Zeppelin and Jupyter, we will step through the computation and visualization of various OpenStreetMap metrics.

tl;dr:

We skip the messy data-wrangling parts of OSM data analysis by pre-processing a number of datasets with osm-wayback and osmesa. This creates a series of CSV files with editing histories for a variety of US cities which workshop participants can immediately load into example analysis notebooks to quickly visualize OSM edits without ever having to touch raw OSM data.

1. Background

OpenStreetMap is more than an open map of the world: it is the cumulative product of billions of edits by nearly 1M active contributors (and another 4M registered users). Each object on the map can be edited multiple times. Each time the major attributes of an object are changed in OSM, the version number is incremented. To get a general idea of how many major changes exist in the current map, we can count the version numbers for every object in the latest osm-qa-tiles. This isn't every single object in OSM, but includes nearly all roads, POIs, and buildings.

 Histogram of Object Versions from OSM-QA-Tiles

OSM object versions by type. 475M objects in OSM have only been edited once, meaning they were created and haven't been subsequently edited in a major way. However, more than 200M have been edited more than once. Note: Less than 10% of these edits are from bots, or imports.

Furthermore, when a contributor edits the map, the effect that their edit has depends on the type of OSM element that was modified. Moving nodes may also affect the geometry of ways and relations (lines and polygons) without those elements needing to be touched. Thus, a contributor's edits may have an indirect effect elsewhere (we track these as "minor versions"). Conversely, when editing a way or relation's tags, no geometries are modified, so counts within defined geographical boundaries often don't incorporate these edits. Therefore, to better understand the evolution of the map, we need analysis tools that can expose and account for these rich and nuanced editing histories. There are a plethora of community-maintained tools out there to help parse and process the massive OSM database though none of them currently handle the full-history and relationship between every object on the map. Questions such as "how many contributors have been active in this particular area?" are then very difficult to answer at scale. As we should expect, this number also varies drastically around the globe:

 Map of 2015 users Map of areas with more than 10 active contributors in 2015 source. The euro-centric editing focus doesn't surprise us, but this map also shows another area with an unprecedented number of active contributors in 2015: Nepal. This was in response to the April 2015 Nepal Earthquake. This is just one of many examples of the OSM editing history being situational, complex and often difficult to conceptualize at scale.

Putting on a Workshop

The purpose of this workshop was two-fold: first, we wanted to take the OSM data analysis discussion past the "how do we best handle the data?" to actual data analysis. The complicated and often messy editing history of objects in OSM make simply transforming the data into something to be read by common data-science tools an exceedingly difficult task (described next). Second, we hoped that providing such an environment to explore the data would in turn generate more questions around the data: What is it that people want to measure? What are the insightful analytics?

2. Preparing the Data: What is Available?

This was the most hand-wavey part of the workshop, and intentionally so. Seth and I have been tackling the problems of historical OpenStreetMap data representation independently for a few years now. Preparing for this workshop was one of the first times we had a chance to compare some of the numbers produced by OSMesa and OSM-Wayback, the respective full-history analysis infrastructures that we're building. As expected, there were some differences in our results based on howe we count objects and measure history, so this was a fantastic opportunity to sit down and talk through these differences and validate our measures. In short, there are many ways that people can edit the map and it's important to distinguish between the following edit types:

  1. Creating a new object
  2. Slightly editing an existing object's geometry (move the nodes around in a way)
  3. Majorly editing an existing object's geometry (delete or add nodes in a way)
  4. Edit an existing object's attributes (tag changes)
  5. Delete an existing object

All but edit type 2 result in an increase in the version number of the OSM object. This makes identifying the edit easier at the OSM element level because the version number is true to the number of times the object has been edited. Edit type 2, however, a slight change to an object's geometry is a common edit that is often overlooked because it is not reflected in the version number. Moving the corners of a building to "square it up" or correcting a road to align better with aerial imagery are just two examples of edit type 2. We call these changes minor versions. To account for these edits, we add a metadata field to an object called minor version that is 0 for newly created objects and > 0 for any number of minor version changes between a major version. When another major version is created, the minor version is reset to 0.

Quantifying Edits

Each of the above edit types refer to a single map object. In this context, we consider map objects to be OSM objects which have some level of detailed attribute. As opposed to OSM elements (nodes, ways, or relations), an object is the logical representation of a real-world object: road, building, or POI. This is an important distinction to make when talking about OSM data because this is not a 1-1 relationship. OSM elements do not represent map objects. A rectangular building object, for example, is at minimum 5 OSM elements: at least 4 nodes (likely untagged) that define the corners and the way that references these nodes with an attribute of building=*. An edit to any one of these objects is then considered an edit to this building.

This may seem obvious when thinking about editing OpenStreetMap and how the map gets made, but reconstructing this version of OSM editing history from the database is difficult and largely remains an unsolved (unimplemented) problem at the global scale: i.e., there does not yet exist a single (public, production) API end-point to reconstruct the history of any arbitrary object with regards to all 5 types of edits mentioned above.

Working towards such an API, another important infrastructure to mention here is the the ohsome project built with the oshdb. This is another approach to working with OSM full-history data that can ingest full-history files and handle each of these edit types.

Making the data Available

For this workshop then, we pre-computed a number of statistics for various cities that describe the historical OSM editing record at per-edit, per-changeset, and per-user granularities (further described below).

3. Interactive Analysis Environment

Jupyter notebooks allowed us to host a single analysis environment for the workshop such that each participant did not have to install or run any analysis software on their own machines. This saved a lot of time and allowed participants to jump right into analysis. For the workshop, we used a single machine operated by ChameleonCloud.org and funded by the National Science Foundation to host the environment. I hope to provide this type of service again at other conferences or workshops. Please be in touch if you are interested in hosting a similar workshop and I can see if hosting a similar environment for a short duration is possible!

Otherwise, it is possible to recreate the analysis environment locally with the following steps:

  1. Download Jupyter
  2. Clone this repository: jenningsanderson/sotmus-analysis
  3. Run Jupyter and navigate to sotmus-analysis/analysis/ for the notebook examples.

4. Available Notebooks & Datasets

We pre-processed data for a variety of regions with the following resolution:

1. Per User Stats

A comprehensive summary of editing statistics (new buildings, edited buildings, km of new roads, edited roads, number of sidewalks, etc.) see full list here that are totaled for each user active in the area of interest. This dataset is ideal for comparing editing activity among users. Who has edited the most? Who is creating the most buildings? This dataset is great for building leaderboards and getting a general idea of how many users are active in an area and what the distribution of work per user looks like.

2. Per Changeset Stats

The same editing statistics as above (see full list of columns here) but with higher resolution: grouped by the changeset. A changeset is a very logical unit of analysis for looking at the evolution of the map in a given area. Since each changeset can only be from one user, this is the next level of detail from user summaries. Since changeset IDs are sequential, this is a great dataset for time-series analysis. Unfortunately, due to a lack of changeset extracts for the selected regions (time constraints, fun!), OSMesa-generated roll-ups do not include actual timestamps. This caused some confusion for a group looking at Chicago, as visualization of their building import did not show the condensed timeframe during which many changesets were made when using changeset ID as the x-axis.

3. Per Edit Stats

This dataset records each individual edit to the map. This dataset is best for understanding exactly what changed on the map with each edit. Each edit tracks the tags changed as well as the geometry changes (if any). This dataset is significantly larger than the other two.

What cities are available?

Detroit is currently available in this repository. See this list in the readme for a handful of North American cities available for download.

5. Example Notebooks

  1. Per User Stats
  2. Per Changeset Stats
  3. Per Edit Stats

Editing heatmap Example heatmap from building edits in Detroit

If you're interested in more of this type of analysis, directions on setting up this analysis environment locally can be found in this repository. Furthermore, much of this is my current dissertation work, so I'm always happy to chat more about it. Thanks!

Location: The Hill, Boulder, Boulder County, Colorado, 80802, USA

10000 Edits Contest

Posted by Toeby on 5 December 2018 in English (English)

Today, i have been recognized, and i am in a contest with fellow mappers to reach 10000 edits by next Wednesday ( 7 days from now). Its a big challenge. This should be fun :) #osmnigeria#hotosm#microgrant2018#mapthedifference2018#uniquemappersteam#Youthmappers#unilagmappersteam

#CrimeaІsUkraine #DWG #CrimeaMap #КримЦеУкраїна #ИхТамНет

Posted by z-yurets on 5 December 2018 in English (English)

The Ukrainian community is concerned about the possible negative impact on the project as a whole, the emergence of lawsuits from users of data and the subsequent decline of the project, and therefore restores the borders of Ukraine to the internationally recognized status. The recent decision of DWG ( https://wiki.osmfoundation.org/wiki/Working_Group_Minutes/DWG_2018-11-14_Crimea ) neglects the wide recognition of Crimea as an integral part of Ukraine expressed by numerous governments and international organizations (in particular, UN General Assembly Resolution 68/262 http://www.un.org/en/ga/68/resolutions.shtml / https://undocs.org/en/A/RES/68/262 ). DWG actions directed to cut off Crimea from the borders of Ukraine are considered to be inadequate to the interests of the project and are not recognized by law. Any blockages (bans) aimed against members who restored the border of Ukraine to the widely internationally recognized status will be seen as unjustified pressure on the entire community and usurpation of power in the OSM.

P.S.

“changing names or country information would require consensus from both the Ukrainian and Russian communities. It is unlikely that any such edit proposals will be able to achieve this.” (с) DWG //

Moreover, according to clause 4, a consensus should be reached between the Ukrainian and Russian communities on changing information about countries. There is no consensus - there is no reason to separate the Crimea from Ukraine.

#CrimeaІsUkraine #DWG #CrimeaMap #КримЦеУкраїна #ИхТамНет

https://www.openstreetmap.org/relation/60199

https://www.openstreetmap.org/user/velmyshanovnyi - https://www.openstreetmap.org/user_blocks/2360 ;

https://www.openstreetmap.org/user/velmyshanovnyi - https://www.openstreetmap.org/user_blocks/2359 ;

https://www.openstreetmap.org/user/andergrin - https://www.openstreetmap.org/user_blocks/2358 ;

https://www.openstreetmap.org/user/Дівчина_з_Коломиї - https://www.openstreetmap.org/user_blocks/2357 ;

https://www.openstreetmap.org/user/паляниця - https://www.openstreetmap.org/user_blocks/2356 ;

https://www.openstreetmap.org/user/frankoivan - https://www.openstreetmap.org/user_blocks/2355 ;

https://www.openstreetmap.org/user/pumpkinpie226 - https://www.openstreetmap.org/user_blocks/2354 ;

https://www.openstreetmap.org/user/Хтосьіншийдятел - https://www.openstreetmap.org/user_blocks/2353 ;

https://www.openstreetmap.org/user/ue0 - https://www.openstreetmap.org/user_blocks/2350 ;

https://www.openstreetmap.org/user/andergrin - https://www.openstreetmap.org/user_blocks/2348 ;

https://www.openstreetmap.org/user/KKS - https://www.openstreetmap.org/user_blocks/2347 ;

...and other

https://www.facebook.com/openstreetmapua // https://twitter.com/osm_ua // https://t.me/osmUA // openstreetmap.ua@gmail.com

#CrimeaІsUkraine #DWG #CrimeaMap #КримЦеУкраїна #ИхТамНет

Posted by Artiom Komolov on 5 December 2018 in English (English)

The Ukrainian community is concerned about the possible negative impact on the project as a whole, the emergence of lawsuits from users of data and the subsequent decline of the project, and therefore restores the borders of Ukraine to the internationally recognized status. The recent decision of DWG ( https://wiki.osmfoundation.org/wiki/Working_Group_Minutes/DWG_2018-11-14_Crimea ) neglects the wide recognition of Crimea as an integral part of Ukraine expressed by numerous governments and international organizations (in particular, UN General Assembly Resolution 68/262 http://www.un.org/en/ga/68/resolutions.shtml / https://undocs.org/en/A/RES/68/262 ). DWG actions directed to cut off Crimea from the borders of Ukraine are considered to be inadequate to the interests of the project and are not recognized by law. Any blockages (bans) aimed against members who restored the border of Ukraine to the widely internationally recognized status will be seen as unjustified pressure on the entire community and usurpation of power in the OSM. It also violates OSM guidelines in respect of using national boundaries: https://wiki.openstreetmap.org/wiki/Boundaries#National

P.S. “changing names or country information would require consensus from both the Ukrainian and Russian communities. It is unlikely that any such edit proposals will be able to achieve this.” (с) DWG //

Moreover, according to clause 4, a consensus should be reached between the Ukrainian and Russian communities on changing information about countries. There is no consensus - there is no reason to separate the Crimea from Ukraine.

#CrimeaІsUkraine #DWG #CrimeaMap #КримЦеУкраїна #ИхТамНет

New OpenStreetBrowser version

Posted by skunk on 4 December 2018 in English (English)

New OpenStreetBrowser version! -> https://www.openstreetbrowser.org

Main new feature: * Export all visible map features as GeoJSON, OSM XML or OSM JSON!

More: https://blog.openstreetbrowser.org/node/59

#openstreetmap #osm

Road categorisation - OpenStreetMap versus Flemish Planning Documents

Posted by Tim Couwelier on 4 December 2018 in English (English)

Intro

First of all, some background. I'm employed by a consultancy agency, and anything traffic/mobility related is part of my job. It should also imply I'm familiar with several planning documents...

When diving into the world that is OpenStreetMap, I've had to realize a few things. Things in OpenStreetMap do NOT always mean what I assume they mean. Road categorisation is one of the key points here, and frequent need to repeat the explanation has finally made me create a diary post on the subject. For clarification I'll try to explain in english, but add the corresponding terms in dutch after.

The problem

At the core of the issue is semantics. OpenStreetMap and 'the real world' but use certain terms, like 'primary roads', 'secondary roads', 'tertiary roads'. These are however not synonymous, and editing OpenStreetMap just to be in line with those planning documents is often well.. wrong.

The planning context

  • Primary roads ('primaire wegen') are selected in the RSV ('Ruimtelijk Structuurplan Vlaanderen', or 'spatial structural plan Flanders'). They deal with the connections between 'Grootstedelijke gebieden' - essentially the main metropoles. Think along the likes of Gent, Antwerpen, Brussel, Brugge, ...
  • Secondary roads ('secundaire wegen') are selected one planning level lower, at a provincial level. The PRS ('Provinciaal Ruimtelijk Structuurplan') selects these. They deal with the connections between regional poles.
  • Tertiary roads are not named as such, but fit under the umbrella of 'lokale wegen'. In planning documents, we speak of 'lokale wegen types I, II and III', where I and II are dealing with the connections between very local poles, and III is basically whatever didn't fit elsewhere'. Selection of these happens in the 'Gemeentelijke Mobiliteitsplan' ('communal mobility plan).

The provincial level used to have a matching responsability for provinces to maintain some of the roads in their category. Around a decade ago, an overlap in responsabilities between Agentschap Wegen en Verkeer en Provinces was sorted out: roads maintained by Province were either transfered to Agentschap Wegen en Verkeer, or to the communities, based on functionality of the road.

In addition, note that there's always multiple versions, where typically the 'I' version means connection between poles of a same scale, and the 'II' version means collecting traffic from a lower level. In an attempt to summarize, here's an attempt at graphical representation of the principles behind it: Summary of the road categorisation according to planning documents. (Based on courses by Joris Willems, as I was taught at Vives Kortrijk)

The OpenStreetMap way

I'll base the comparsion of the wiki page for the belgian conventions: wiki page

At first glance, the comparison still seems straightforward:

  • Hoofdwegen vs Motorway

  • Primaire wegen vs Primary

  • Secundaire wegen vs Secondary

  • Lokale wegen I & II vs Tertiary

  • Options to specify more clearly what 'lokale wegen III' are (paths, service roads, residential roads, ...)

... leaving only 'trunk' as the odd one out.

Upon digging in a bit deeper however, the criteria as described are detailed in part through relation with the road numbers.

What once started as a ideal world (N1-N9 connect province capitals with Brussels, Nx0 add on to those in a radial pattern, interconnection those), with Nxx turning those basic structures into a finer mesh based on local city distribution. Proper 'ringroads' around cities get an Rxx number, which lets you filter out those functions too. And where there's only 'partial ring roads' around smaller towns, the part that circles around gets the Nxx number, while the old downgraded trajectory (which hopefully got an ACTUAL downgrade in profile) gets a number like Nxxa or Nxxd, or any variation of the sorts.

That seems sensible enough to link it to the categorisation (given there's a form of similarity to the earlier stated reasoning), regardless of what interpretation (OSM vs planning) is taken into account.

Intended use versus actual use

One of the key principles we adhere to when mapping is 'sticking to ground truth'. It's at times a sad realisation, but many roads have a lay-out, a certain 'look and feel', that leans more towards a different function then the actual intended use.

This inherently implies that for cases a road that should only give access to a small residential area (highway = residential, or a 'lokale weg III') actually has the appearance of a tertiary road, or that what should be a major axis in the traffic network has a rather underwhelming profile making it look and feel like a lower category..

We (the belgian community, and I'd assume by extension most communities) chose not to map according to 'vision documents', but to what we can see / survey in the field. Especially tertiary / secondary roads are subject to different interpretations between the two.

I still don't get the fuss - where does it actually differ?

Nothing like a good example to make a point. I've collected a few examples from my own region, as these were the cases that initially got me stumped.

The R32 - Ringroad around Roeselare

The R32, according to OSM logic, is a ring-road, with an Rxx number, and should therefor entirely be classified as 'primary' road all along the way. However, structural planning documents disagree: rather then seeing it as an actual 'ring-road' to divert traffic around the city, they see it as a 'traffic collecting road' towards the E403 - which 'collects' for example traffic from the northern branch of the N32 and the N36, then 'connects' those streams to a higher degree network...
The R32 This would imply it being a secondary road in the sections west of the N32, and primary between N32 and E403 both ways.. which is means the majority of the length is judged differently.

Meiboomlaan Roeselare

In OpenStreetMap you'll see Meiboomlaan in Roeselare marked as 'secondary'. It carries a number (N37), and it also looks like it's a road for a connection at a semi-high scale (watch it here ).

If we were to judge that road merely on planning documents alone, it is a 'lokale weg II', which boils down to a tertiary road...

In comparison, Hoogleedsesteenweg (connection to neighbouring town Hooglede) and Beversesteenweg (connection between Roeselare and part-community Beveren) which do form a connection between poles at a local level, do not exceed 'tertiary' status in OSM, largely due to a lack an N-number.... Somewhat ironical, within the limits of the so called 'Kleine Ring' (Westlaan - Noordlaan - Brugsesteenweg - Koning Leopold III-laan - Mandellaan - Kaaistraat - Koning Albert I-laan), having or not having an N-number doesn't mean all that much - these roads are now owned and maintained by the city of Roeselare.

But my choice is the better one!

Err.. maybe it is, maybe it's not. We need some common ground to determine how we tag things, and obtain a database (plus a rendered map) that fits as close to reality as we can get it. Sure, given the very very very similar structure and terminology it's easy to get carried away and 'straighten out this mess'. Believe me, I've been there. But I'm used to thinking in 'planning' and 'networks' and the theoretical part of how we'd like things to have. But untill that day, we should map things as they are.

We see similar patterns in other applications: - Landuse. Something may be designated for industrial buildings, if we go there and we see it's still entirely a meadow or farmland, we map it as such. We do not map it as an industrial zone, untill it actually is one. - Buildings: Even if we KNOW what's going up there, we don't map the building as long as it's not there. We don't map fancy landscaping around an appartment building, if it's still rough terrain as a building site - ...

You didn't mention Living Streets, School Streets, Cyclestreets, ...'

You're absolutely right, I didn't. These are all subcategories of what we'd call 'Lokale wegen III', and by default, OSM has a lot more flexibility to classify what's what. You'll never get overruled on these purely based on what's in any sort of planning document. I took the liberty of considering these out of the scope of this diary post.

Location: Krottegem, Aardappelhoek, Roeselare, West Flanders, Flanders, 8800, Belgium

Facebook Fort McMurray Alberta

Posted by Respectfully on 4 December 2018 in English (English)

Facebook uses a filter to limit distances for interested items. Example the search will only take results from this area. Would someone be able to get the distance in kilometers please. Currently it is only in miles.

Kind regards and thank you in advance.

Some numbers about mailing lists (part 2): Number of messages per mailing list and year, most active authors since 2016

Posted by Nakaner on 3 December 2018 in English (English)

A new evening is a new evening to play around with SQL. This is part 2 of my series on the number of messages and the most active authors on the public mailing lists hosted a lists.openstreetmap.org. Part 1 was published on 2 December 2018.

Here are my latest queries and their results:

Numbers of messages per year

SELECT EXTRACT(YEAR FROM "date") AS year, COUNT(1) AS message_count
  FROM mails
  GROUP BY year
  ORDER BY year;

 year | message_count 
------+---------------
 2004 |           180
 2005 |          2343
 2006 |         10481
 2007 |         34059
 2008 |         70120
 2009 |        102357
 2010 |         96569
 2011 |         66484
 2012 |         69943
 2013 |         67428
 2014 |         55866
 2015 |         52484
 2016 |         38914
 2017 |         36815
 2018 |         34271

Numbers of messages per mailing list and year (2004–2008)

Mailing lists with less than 50 messages in all of these years are not shown in the following table. Rows are sorted by number of messages in their first year.

WITH
  postings_years AS (
    SELECT list_id, EXTRACT(YEAR FROM "date") AS year, COUNT(1) AS message_count                                                       
      FROM mails          
      GROUP BY list_id, year     
  ), 
  list_ids AS (
    SELECT list_id        
      FROM mails          
      GROUP BY list_id    
  )
SELECT *
  FROM (
    SELECT
        l.list_id AS list_id,
        y2004.message_count AS "2004",  
        y2005.message_count AS "2005",  
        y2006.message_count AS "2006",  
        y2007.message_count AS "2007",  
        y2008.message_count AS "2008"
      FROM list_ids AS l  
      LEFT OUTER JOIN postings_years AS y2004
        ON (l.list_id = y2004.list_id AND y2004.year = 2004)                                     
      LEFT OUTER JOIN postings_years AS y2005
        ON (l.list_id = y2005.list_id AND y2005.year = 2005)                                     
      LEFT OUTER JOIN postings_years AS y2006
        ON (l.list_id = y2006.list_id AND y2006.year = 2006)                                     
      LEFT OUTER JOIN postings_years AS y2007
        ON (l.list_id = y2007.list_id AND y2007.year = 2007)                                     
      LEFT OUTER JOIN postings_years AS y2008
        ON (l.list_id = y2008.list_id AND y2008.year = 2008)
      ORDER BY
        y2004.message_count DESC NULLS LAST,
        y2005.message_count DESC NULLS LAST,
        y2006.message_count DESC NULLS LAST,
        y2007.message_count DESC NULLS LAST,
        y2008.message_count DESC NULLS LAST,
        list_id
    ) AS stats
WHERE "2004" >= 50
  OR "2005" >= 50
  OR "2006" >= 50
  OR "2007" >= 50
  OR "2008" >= 50;

      list_id      | 2004 | 2005 | 2006 | 2007  | 2008  
-------------------+------+------+------+-------+-------
 talk              |  180 | 1771 | 7772 | 11656 | 11310
 dev               |      |  572 | 2119 |  5625 |  4979
 talk-gb           |      |      |  239 |  1952 |  1524
 talk-de           |      |      |  235 |  5953 | 26392
 legal-talk        |      |      |   47 |   452 |  1285
 talk-fr           |      |      |   42 |   763 |  4579
 talk-it           |      |      |   27 |   322 |  4381
 talk-nl           |      |      |      |  4107 |  3873
 talk-es           |      |      |      |   888 |  1473
 josm-dev          |      |      |      |   597 |  1689
 talk-cz           |      |      |      |   590 |  1727
 talk-au           |      |      |      |   487 |   771
 talk-za           |      |      |      |   184 |   104
 routing           |      |      |      |   162 |   432
 talk-tr           |      |      |      |   103 |    26
 party             |      |      |      |    69 |     7
 talk-ie           |      |      |      |    54 |   192
 talk-fi           |      |      |      |    52 |    30
 talk-se           |      |      |      |    19 |    58
 talk-us           |      |      |      |    13 |   619
 merkaartor        |      |      |      |       |  1001
 talk-ja           |      |      |      |       |   897
 talk-be           |      |      |      |       |   612
 talk-ca           |      |      |      |       |   471
 talk-at           |      |      |      |       |   441
 talk-ph           |      |      |      |       |   261
 talk-in           |      |      |      |       |   220
 talk-ar           |      |      |      |       |   181
 talk-gb-midanglia |      |      |      |       |   116
 legal-general     |      |      |      |       |   111
 talk-co           |      |      |      |       |   100
 talk-il           |      |      |      |       |    69
 talk-is           |      |      |      |       |    69
(33 rows)

Number of messages per mailing list and year (2009–2013)

Mailing lists with less than 50 messages in all of these years are not shown in the following table. Rows are sorted by number of messages in their first year.

WITH
  postings_years AS (
    SELECT list_id, EXTRACT(YEAR FROM "date") AS year, COUNT(1) AS message_count
      FROM mails
      GROUP BY list_id, year
  ),
  list_ids AS (
    SELECT list_id
      FROM mails
      GROUP BY list_id
  )
SELECT *
  FROM (
    SELECT
        l.list_id AS list_id,
        y2009.message_count AS "2009",
        y2010.message_count AS "2010",
        y2011.message_count AS "2011",
        y2012.message_count AS "2012",
        y2013.message_count AS "2013"
      FROM list_ids AS l
      LEFT OUTER JOIN postings_years AS y2009 ON (l.list_id = y2009.list_id AND y2009.year = 2009)
      LEFT OUTER JOIN postings_years AS y2010 ON (l.list_id = y2010.list_id AND y2010.year = 2010)
      LEFT OUTER JOIN postings_years AS y2011 ON (l.list_id = y2011.list_id AND y2011.year = 2011)
      LEFT OUTER JOIN postings_years AS y2012 ON (l.list_id = y2012.list_id AND y2012.year = 2012)
      LEFT OUTER JOIN postings_years AS y2013 ON (l.list_id = y2013.list_id AND y2013.year = 2013)
      ORDER BY
        y2009.message_count DESC NULLS LAST,
        y2010.message_count DESC NULLS LAST,
        y2011.message_count DESC NULLS LAST,
        y2012.message_count DESC NULLS LAST,
        y2013.message_count DESC NULLS LAST,
        list_id
    ) AS stats
WHERE "2009" >= 50
  OR "2010" >= 50
  OR "2011" >= 50
  OR "2012" >= 50
  OR "2013" >= 50;

        list_id        | 2009  | 2010  | 2011  | 2012  | 2013  
-----------------------+-------+-------+-------+-------+-------
 talk-de               | 28125 | 20392 | 10276 |  8758 |  6088
 talk                  | 13570 |  9296 |  5724 |  4095 |  3336
 talk-fr               | 11994 | 12072 |  9541 | 13755 | 12880
 talk-it               |  8267 |  7896 |  5182 |  6285 |  8046
 dev                   |  4720 |  3398 |  2613 |  2235 |  1236
 talk-au               |  3515 |  2617 |  1291 |  1019 |   450
 talk-gb               |  3358 |  3428 |  2027 |  1669 |  1386
 talk-nl               |  2511 |  1824 |  1236 |   721 |   328
 talk-us               |  1942 |  2436 |  2101 |  2885 |  2385
 talk-es               |  1892 |  2281 |  2121 |  2265 |  1063
 talk-cz               |  1729 |  2019 |   890 |  1064 |   888
 josm-dev              |  1587 |  1217 |   792 |   464 |   428
 talk-ca               |  1539 |  1511 |   832 |   895 |   659
 talk-ph               |  1345 |  1185 |   835 |   584 |   639
 talk-lv               |  1304 |   967 |  1724 |   985 |   543
 legal-talk            |  1291 |  2468 |  1226 |   510 |   310
 talk-at               |  1278 |   939 |   934 |  1640 |   990
 talk-ja               |  1224 |  1844 |  1667 |  1222 |   917
 tagging               |  1011 |  5118 |  2900 |  3368 |  3551
 talk-ro               |   944 |   786 |   292 |   238 |   606
 merkaartor            |   942 |  1022 |   158 |    94 |    20
 talk-br               |   901 |   971 |   480 |   671 |  1728
 talk-transit          |   858 |   391 |   290 |    50 |    83
 talk-in               |   824 |   355 |    92 |   210 |   177
 talk-hr               |   553 |   478 |   316 |   344 |   285
 imports               |   440 |   280 |   504 |   548 |   848
 osmosis-dev           |   418 |   446 |   317 |   249 |   140
 talk-be               |   416 |   674 |   733 |   921 |  2098
 talk-gb-westmidlands  |   364 |   261 |   237 |   300 |   229
 talk-co               |   363 |  1148 |   964 |   500 |   321
 talk-dk               |   357 |   708 |  1055 |  1063 |   411
 talk-is               |   269 |   207 |    65 |   145 |    92
 talk-za               |   257 |   189 |    78 |    82 |   176
 routing               |   247 |   233 |    78 |    76 |    14
 talk-ee               |   228 |   548 |   357 |   199 |   154
 talk-se               |   211 |   411 |   414 |   676 |   572
 talk-ar               |   211 |   244 |   219 |   155 |   289
 talk-lt               |   148 |   484 |   285 |   251 |   334
 talk-fr-bzh           |   117 |   114 |   290 |   279 |   194
 talk-gb-midanglia     |   114 |    48 |    62 |    16 |     8
 talk-cu               |   108 |    34 |       |    11 |     3
 talk-si               |   107 |    55 |    26 |    34 |     2
 talk-gb-oxoncotswolds |    87 |    20 |   101 |     8 |    85
 local-chapters        |    65 |   118 |    20 |       |    47
 geocoding             |    65 |    77 |    48 |   423 |   430
 talk-ie               |    61 |    50 |    64 |    26 |    44
 moderation            |    59 |    15 |       |       |      
 accessibility         |    55 |   101 |    55 |    75 |      
 talk-vi               |    54 |     1 |     1 |     7 |      
 potlatch-dev          |    51 |   326 |   984 |   540 |    79
 talk-cl               |    33 |   857 |   120 |    83 |    36
 talk-ko               |    21 |    34 |    11 |    67 |    11
 talk-pt               |    14 |    70 |   232 |   171 |   178
 talk-tr               |    12 |    18 |    21 |    38 |    64
 talk-pe               |     3 |    65 |    53 |   127 |    40
 hot                   |       |   456 |   861 |  1179 |  1787
 talk-ht               |       |   347 |    64 |   183 |   337
 talk-it-fvg           |       |   206 |    75 |    66 |   180
 strategic             |       |   185 |   398 |    14 |      
 dev-fr                |       |   140 |   480 |   821 |   449
 mapcss                |       |   134 |   116 |    61 |    65
 talk-ve               |       |    86 |   255 |   226 |    19
 talk-tw               |       |    53 |   112 |   467 |   412
 talk-rs               |       |    30 |   199 |    71 |     1
 talk-uy               |       |    12 |    17 |    24 |    68
 talk-it-piemonte      |       |       |   281 |   143 |    47
 rails-dev             |       |       |   187 |  1408 |  3458
 taginfo-dev           |       |       |    74 |    52 |    59
 talk-id               |       |       |    70 |    64 |    21
 talk-it-bici          |       |       |    54 |    47 |      
 talk-it-lazio         |       |       |    49 |    36 |   201
 talk-it-trentino      |       |       |    33 |   133 |   323
 talk-lu               |       |       |    33 |    59 |    16
 talk-np               |       |       |    14 |   163 |    31
 talk-it-liguria       |       |       |    13 |   513 |   153
 talk-pl               |       |       |     7 |   449 |   346
 rebuild               |       |       |       |   353 |      
 historic              |       |       |       |    72 |   265
 talk-sn               |       |       |       |    36 |   152
 talk-ni               |       |       |       |    15 |   310
 talk-baltics          |       |       |       |     5 |    91
 sotm-asia             |       |       |       |     4 |    86
 talk-mx               |       |       |       |     2 |    57
 tile-serving          |       |       |       |       |   778
 imports-us            |       |       |       |       |   481
 osrm-talk             |       |       |       |       |   359
 talk-cat              |       |       |       |       |   230
 diversity-talk        |       |       |       |       |   130
 talk-us-nps           |       |       |       |       |   106
 talk-it-southtyrol    |       |       |       |       |    69
 talk-it-sardinia      |       |       |       |       |    61
 talk-ke               |       |       |       |       |    61
(92 rows)

Some mailing lists changed their focus or became unnessary after some time. For example, the legal-talk mailing list became a bit more quiet after the license change in mid 2012 was over. The "rebuild" mailing list had the only purpose to discuss the decisions implemented in the Redaction Bot.

Number of messages per mailing list and year (2014 to November 2018)

Mailing lists with less than 50 messages in all of these years are not shown in the following table. Rows are sorted alphabetically by the ID of the mailing list.

WITH
  postings_years AS (
    SELECT list_id, EXTRACT(YEAR FROM "date") AS year, COUNT(1) AS message_count
      FROM mails
      GROUP BY list_id, year
  ),
  list_ids AS (
    SELECT list_id
      FROM mails
      GROUP BY list_id
  )
SELECT *
  FROM (
    SELECT
        l.list_id AS list_id,
        y2014.message_count AS "2014",
        y2015.message_count AS "2015",
        y2016.message_count AS "2016",
        y2017.message_count AS "2017",
        y2018.message_count AS "2018"
      FROM list_ids AS l
      LEFT OUTER JOIN postings_years AS y2014 ON (l.list_id = y2014.list_id AND y2014.year = 2014)
      LEFT OUTER JOIN postings_years AS y2015 ON (l.list_id = y2015.list_id AND y2015.year = 2015)
      LEFT OUTER JOIN postings_years AS y2016 ON (l.list_id = y2016.list_id AND y2016.year = 2016)
      LEFT OUTER JOIN postings_years AS y2017 ON (l.list_id = y2017.list_id AND y2017.year = 2017)
      LEFT OUTER JOIN postings_years AS y2018 ON (l.list_id = y2018.list_id AND y2018.year = 2018)
      ORDER BY
        list_id
    ) AS stats
WHERE "2014" >= 50
  OR "2015" >= 50
  OR "2016" >= 50
  OR "2017" >= 50
  OR "2018" >= 50;

        list_id        | 2014 | 2015 | 2016 | 2017 | 2018 
-----------------------+------+------+------+------+------
 dev                   |  596 |  758 |  633 |  393 |  390
 dev-fr                |  365 |  144 |   74 |   54 |   14
 dev-italia            |   90 |   16 |  165 |   11 |    5
 diversity-talk        |  118 |    4 |      |    5 |  100
 geocoding             |  318 |  300 |  104 |   57 |   29
 historic              |  205 |  380 |   55 |   83 |   59
 hot                   | 2669 | 3729 | 2088 | 1272 |  503
 hot-francophone       |  305 |  497 |  182 |  123 |  107
 imports               |  914 |  604 |  518 |  564 |  515
 imports-us            |  154 |   90 |   47 |   12 |   52
 josm-dev              |  288 |  437 |  172 |  157 |  217
 learnosm-coord        |   50 |   57 |    4 |      |     
 legal-talk            |  367 |  276 |  197 |   74 |   29
 osmf-membership       |    2 |   59 |      |      |     
 osmosis-dev           |   23 |   89 |   35 |   19 |   19
 osrm-talk             |  343 |  345 |  293 |  179 |  156
 potlatch-dev          |   14 |   22 |    2 |   71 |   12
 rails-dev             | 1309 | 1783 | 1781 | 2232 | 2494
 tagging               | 4685 | 7217 | 2803 | 3734 | 6602
 talk                  | 2814 | 3584 | 1975 | 2668 | 1778
 talk-africa           |      |    1 |   16 |  100 |  114
 talk-ar               |  228 |   77 |    4 |   11 |     
 talk-at               |  851 |  938 |  596 |  334 |  729
 talk-au               |  248 |  279 |  370 |  416 |  592
 talk-be               | 1119 | 1569 | 1234 |  837 |  260
 talk-bf               |   84 |   45 |   27 |   18 |    1
 talk-bj               |    3 |    5 |   26 |   64 |    5
 talk-blr              |    3 |   52 |      |      |     
 talk-bo               |   24 |   32 |  137 |  153 |  210
 talk-br               | 4412 | 1809 |  730 |  461 |  189
 talk-ca               |  370 |  311 |  906 |  611 |  679
 talk-cat              |  368 |  374 |   83 |   74 |   45
 talk-cl               |  106 |  110 |  165 |  159 |   83
 talk-cm               |   31 |    8 |    5 |   41 |   50
 talk-co               |  200 |  268 |  264 |  128 |   55
 talk-cu               |    1 |   13 |  131 |   42 |   35
 talk-cz               | 2290 | 1720 | 2594 | 2490 | 2140
 talk-de               | 3615 | 2306 | 1335 |  764 | 1138
 talk-dk               |  359 |  415 |  299 |  156 |  212
 talk-es               |  786 |  762 | 1121 |  814 |  856
 talk-fr               | 8479 | 5082 | 3690 | 4110 | 4008
 talk-fr-bzh           |  399 |  332 |  157 |  153 |  169
 talk-gb               | 1300 | 1136 | 1524 | 1390 | 1151
 talk-gb-westmidlands  |  262 |  161 |  123 |  180 |   65
 talk-gh               |    1 |   44 |   30 |   77 |   74
 talk-hr               |  238 |  164 |   35 |   32 |   65
 talk-ht               |  144 |   62 |  131 |   72 |   46
 talk-ie               |  352 |  481 |  297 |  298 |  159
 talk-in               |  125 |  362 |  271 |  275 |  189
 talk-it               | 5730 | 4646 | 5645 | 4843 | 3675
 talk-it-cai           |      |      |      |  336 |  173
 talk-it-fvg           |  139 |  219 |  269 |  236 |   96
 talk-it-lazio         |   64 |   23 |   47 |  231 |   98
 talk-it-liguria       |   73 |   19 |   12 |   10 |    2
 talk-it-piemonte      |  118 |   49 |   17 |   73 |  113
 talk-it-sardinia      |  110 |   12 |   28 |      |     
 talk-it-southtyrol    |   69 |   10 |      |    2 |    1
 talk-it-trentino      |  163 |  159 |  277 |   71 |   68
 talk-ja               |  635 |  543 |  356 |  378 |  370
 talk-ko               |   26 |   30 |    5 |   97 |   36
 talk-latam            |  165 |  235 |  167 |  119 |   70
 talk-lt               |   81 |  122 |  215 |  163 |   86
 talk-lv               |  153 |  132 |   57 |   63 |   28
 talk-mg               |      |   68 |   20 |   28 |   46
 talk-ml               |    9 |  101 |   33 |   29 |    4
 talk-mx               |  139 |  252 |  127 |   79 |   53
 talk-ni               |  169 |  113 |  125 |   59 |   29
 talk-nl               |  417 |  199 |  133 |   53 |   33
 talk-pe               |   71 |  187 |  137 |   40 |   12
 talk-ph               |  547 |  356 |  177 |  155 |  117
 talk-pl               |  726 |  614 |  531 |   74 |   98
 talk-pr               |  142 |   92 |   53 |   29 |    4
 talk-pt               |  372 |  190 |  110 |  113 |   71
 talk-ro               |  204 |  134 |   39 |    2 |    3
 talk-scotland         |   26 |   69 |   42 |   96 |   39
 talk-se               |  230 |  146 |  247 |  263 |  129
 talk-sn               |   64 |  114 |  113 |   75 |   57
 talk-tn               |      |      |   58 |   58 |   37
 talk-transit          |   33 |   42 |   51 |   11 |  131
 talk-tw               |   91 |   54 |   64 |   13 |   18
 talk-us               | 1552 | 1771 | 1108 | 1300 |  847
 talk-us-massachusetts |      |      |   37 |  114 |  248
 tile-serving          |  908 | 1870 |  709 | 1006 |  648
 umap                  |   17 |   49 |   79 |   63 |   36
(84 rows)

Top Authors per Year

SELECT 
    row_number() OVER (ORDER BY message_count DESC) AS rank,
    author,
    message_count
  FROM (
    SELECT
        from_short AS author,
        COUNT(1) AS message_count
      FROM mails
      WHERE
        EXTRACT(YEAR FROM "date") = 2018
        AND from_short NOT IN ('notifications@github.com', 'theweekly.osm@gmail.com')
      GROUP BY author
      ORDER BY message_count DESC
  ) AS a
  LIMIT 20;

 rank |          author           | message_count 
------+---------------------------+---------------
    1 | Martin Koppenhoefer       |          1819
    2 | Warin                     |           569
    3 | Marc Marc                 |           554
    4 | Philippe Verdy            |           506
    5 | Mateusz Konieczny         |           478
    6 | Giovanni Cascafico        |           344
    7 | Paul Allen                |           322
    8 | Graeme Fitzpatrick        |           295
    9 | François Lacombe          |           272
   10 | Simone Girardelli         |           271
   11 | Marián Kyral              |           271
   12 | Christoph Hormann         |           257
   13 | Tom Ka                    |           249
   14 | Volker Schmidt            |           248
   15 | Polyglot                  |           244
   16 | majka                     |           244
   17 | Frederik Ramm             |           236
   18 | Javier Sánchez Portero    |           230
   19 | John Whelan               |           229
   20 | Christan Quest            |           226
(20 rows)

SELECT 
    row_number() OVER (ORDER BY message_count DESC) AS rank,
    author,
    message_count
  FROM (
    SELECT
        from_short AS author,
        COUNT(1) AS message_count
      FROM mails
      WHERE
        EXTRACT(YEAR FROM "date") = 2017
        AND from_short NOT IN ('notifications@github.com', 'theweekly.osm@gmail.com')
      GROUP BY author
      ORDER BY message_count DESC
  ) AS a
  LIMIT 20;

 rank |              author              | message_count 
------+----------------------------------+---------------
    1 | Martin Koppenhoefer              |          1598
    2 | Philippe Verdy                   |           581
    3 | Marc Marc                        |           499
    4 | Marián Kyral                     |           481
    5 | Warin                            |           414
    6 | Marc Gemis                       |           372
    7 | Giovanni Cascafico               |           357
    8 | Simone Girardelli                |           352
    9 | John Whelan                      |           351
   10 | osm.sanspourriel                 |           296
   11 | Christian Quest                  |           291
   12 | Andy Townsend                    |           248
   13 | Volker Schmidt                   |           235
   14 | LogicalViolinist/James2432       |           233
   15 | Polyglot                         |           229
   16 | Frederik Ramm                    |           225
   17 | Dave F                           |           222
   18 | Alessandro Palmas                |           221
   19 | Joost Schouppe                   |           210
   20 | Paul Johnson                     |           210
(20 rows)


SELECT 
    row_number() OVER (ORDER BY message_count DESC) AS rank,
    author,
    message_count
  FROM (
    SELECT
        from_short AS author,
        COUNT(1) AS message_count
      FROM mails
      WHERE
        EXTRACT(YEAR FROM "date") = 2016
        AND from_short NOT IN ('notifications@github.com', 'theweekly.osm@gmail.com')
      GROUP BY author
      ORDER BY message_count DESC
  ) AS a
  LIMIT 20;

 rank |              author              | message_count 
------+----------------------------------+---------------
    1 | Martin Koppenhoefer              |          1781
    2 | Philippe Verdy                   |           506
    3 | Marián Kyral                     |           431
    4 | Simone Girardelli                |           394
    5 | Marc Gemis                       |           385
    6 | John Whelan                      |           347
    7 | osm.sanspourriel                 |           319
    8 | Christan Quest                   |           315
    9 | Giovanni Cascafico               |           306
   10 | Joost Schouppe                   |           302
   11 | Warin                            |           267
   12 | Polyglot                         |           245
   13 | Luca Delucchi                    |           240
   14 | Frederico Cortese                |           224
   15 | Maurizio Napolitano              |           223
   16 | Frederik Ramm                    |           223
   17 | Andrea Lattmann                  |           222
   18 | LogicalViolinist/James2432       |           220
   19 | Colin Smale                      |           215
   20 | Andy Townsend                    |           211
(20 rows)

I converted the email addresses into names manually by entering all email addresses into Google. While copying the results, I noticed that among the top 20 posters since 2016 only a few seem to have English as their native language. There are a lot of people from the French, Italian and Czech community. This is no surprise if you look for the most active mailing lists (see table above).

The OSMF-Talk mailing list was not taken into account for all these numbers (and it was not taken into account for the numbers published in part 1).

This list of the most active posters measured in messages per year will follow in the next part of this series. If you have any other suggestions what to query, don't hesitate to comment.

I would like to ask all who intend to write a comment to stay on topic and don't repeat themselves.

OSMF should vote more often...... 2018 Edition

Posted by SunCobalt on 3 December 2018 in English (English)

Last year I wrote a short blog post titled OSMF should vote more often...... after I have seen the rise of the number of OSMF members just before the election. This holds true this year as well.

Alt-Text

As the most recent numbers are as of Nov 16, also members not yet eligible for voting at this year's election are included.

convert osm offline viewer lg2 file to gpx

Posted by hungerburg on 3 December 2018 in English (English)

Below script converts tracks in lg2 format of Marek Baranowski's withdrawn IOS app to gpx

#!/usr/bin/perl
use POSIX qw(strftime);

my $name = shift or die "Usage: $0 filename\n";

open (FH, "<$name") or die 'No such file';
binmode(FH);

my $buf;
my $mul = 3600000;
my $fmt = "vVVvV";

print '<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.0">
    <time>'.strftime("%FT%X", localtime).'</time>
    <trk>
        <name>'.(split(/\./,$name))[0].'</name>
        <trkseg>';

while (my $got = read (FH, $buf, 16)) {
    my ($H, $B, $L, $X, $T) = unpack $fmt, $buf;
    print '
            <trkpt lat="'.$B/$mul.'" lon="'.$L/$mul.'"><ele>'.$H.'</ele><time>'.strftime("%FT%X", $T, 1, 1, 1, 0, 70).'</time></trkpt>' if $T > 100000;
}

print'
        </trkseg>
    </trk>
</gpx>
'

Ad hominem

Posted by Kilkenni on 2 December 2018 in English (English)

A funny thing OSMWeekly mentions my posts as "ad hominem". Whether it truly is or is not, I'm not sure.

However the thing I know is, this discussion is not about exclusively professional topics. It is not a "perfectly sterile scientific discussion". We are talking politics since OSMF and DWG are essentially OSM politicians. OSM is heavily used by many organizations related to governments. OSM depends on grants offered by international organizations and corporate sponsorship, it uses infrastructure provided by corporations. It does not exist in a vacuum, and is related to the processes happening in the real world. It influences said processes and is, in turn, influenced by them, whether we like it or not. OSM's independence is important - but it is independence from corporate politics, not public responsibility. Since OSM itself is not commercial, its success is greatly dependent on its reputation and public image. In this situation the one who makes decisions should understand that his decisions do not concern him alone. Like circles on the water from a thrown stone, these decisions cause consequences, both close and far-reaching, both for OSM and outside OSM, and it is paramount to understand these consequences. Legal consequences. Commercial consequences. Influence on mass-media and public opinion. In our century sources of information can start and stop wars, make or break a man. Or an organization, for that matter.

(Seriously though, I feel really uneasy now, as I perceive all of the above as self-explanatory, and it is extremely hard to explain something you see as obvious).

OSMF officials and DWG members like to call themselves volunteers. In a sense, they are. But they also are something much more than that, and I suspect they are still oblivious to that. OSM is not a small sandbox for a handful of programmer enthusiasts any more, it is an international project with thousands of people contributing to it and millions using it on a daily basis. Behavior that was acceptable for a small project is no longer acceptable of a large one. Any OSMF member, any DWG member is a public figure. He trades a part of his privacy for a chance to steer the project. This is politics. And just like common politics, this one requires total transparency, honesty and wisdom. If a judge or a police officer makes questionable decisions, people start digging information on him and pointing at questionable and suspicious details. It is a collective immune system of sorts, preventing corruption and ensuring the governing body is healthy. Can it really be called "ad hominem"?

When I criticize concrete people, I am driven by two ideas:

*1. If a person presents OSM on an official level, we should always ask ourselves if his actions benefit the project or harm it.

It is my firm belief that lack of transparency (and calls to "be quiet" and "avoid pressure on OSMF") harms the project. Once again, it is politics. Organized peaceful actions, manifestations, petitions are all legit forms of politics. Complaining that these forms "harm someone's feelings" show that these people are not ready for their position. While the protest stays civilized (and I'm all in for civilized) I see no reason why it should be avoided.

*2. Public figures either present their own opinions or collective opinions.

As I've said elsewhere in my blog and posts, a public figure takes full responsibility for his actions. We either hold him accountable, or we hold the entire collective accountable. I, for one, don't think we should extrapolate responsibility of one man on the group without having facts supporting this. We should give everyone a chance to explain the situation. Still, I've yet to see detailed DWG explanation beyond their purely official "resolution".

Another thing concerning individual and collective. I see some people calling not to question DWG decision but to explain its "non-political" ground. For me, this stance is a faulty one. As I've mentioned, OSM has many users, both corporate and private. Explaining it on a person-to-person level is not an appropriate way to resolve the situation. Nor is it right to tell people who do not support your decisions to essentially protect you from possible repercussions, shifting and extrapolating responsibility from a handful of people to the entirety of OSM community. If we talk about the future of OSM, it is especially questionable in corporate part. "You need it, you fix it" works good for small opensource projects. For global projects (and I see OSM as one) this approach is irresponsible and can do a lot of harm, seeing that we are said to value quality data.

Still, I once again stress that this discussion should remain civilized. If I allowed myself some rough words in the past, I apologize for them.

That being said, caution this is Internet.

Some analysis of the answers and manifestos of the candidates for the OSMF board election 2018

Posted by imagico on 2 December 2018 in English (English)

Yesterday the answers of the candidates to the official question for the OSMF board election 2018 were published.

This is a lot of material to work through and to simplify this for me i produced a summary of the key points i read in the candidates' answers and manifestos. In addition i put together some further research from publicly available sources. This is available here:

https://wiki.openstreetmap.org/wiki/User:Imagico/Analysis_of_OSMF_board_candidates_2018/

Although this does not really aim to communicate an opinion on the candidates - i try to describe the positions of the candidates rather than to judge them - this is of course not meant to be in any way an objective summary of the positions. I none the less thought it might be interesting for others to read and it might also be useful for candidates because there are comments pointing to answers that i perceive as strange and ambiguous. I would welcome any clarifying statements - here, on osmf-talk or on the wiki talk page.

I would also encourage others to present their own reading of the presentations of the candidates - because it is very valuable to learn how different people have different perspectives on the same matter. With the large spectrum of cultural backgrounds of the candidates this would help developing a clear picture of everyone and make a fact based decision in the election.

Some numbers about mailing lists

Posted by Nakaner on 2 December 2018 in English (English)

While reading the manifestos of the seven candidates of this year's OSMF board elections, I though that getting some numbers about their activity could add a little bit more information to my personal decision whom to vote.

Numbers? Many readers will think of the number of changesets which is the most visible number of an OSM account. However, this number should be interpreted with care because changesets can contain between 0 and 50,000 (nowadays 10,000) changes. Users of different editors create changesets of different size. To get a better picture of the activity of an OSM user, I recommend to use Pascal Neis's HDYC instead.

The number of data edits or the HDYC level (e.g. "super mapper (very active)") is only one information. There are a few more contributions which can be counted:

  • Users of the OSM Forum (you have to be logged in but you don't have to be a moderator) can search other users' accounts by their names. It's the menu item "members". It will show the number of postings of a user.
  • The OSMF runs a StackOverflow-like system where people can ask questions, e.g. how to use OSM or how to map things. It's called OSM Help. Answers can be up- or downvoted. This leads to an reputation. You can search for a user names and get their reputation. The forum and OSM Help have a a 1:1 mapping of Help/Forum and OSM API accounts but users who rename their accounts might get a new one (at least on the forum).
  • I haven't found where to get the "editcount" using the MediaWiki front end but you can query the API (you might have to be logged in). The URL is https://wiki.openstreetmap.org/w/api.php?action=query&list=users&usprop=editcount&format=json&ususers=USERNAME where USERNAME is the display name of the user.

That was the easy part. The most difficult follows – our Mailman mailing lists.

The archives of all public mailing lists can be download as GZIP compressed plain text files. I used mailman-download by Matt Hicks, a Ruby script without a proper license, for that purpose. Afterwards I had a set of directories, each containing the archives of one mailing list. As the next step, I loaded all of them into a PostgreSQL database to run queries against them. I thought it would be easy (I am used to work with Python and the Psycopg2, its PostgreSQL library) but it was more difficult. Here are the problems I encountered:

  • Archives prior to November 2014 are not encoded in UTF-8 but in a encoding which is used in the language of the mailing list, e.g. Latin1 for Talk-de or a Japanese encoding for Talk-ja. Later archives are encoded in UTF-8. This does not affect the Subject headers and mail bodies but the From headers as well if the name of the sender contains non-ASCII characters. However, the email address is readable if your used encoding has the same character mapping for character positions 1–127.
  • An email begins with two lines matching the regular expression ^From. But it is more difficult that it sounds. If people use the TOFU quoting style, this regular expression matches for embedded, quoted emails as well. Use the regular expessions ^From +[^: ] for the first line and ^From: the next line instead to be sure not to count an email twice. The header of all emails ends with a Message-ID header.
  • The separator of the local and the domain part of email addresses is not an @ but a string which is different between languages (at or a for most mailing lists but en on Talk-ar or a @-like character on Talk-jp).

Here are some results I got:

mails=# SELECT COUNT(1) AS count FROM mails;
 count  
--------
 738498

mails=# SELECT list_id, COUNT(1) AS message_count FROM mails GROUP BY list_id ORDER BY message_count DESC;
        list_id        | message_count 
-----------------------+---------------
 talk-de               |        115377
 talk-fr               |         90995
 talk                  |         81529
 talk-it               |         64945
 tagging               |         40989
 dev                   |         30267
 talk-gb               |         22084
 talk-cz               |         20141
 talk-us               |         18959
 talk-es               |         16322
 talk-nl               |         15435
 rails-dev             |         14652
 hot                   |         14544
 talk-br               |         12375
 talk-au               |         12055
 talk-be               |         10473
 talk-ja               |         10237
 talk-at               |          9670
 talk-ca               |          8784
 legal-talk            |          8532
 josm-dev              |          8045
 talk-ph               |          6201
 talk-lv               |          5956
 tile-serving          |          5919
 imports               |          5735
 talk-dk               |          5043
 talk-co               |          4311
 talk-se               |          3376
 merkaartor            |          3302
 talk-ro               |          3248
 talk-in               |          3100
 talk-pl               |          2845
 dev-fr                |          2541
 talk-hr               |          2510
 talk-gb-westmidlands  |          2219
 talk-fr-bzh           |          2204
 talk-lt               |          2171
 potlatch-dev          |          2101
 talk-ie               |          2078
 talk-transit          |          1940
 geocoding             |          1851
 osmosis-dev           |          1755
 talk-cl               |          1752
 talk-ee               |          1679
 osrm-talk             |          1675
 talk-ar               |          1619
 talk-pt               |          1521
 talk-it-fvg           |          1486
 talk-ht               |          1386
 routing               |          1319
 talk-tw               |          1284
 talk-it-trentino      |          1227
 hot-francophone       |          1214
 talk-cat              |          1174
 talk-za               |          1141
 historic              |          1119
 talk-is               |           939
 talk-it-piemonte      |           841
 imports-us            |           836
 talk-ni               |           820
 talk-it-liguria       |           795
 talk-latam            |           756
 talk-it-lazio         |           749
 talk-pe               |           735
 talk-mx               |           709
 talk-ve               |           621
 talk-sn               |           611
 strategic             |           597
 talk-bo               |           575
 talk-it-cai           |           509
 talk-us-massachusetts |           399
 mapcss                |           397
 talk-cu               |           378
 talk-tr               |           377
 talk-gb-midanglia     |           368
 diversity-talk        |           357
 rebuild               |           353
 talk-rs               |           346
 talk-ko               |           338
 talk-gb-oxoncotswolds |           333
 dev-italia            |           333
 talk-pr               |           332
 accessibility         |           324
 talk-si               |           301
 local-chapters        |           291
 talk-np               |           276
 talk-scotland         |           272
 umap                  |           244
 taginfo-dev           |           241
 talk-africa           |           232
 talk-gh               |           226
 talk-it-sardinia      |           211
 talk-tn               |           198
 talk-bf               |           192
 legal-general         |           188
 talk-ml               |           176
 talk-uy               |           166
 talk-fi               |           165
 talk-id               |           164
 talk-mg               |           162
 talk-cm               |           153
 talk-it-southtyrol    |           151
 talk-lu               |           149
 talk-it-bici          |           134
 talk-us-nps           |           131
 learnosm-coord        |           128
 talk-il               |           121
 talk-gb-london        |           115
 talk-bj               |           106
 talk-cn               |           103
 talk-ru               |           102
 talk-bd               |           101
 maproulette           |            98
 talk-tg               |            98
 photon                |            98
 talk-ug               |            97
 talk-baltics          |            96
 talk-ci               |            91
 sotm-asia             |            90
 talk-ke               |            87
 talk-it-veneto        |            84
 talk-kosovo           |            78
 party                 |            76
 moderation            |            74
 talk-gb-thenorth      |            72
 teachosm              |            66
 talk-vi               |            65
 tagging-fr            |            63
 osmf-membership       |            61
 announce              |            61
 talk-blr              |            55
 talk-cd               |            52
 talk-py               |            47
 talk-us-pugetsound    |            36
 talk-us-newyork       |            35
 talk-ma               |            35
 talk-it-sicilia       |            34
 talk-lk               |            31
 talk-ba               |            28
 talk-al               |            26
 talk-do               |            23
 welcomewg             |            23
 talk-nc               |            19
 talk-ne               |            19
 osm-professional      |            19
 talk-mw               |            18
 talk-ng               |            17
 talk-cr               |            17
 osmcha-dev            |            15
 talk-it-marche        |            14
 talk-tz               |            12
 talk-ps               |            12
 talk-mm               |            12
 talk-et               |            10
 openhiking            |             9
 science               |             9
 lorodux-dev           |             9
 talk-gr               |             7
 talk-us-sfbay         |             7
 talk-hn               |             7
 talk-it-appulo-lucana |             7
 hot-announce          |             6
 talk-kg               |             6
 talk-my               |             5
 osm2world             |             5
 talk-sc               |             3
 talk-lb               |             3
 talk-carib            |             3
 outdoor-natural       |             3
 talk-mn               |             2
 talk-iq               |             1
 talk-bi               |             1
 talk-asia             |             1
 talk-zm               |             1
(174 Zeilen)

mails=# SELECT row_number() OVER (ORDER BY message_count DESC) AS rank, address, message_count
  FROM (
    SELECT
        from_short AS address,
        COUNT(1) AS message_count
      FROM mails WHERE from_short NOT LIKE '%@github.com'
      GROUP BY address
  ) AS a
  ORDER BY message_count DESC
  LIMIT 20;
  -- email addresses have been manually replace by names or nicknames in the following listing. People who have used multiple email addresses on the mailing lists are treated as multiple persons.
 rank |             address             | message_count
------+---------------------------------+---------------
    1 | Martin Koppenhoefer             |         25634
    2 | Frederik Ramm                   |         12274
    3 | Pieren                          |          6537
    4 | Philippe Verdy                  |          5731
    5 | Christian Quest                 |          4956
    6 | John Smith                      |          4582
    7 | Tobias Wendorff                 |          3494
    8 | Luca Delucchi                   |          3171
    9 | Sven Geggus                     |          3113
   10 | Jan Tappenbeck                  |          3025
   11 | Steve Coast                     |          3024
   12 | Maning Sambale                  |          2849
   13 | Polyglot                        |          2831
   14 | Tom Hughes                      |          2668
   15 | Paul Johnson                    |          2559
   16 | Vincent Pottier                 |          2516
   17 | Markus                          |          2506
   18 | Simone Cortesi                  |          2415
   19 | Richard Fairhurst               |          2393
   20 | Bernd Wurst                     |          2347

I will publish more results of more queries soon. If you have any suggestions what to query, write a comment to this user diary entry.

The Python script I used is quite ugly. That's why I have not published its source code. However, if you are interested in doing similar analysis, write me an email and I will send it to you. The same applies for the dump of the database.

Crimea dispute

Posted by ika-chan! on 2 December 2018 in English (English)

I have not really been following the OSM dispute over the borders of Crimea for health reasons, but I think that the problem lies with how we portray disputed borders and how we are not really providing the appropriate tag schemes to make it easy for data users to portray borders as they see fit.

I do not know if anyone is considering proposing the appropriate tag schemes that make it easy for data users to portray borders as they see fit, instead of arguing over the "on the ground" rule and whether Crimea belongs to Ukraine, Russia or the Isle of Wight (hint: they are both diamond shaped).

As for OSM-carto, the stylesheet should be truly neutral, showing all disputed borders as a dotted line version of the relevant administrative level.

There is a real danger of the project falling apart if we keep limiting ourselves to choosing one side over another. There are people like me who just want to make maps, instead of arguing over politics, and in my case, putting my already fragile mental health in danger.

Meanwhile, how do we get Nominatim to recognise Bir Tawil (3335661) as not part of any country?

Systematizing Turkmenistan's Data

Posted by apm-wa on 2 December 2018 in English (English)

All 13 border crossings between Turkmenistan and its neighbors are now mapped, named, tagged. All but one of the known national highways has been drawn, tagged, and had relations established (the missing one is in a border zone and I cannot drive on it without special permission). Next I have gone down the list of municipalities in search of a) missing cities and towns, and b) mistagged and misnamed cities and towns. Some bear names from the Soviet period, others have not been updated since the current government either restored an old name or assigned a new one. Slowly but surely the data in the Turkmenistan map are improving. The next editions of Lambertus' Garmin map, MAPS.ME, and Pocket Earth should have much improved routable maps for Turkmenistan.

When Ann and I started this 3-1/2 years ago we focused on simply getting street names and POIs entered--raw data were lacking. Increasingly we find ourselves doing quality control, cross-checking names of municipalities with locals who live there and can tell us both what is on the sign and what it used to be called. For example, I suspected that "Imeni Kirova" had been renamed because it is not listed in "Districts in Turkmenistan" and Kirov was a Russian Bolshevik. Sure enough, a check with a resident of that area revealed a new name for the municipality. Some municipality names are transliterations from the Russian and thus are not what is on the signs, which are all Turkmenized. This takes detective work, but again, slowly but surely it is paying off with improvement of the database. I take all this as a sign of progress.

OSM for Virtual Reality

Posted by escallic on 2 December 2018 in English (English)

I just discovered webglearth.com, which is similar to osmgo.org. Both 3D renderers are open source web applications that have been in development for many years. These renderers can help us understand data in new ways. Each differs by what is rendered and by navigation style.

In WebGL Earth, 3D terrain is visible on the globe after zooming in. A click and drag moves the camera along the ground. Along with two modifier keys, the Alt key rotates the camera from a fixed position to allow looking around, and the Ctrl/Shift key rotates the visible surface about a fixed distance to allow orbiting.

In OSM Go, every building, tree, fence, utility pole, hedge, etc. is rendered as 3D. Selecting an object will display info and tags for that feature. Moving along the ground is possible with arrow keys. Holding shift allows looking around. Additionally, PgUp/ PgDn moves the camera up/down.

Neither project overlaps with the other. WebGL Earth considers a holistic reality by animating the earth. It has the potential to provide APIs for showing interplanetary trajectories. Meanwhile, OSM Go renders every urban cartographic feature, including underground ones. Its interactivity feels like the precursor to a Sims-like garden and floor plan modeler.

Location: Avannaata, Greenland

Please check NH 930 near Bhadravati

Posted by aintgd on 2 December 2018 in English (English)

I traveled on this highway yesterday and uploaded a GPS trace https://www.openstreetmap.org/user/aintgd/traces/2854472. To my surprise, NH 930 is mapped from a very different route near Bhadravati (at the start of above trace). Probably, the route on OSM is an older one. Can someone please check? (Google shows the correct road as per GPS)

I don't want to touch an important highway myself.

First Editing Attempt

Posted by Toby H Ferguson on 1 December 2018 in English (English)

Today is my first day at editing. I'm planning on adding some mountain bike routes to part of Fort Ord, in CA. We'll see how that goes :-)

OSM 0, Google 1

Posted by alexkemp on 1 December 2018 in English (English)

As I was walking through NG3 Nottingham this afternoon, on my way towards Tesco at NG1, I was stopped by a chap in an electric car with his wife & 2 kids (lots of electric cars in Nottingham these days) and asked if I knew where Hartley Road was. They were trying to get to the House of Pain Wrestling event in St Peter's Church Hall.

I have mapped most of NG3, 4 & 5 in Nottingham, but not NG7. He had internet on his mobile so I suggested that he bring up osm.org in his browser. To my surprise, this brought up a long list of short lines of text (I could not read them) (desktop OSM brings up the map) but after a short while he got the map of England up. I told him confidently that he could now search for Hartley Road, but he could not find a search box nor any link to a search box. By this time his wife had used Google to locate the road & also discover the route from ourselves to it (3½ miles). He thanked me and set off.

Oops. Google has got user needs & interactions well worked out, whilst OSM struggles in it's dust to even get started.

Location: Radford, Wollaton, City of Nottingham, East Midlands, England, NG7 3HX, United Kingdom