Diary Entries in English

Recent diary entries


Posted by tregeagle on 30 October 2014 in English (English)

I recently finished my 2nd year of Uni (Nursing) and am now having a break. It's hard to stop being busy so I have been out walking a fair bit and recording traces to stick into OSM. I was inspired by volunteering for The Humanitarian OpenStreetMap Team - creating maps for aide workers around the world.

It's kind of addictive, mapping stuff. I would like to encourage others to contribute it's easy once you know what to do. Maybe I'll arrange a meet-up over the summer....

Location: Bayldon, Toormina, New South Wales, 2452, Australia

Rendering Oddness for Runway Refs

Posted by Vid the Kid on 30 October 2014 in English (English)

It seems rendering of ref tags on aeroway=runway is slightly broken. Runways always have two identifiers (one for each direction) and these are typically written together in the ref tag separated by a semicolon. It appears this semicolon is being replaced by a linefeed character, which makes sense for the current highway shield rendering style. But on runways, the value of the ref tag is still drawn along the way, and multiline text along a path doesn't seem to work, so instead we get a placeholder glyph that looks like a tiny "LF" in a box.

I think if we could get that semicolon replaced by an em dash ("—") instead (only for aeroway=runway of course) we'd have a satisfactory resolution.

IndoorOSM and Wheelchair-Navigation

Posted by babalone on 29 October 2014 in English (English)

I'm currently working on a project at my university where we try to build a proof-of-concept-app (android) for indoor navigation with focus on wheelchair-users and their needs. Wheelchair-users have special needs as the width of a door might be to small for the wheelchair, stairs are not usable, ramps might be too steep, find an alternative route if an elevator is 'out of order'. We are still in the planning phase and currently evaluating what technologies we will use. To create maps of the buildings we will most likely use IndoorOSM-features [0]. The plan is to create the building-maps from emergency-maps (German Wikipedia [1]) if we don't get better ones. We will use those maps as background in JOSM [2] and draw the buildings with it. As those maps are created by following an official norm I hope, that there is no licensing problem with it as they are not a product of creativity or invention.

There is already a similar app for outdoor-wheelchair-navigation built at our university, but it is not officially published. We are adding this feature of indoor-navigation and both project will be merged. There are already industry-partners, who are interested in this app. This app will very likely be published some time and not just stay a research-project.


Today's stats

Posted by AndrewSnow on 29 October 2014 in English (English)

In my stats today on Pascal Neis' site, I reached 911 in both nodes and ways. A fitting milestone for triage work done in the unkempt corners of the map, clearing Tiger debris.

Ethiopia, Sudan, Nicaragua...

Posted by bdiscoe on 29 October 2014 in English (English)

Some recent work i'm proud of:

  1. Fixed the tags (and in some cases the boundaries) of all of Ethiopia's national parks, including Gambella, Bale Mountains, Awash, etc. I even added the Alatish National Park which was entirely missing.

  2. Nearby on the Ethiopia/Sudan border, improved the area where they are building the Grand Ethiopian Renaissance Dam on the Blue Nile.

  3. In Sudan, added the massive Khashm el-Girba Reservoir and nearby city of Al-Qadarif which needed lots of work.

  4. A large number of waterways in the wild eastern parts of Nicaragua (like here) and Honduras (around here), although sadly most of the streams aren't visible until zoom level 13.

  5. Just now, a complex relation for the Las Trampas Regional Wilderness, near San Ramon, CA, USA

Unofficial Bus station/bus stop

Posted by sylverhawk on 28 October 2014 in English (English)

Most Buses start their journey from here though they come from other places and may be going through that point as a bus stop. so it's both a bus stop and bus station

Location: Tabora, Ga West, Greater Accra Region, Ghana

Can anyone explain this?

Posted by dcp on 28 October 2014 in English (English)

How about: "A school for OSM newbies"

The trouble with the ODbL - summarized

Posted by lxbarth on 28 October 2014 in English (English)

Kevin Pomfret from the Centre of Spatial Law just published The ODbL and OpenStreetMap: Analysis and Use Cases a white paper reviewing pain points in the ODbL - OpenStreetMap's current license.

2.5 billion OpenStreetMap nodes!

The paper provides a comprehensive review of issues broached in talks at State of the Map US (More Open, OpenStreetMap Data in Production) and State of the Map EU (The State of the License) and discussions thereafter. It offers an assessment of legal risks and includes a series of case studies focusing on legitimate use cases of OpenStreetMap that are currently impeded or complicated by the ODbL. At both State of the Map conferences I have heard requests from the Licensing Working Group, the OpenStreetMap Foundation board and others for a more solid summary of problems and actual real world use cases that are impeded by the license. This is why over here at Mapbox we have supported the Centre of Spatial Law to compile this white paper.

Here's an overview of issues identified by the paper in order of appearance in the ODbL license:

  1. License does not cover contents - the ODbL covers the database, but not its contents. OpenStreetMap does not make clear under what conditions the actual contents of the OpenStreetMap database are available.
  2. Rights of contributors is uncertain - neither the ODbL nor the Contributor Terms protect a licensee from third party intellectual property claims. Note that a third party here is not limited to contributors, but would also include parties whose data has been imported to OpenStreetMap.
  3. Uncertain if and to what extent "share-alike" applies - the delineation between Produced Work and Derived Database is fuzzy and the crucial concept of Substantial is entirely undefined. This makes the extent to which share-alike applies to data that is combined with OpenStreetMap data guesswork.
  4. Uncertainty as to which jurisdiction's law applies - the ODbL states it will be governed by the laws of the relevant jurisdiction in which the License terms are sought to be enforced. - the global nature of OpenStreetMap together with (1) makes it unpredictable as to in which jurisdictions to expect claims.
  5. Lack of a cure period for a breach - there's no grace period to make amends. If you're in breach of the license you have to stop using it right away.
  6. Unclear governance - there is no authority to ask for definitive clarifications around the license. When posing questions on related mailing lists or the OpenStreetMap Foundation the standing practice is to defer to license interpretation and non-existing case law.

The paper's case studies illustrate how potential OpenStreetMap users don't use OpenStreetMap at all - or not to the extent they could - due to the problems outlined above. This is a crucial issue - we're not a community of givers on the one side and takers on the other, there's a large overlap between data users and data contributors and the more we can get OpenStreetMap used in the real world, the more exposure we have to potential contributors, the more contributors we'll have.

Here are some of the case studies:

Yale University does not use OpenStreetMap in research under HIPAA or similar privacy regimens because of concerns that ODbL's share alike provisions could force researchers to open sensitive data - for instance when geocoding research data with OpenStreetMap data. This example highlights the issues with share alike (3) but also with governance (6). Some of the concerns expressed by Yale may be based on a conservative reading of the ODbL, but the absence of license governance in OpenStreetMap (6) and the understandable desire to avoid any risk of violating federal law rule out OpenStreetMap as an option where it should be a prime candidate.

As the Wikimedia Foundation is exploring opportunities to integrate tighter with OpenStreetMap they are running into incompatibilities between Wikipedia's CC-BY-SA license, Wikidata's CC0 license and the OpenStreetMap's ODbL. It should be a no brainer that OpenStreetMap and Wikipedia should work as close as possible with each other for the benefit of both projects. Maybe a good real world use case we can get all moving on?

Foursquare is not using OpenStreetMap for reverse geocoding where they could due to concerns about share-alike extending to Foursquare data. Foursquare has been an awesome engine for driving people to become contributors and they show willingness to contribute data but can't commit if the extent of the commitment is not clear. This is a great example of where we're loosing out on contributions with a license that tries to take it all. To hear it directly from the source, listen to Dave Blackman's talk at State of the Map US.

The National Park Service is working on standing up their own OpenStreetMap like service where they could otherwise use OpenStreetMap directly to power Park Service maps. This is due to the fact that OpenStreetMap's share alike provisions are not compatible with the National Park Service's policy to keep their data in the public domain.

What next?

For a full read of the white paper, head over to the Centre for Spatial Law blog.


Posted by SimonPoole on 27 October 2014 in English (English)

One of Steves favourite sound bites is “we have the money, lets employ somebody” in a couple of variants, used again in his current manifesto to convince people in to electing him back to the OSMF board.

During my time I simply ignored them, not wanting to derail things more than they typically were, but the obvious response would have been “how on earth would you know?” (well that is actually the polite version).

The OSMF has limited financial reporting (it has improved over the last two years) and essentially no planning, also known as a budget. It is currently simply not possible to know what the financial status of the OSMF is or should be at point in time in the future. And while the OSMF has roughly £80'000 of cash available we shouldn't be eating too much in to that given that substantial amounts of it would be needed in case of larger hardware problems (imagine one of our hosting locations catching fire) or other shortfalls and unexpected costs. Not even mentioning any kind of planning for short term cash needs.

It is one of the things that I consider a personal fail that we didn't manage to get something resembling a conventional budget in place over the last two years. It is not difficult, but it would have required cooperation from multiple board members to be actually meaningful and that was not forthcoming and so I concentrated on other pressing issues.

One of the larger planning issues is the largest line item, SOTM. Right now a couple of days before this years event, the OSMF board has no idea what has been signed in its name (if anything has been signed that is), what the potential risks are, nothing. And this was the same the previous year, the year before, the year before that and so on ….

Last year the board faced a potential GAU (German expression for the largest conceivable accident) when the local organizers for Birmingham threatened to walk out, we patched that up, but till this day we don't quite know how much we would have been in the red had the worst case happened. [I would like to point out that the local team was doing the right thing. I was simply showing that, yes, even "safe" things go wrong, and without planning you can't even do "what if"s in such a situation]

Now there is quite a lot of potential sources of funding available for OSM, but we are kidding ourselves if we think it is going to rain manna if we don't get our act together, get our financials under control and actually have projects worth funding (a separate topic).

To be clear, from a legal point of view there is nothing wrong with the OSMF finances and the way we have financed hardware purchases in the past has been low risk, leaving SOTM and our G&A as the larger variables. But it has trapped us in a vicious circle.

Rebooting (with an emphasis on boot) the OSMF board might be a way of breaking out, installing a Junta with the person mainly responsible for the current state of affairs in charge, not.

Middelheim museum, Antwerp

Posted by escada on 27 October 2014 in English (English)

I have regularly visited the Middelheim museum in Antwerp. It is part of a park, free to visit, dogs are allowed (on the lead) and it's not too far away. Until 2 weeks ago I never bothered to map it. it was just of of those places that I visit.

Two weeks ago, we had a meetup in Antwerp. One of the topics we discussed was mapping the Antwerp Zoo. There I got the idea to map the museum in more detail.

So after 2 visits, I have mapped about 2/3 of the artwork (statues, sculptures, constructions). For each piece I recorded the name, the artist and the "construction" date as indicated on the information panels next to each item.

This is the map so far

I cannot add pictures of all items, as in Belgium we do not have freedom-of-panorama. I could have done it for older work, but I tend to forget the exact age.

I hope to add the last third with my next visit.

Location: Middelheim, Antwerpen, Antwerp, Flanders, 2020, Belgium


Posted by SimonPoole on 27 October 2014 in English (English)

In 2011 the then current board had a face to face meeting in Seattle and produced a set of "audacious goals" for OSM that were

  • The World’s Most Used Map
  • More Than Just Streets
  • Cultivating Leadership of Mappers
  • Easier Contribution for Non-Geeks

Now some of these would have been directly actionable, but the first two are really "visions" under a different name. And as visions goes they are quite good, far out goal posts that nobody expects to be achieved soon.

And we are still far away from reaching the first goal . There are lots of pieces missing before we can remotely hope to achieve it, but we are on the way.

Steve has long harped about adding more addresses to OSM, matter of fact I have too. From a pure usability point of view if you want to produce a competitive device or app with door to door navigation support you are going to need address data. To be more precise this boils down to adding hose/building numbers with their associated streets (intentional pun) to our dataset in some form.

Note on the side: there are lots of regions that don't have a conventional western way of describing locations, we don't have good support for that yet in OSM, solving the “address problem” tends to centred around 1st world countries.

The OSM community has lots of experience with adding addresses by both on the ground surveying and imports of suitable open data, matter of fact we've had a full country “complete” for half a decade now. What is however undeniable is that it is slow progress, even importing a couple of million addresses (we don't have good numbers, but it is likely that there is something between half and one billion address conventional house addresses out there) takes a lot of time if you want to provide some minimum quality and the preferable on the ground surveying tends to be even slower.

But we are making progress and looking at one of the larger countries, Germany, with 81 million population, we are now at roughly 1/3 coverage with a combination of on the ground survey supported by open data and smaller imports, completion likely in 2 years at the current rate.

There are some places where we don't have a good handle on the issue, for example in the “original OSM country”, the UK, due to the addressing system revolving mainly around non-surveyable proprietary schemes and house numbers playing a secondary role. But we are not the only group feeling the pain there and I'm optimistic that we will find a way out, and if it is simply by replacing such proprietary systems.

To summarize: addresses are important and yes it is something that the community is working on with dedication. There is really nothing visionary about it at all at this point in time, not more than “lets map all roads in Germany”.


Steve has a legitimate commercial itch that he wants scratched and he wants that fast. This is not unique, readers following the “imports” mailing list have seen similar requests from HOT, matter of fact one of the HOT requests had that other problematic attribute “non-editable”.

Luckily HOT hasn't tried to stage a coup d’état to get their way. But what it does tell us is that the OSM data format, its tool chains, its editors and other applications have become extremely popular and people prefer our tools over others. It is a great testimonial to OSM.

OSM was built around the notion of mappers collecting or curating data and then adding it to our central repository, iteratively improving the quality and completeness over time. “fast” and “non-editable” are the antithesis to what normal OSM is about.

Now I'm sure ESRI doesn't get many complaints about them not providing the ultimate multi-100 GB all free geo-data of the world shape file that includes everything, so why do we? I can only guess that it is because people are feeling left out and think that if their favourite dataset is not included that they are not part of this great community.

But, really, it is just a superficial marketing and packaging issue.

If Steve or better his employer want to create the


containing a quickly thrown together conversion of all the open data address sets in the US, great, the OSMF might even be convinced to distribute it in the same place as the planet dumps. And we have more than enough id space to support conflict free merging.

There is simply no need to compromise the normal OSM community process for a short term gain.

We don't only have hammers, please stop just seeing nails.

OSMF Board of Directors Election

Posted by FTA on 26 October 2014 in English (English)

I decided I wanted to take my OSM involvement to the next level and run in the next OSMF Board of Directors election. Please check out my OSM wiki page for information about my background, why I am running, and some of the things I would like to do.

Location: Maney Hill Road, Birmingham, Warwickshire, West Midlands, England, B72, United Kingdom

Entered minor changes, JOSM appeared to hang

Posted by Richard Hufford on 26 October 2014 in English (English)

I entered a few corrections near Geronimo, Arizona, because I drove through the intersection of Geronimo Lane and Black Lane and it was a little bit different from what the map showed. When I closed JOSM, selecting the Perform Changes button, JOSM appeared to hang with the message "Preparing layer xxx for upload." showing on the dialog window. It appears that the changes were successfully uploaded. I will check in a day or two to see that my changes are intact. Note that I updated my Java version today (Oct 26, 2014) before running JOSM.

I should mention that I'm using Windows 7, 64-bit

Location: Geronimo Lane, Emery, Graham County, Arizona, United States of America

Dutton Park Cemetery

Posted by ShaunKolo on 26 October 2014 in English (English)

Corrected a number of incorrect roads within the Dutton Park Cemetery after walking the centrelines with a GPS enabled smartphone and the TwoNav mapping software.

Location: South Brisbane Cemetery, Dutton Park, Brisbane, Queensland, 4072, Australia

Thank you for your trust!

Posted by SimonPoole on 26 October 2014 in English (English)

Rewind two years, September 2012. I'm rather fuzzy on why I stood for the OSMF board in the first place, but I remember at least being annoyed by the treatment of the working groups by the board, experiencing first hand a dress down of one our most respected community members just because he didn't jump when the then chairperson said jump. And yes something didn't feel quite right about the organisation.

To make a long story short, I wasn't expecting to end up being elected the chairperson and further was expecting the fault lines inside the board to be in other places than they turned out to be. Naive me assumed that we would be having heated discussions about the scope of services the OSMF provides and how we delineate what the OSMF does from commercial and other services providers, how to grow the OSMF membership base and how we could best support the wider OSM community. Alas things were not quite ready for that.

My approach was let the past be the past, fix the issues and then move on. The net result was an uphill battle every foot of the way over the last two years and for the moving on part, well that might have happened now.

Why did I step down?

No, I didn't step down because of some flak on the mailing lists. I'm a big boy (well I am kind of small actually) and can easily take the heat. A consequence of exposing yourself and engaging with the community is that you tend to function as a focal point for everything real or just perceived wrong about the organisation you are representing, it simply comes with the territory. But I still think it is necessary out of respect to the OSMF members to be open about what I think and where I believe the sticky issues are, instead of spewing safe political correct canned sound bites or hiding away somewhere.

Steve Coast has decided to stage a come back after two years as chairman emeritus, as board member and chairperson that put me between a rock and a hard place. On the one hand I have to respect our democratic processes and not interfere with the ongoing election, on the other hand I do hold strong opinions on where we should be heading and in what fashion.

To make things clear: I consider it a very silly thing for Steve to do, bad for him, for the OSMF and for the whole project.

As the board member that has driven most of the change over the last two years, I see the danger of old alliances undoing the baby steps towards a more mature organisation we have achieved. An organisation that could actually engage in all the things the current board candidates envision. An organisation that holds a stable course and can function as a trustworthy intermediary between our fantastic community and the rest of the world.

I stepped down to make clear this is not about power, or my personal position, but about what is best for OSM.

Somebody who has been to a recent talk of mine on OSM might remember a slide representing OSMspace, showing some of our fairly complicated interactions and overlaps with the outside world, the OSMF a tiny speck somewhere off centre. Yes the OSMF is just a tiny piece of the puzzle with limited influence on the project as a whole. Most of our contributors will likely never know that it exists, no different than the Wikimedia Foundation. What is different is that OSM is a significant player in a multi-billion dollar market and the outside forces are of a very different nature.

While the OSMF is just that tiny speck in the OSM ocean it does hold the keys to our core product and with control over the foundation it is completely possible to misuse that. Yes, even handed, community control of the OSMF is important for everybody in OSMspace, and it scares me when somebody seriously proposes handing those keys to a Junta, even if it is “only for a year”.

I would like to end for now with thanking everybody that was kind enough to show me support over the last few days it is much appreciated.

More to come .....

seeking Administrator

Posted by သိန်း​အောင် on 25 October 2014 in English (English)


Please advise me how to communicate with Administrator ?



Posted by SimonPoole on 25 October 2014 in English (English)

A bit over 10 months ago the SOSM board decided that we would try and systematically send a welcome note to every new contributor in Switzerland Today I just sent off the 1000th such message, not a big deal, takes a couple of seconds per mail and I typically send them every 2nd day or so. The mail (in 4 languages, really should be in five), tries to be as low key as possible and simply welcomes the new mapper and points out our local and international resources.


  • there hasn't been a noticeable increase in SOSM membership signups, on the one hand 10 months is too short term to expect that to happen, and on the other hand we don't actively solicit that in the mail anyway.
  • just as above direct feedback is quite rare.
  • the "one edit user" seems to be on the way out, most of the users already have more than one changeset by the time we send the mail (nailing that down in hard numbers is a project for another day).
  • given that Switzerland is rather small (8 million population), well mapped (as all DACH countries) you wouldn't be surprised to see some indications of saturation after 9 years, but there are no such effects noticeable at all.

If you think this is a good idea and are considering doing something similar yourself, please note:

  • in most countries you will want to split the work up and coordinate, swamping new users with mail is likely not a good idea.
  • Pascal Neis provides a RSS with new users on a country by country base here:
  • users signing up to OSM don't have to agree to receiving unsolicited mail, matter of fact there is currently no way that they can opt-in at all. This implies that you should consider the legal implications in whatever jurisdiction you are and be careful how you word your mails. I haven't heard of any problems, but you never know.

As a final note: most old timers know about or have participated in the reoccurring discussions on why people sign up for an OSM account and then never edit (there is currently no way of getting zero edits sign ups per region, so you can't contact those). I'm personally fairly sure that we have quite a good conversion rate, there are simply no comparable numbers from other projects available to prove that, others disagree. But as this example shows in the end we get them all :-).

PS: I should point out that by no way this is particularly original, we have had users doing this for a long time, it should be seen more as a reminder that this can be done.

HOT mapping with iD made a little bit smoother - pre-fill changeset comments

Posted by mcld on 25 October 2014 in English (English)

When we run mapping parties as part of the HOT work, we see lots and lots of newcomers mapping for the first time. Increasingly we're getting them using iD which is very easy for them to get started with.

One little issue I noticed in sessions is that for HOT we ask people to use very specific changeset comments - essentially to "tag" the changesets as belonging to a particular labelled task. It was very easy for people to spend half an hour mapping and after half an hour have no memory of what we said about copying-and-pasting a specific comment. Workflow problem!

Now, the team who create the iD editor kindly added my feature request which means that the HOT Tasking Manager can now "pre-fill" the changeset comment in the iD editor. So no need to copy and paste, it should be there when you click through from the Tasking Manager.

What does this mean? It means that in future, HOT mappers using iD will not need any reminding about what to put in the comment box! Easier mapping, easier training, more consistent changeset comments.

Thanks everyone who helped put this through.

(P.S. There is one little technical niggle to resolve - if the comment contains an equals sign then the pre-fill doesn't work on firefox. Hopefully sorted soon.)

Personal benefits of contributing to OSM

Posted by bdhurkett on 25 October 2014 in English (English)

The big goal of OpenStreetMap and the various themed projects is to provide a free & editable map of the world, or something like that, which is great (and so is the progress). But altruism isn't the only reason I've found myself continuing to edit and contribute for the last 18 months or so.


I suppose this is dependent on how and where you're mapping. WIth the streets and major items filled out in my local area, I've moved on to the smaller things like addresses and sidewalks and even benches and picnic tables. That sort of stuff can't really be done at typical driving speed, and it's a lot more relaxing to just go for a stroll with a camera or notepad. I don't know if it's made a huge difference in terms of activity, but it sure beats watching TV on the couch. And it makes for good motivation - sometimes I can't be bothered walking around the block a few times, but who knows what might be on an unseen street! Which brings me to another benefit...

Exploration & Awareness

Generally if I went somewhere in the past I wouldn't dawdle - the journey was just time to kill until I got to the destination. Now though, even when I'm not directly mapping I'm more likely to take a different route, or see what's down that side-street I've driven past hundreds of times but never actually needed to traverse, or pull over and take a look at the view instead of glancing out the car window. And though I'm still happy to let my mind wander on a good relaxing walk, I generally pay more attention to my surroundings, whether it's an unsigned shortcut to the next street, or smaller stuff like "that's a lovely garden" or "this would make a great photo". Even on a larger scale, it's a good excuse to go for a trip to a neighbouring town or park - not something particularly unusual, perhaps, but at least something I find myself doing more. And even if I get a little carried away or too far off the beaten track, it's less of a problem, thanks to...


All that time looking at street signs and shop fronts in the real world, and digital maps on the computer later, has led to the expected result - I know my area a lot better than I did before. It's just like most other skills, where if you just keep practising and spending time on it you improve, but I doubt I'd have spent nearly as much time "memorising" voluntarily otherwise. My mind's-eye-copy is hardly perfect - I'm still terrible at most of the street names around here - but it's handy to be able to have a pretty good idea of what I can buy where in town, or how to get from here to there in five different ways, or even just what's over the next hill.

Oh, and knowing something I drew is on maps and websites all over the place is pretty nice too.

Who uses OpenStreetMap?

Posted by dalek2point3 on 24 October 2014 in English (English)

I have recently been interested in measuring how openstreetmap is being used in different services around the world. Now obviously, this is a very hard question to answer, because, being an open project, OSM data can be downloaded at any point in time, and you can start playing around with it. We dont require any permission for this action, and while the Odbl license does require attribution if you use the data in production, such attribution is hard to track. Openstreetmap data can be found on planes, in disaster relief -- not to mention the thousands of web and mobile applications that use it for different intents and purposes.

Alright, having convinced you that its quite hard to track all possible uses of openstreetmap, perhaps it is possible to track usage of OSM tiles in web applications online? Now, while still difficult, this is easier to accomplish, because at the very least the question is well defined, and in theory, answerable. If we could survey each and every website out there, see if they use tiles from an OpenStreetMap server (or Mapbox server) we might be able to say something about OSM usage. Now, this still would not cover cases where folks have set up their own tileserver with OSM data -- which one might argue is a quite common way to use OSM data.

Either way, I recently discovered HTTPArchive and thought it would be a cool project to track the usage of different mapping APIs online, including Mapbox and folks using OpenStreetMap tiles (which you're not supposed to do for heavy usage!). What HTTPArchive does it crawl about the top million websites, and for each website it records the HTTP requests that the site is making. Now, this it turns out, is a great way to measure the usage of javascript frameworks, Google Analytics etc -- except, no one so far has used it to look at mapping APIs and OpenStreetMap in particular!-

So, I thought I would do that! Now, the HTTPArchive data is quite large -- (petabytes I hear) -- but fortunately it's all available on Google Big Query which makes it a cinch to query. Results from now of my explorations are below.

HTTPArchive data is stored in two important tables (two for each "run") -- pages and requests, and the latest versions can be always found at latest_pages and latest_requests. The `pages' tables contains information like the url scraped, number of bytes etc. Lets see if the main openstreetmap website is in the data. The following query does the job:

select pageid, url,urlShort from httparchive:runs.latest_pages
where  REGEXP_MATCH(urlShort, r'');

Yes it is! This query returns:

Row pageid url urlShort
1 17330926

Seems like the pageid is 17330926. Now, the `requests' table is where all the juicy information is contained. Lets see what requests, the OSM website makes:

select * from httparchive:runs.latest_requests
where pageid == 17330926;

And this is the response that you get -- about 35 requests. That data is here. As you can see, this includes a number of requests for *, OSM's public tileserver.

Which other websites make similar requests? This is where the HTTPArchive really shines. After some experimentation, the following SQL query does the trick:

  SELECT urlShort FROM [httparchive:runs.latest_pages] as pages JOIN (
    SELECT pageid, REGEXP_EXTRACT(url, r'(') AS link2
    FROM [httparchive:runs.latest_requests] as requests
  ) AS lib ON pages.pageid = lib.pageid
  GROUP BY urlShort;

Not many websites seem to be hitting the tileserver directly -- which is reassuring. That data is here.

A final interesting thing to run would be a similar analysis for Google Maps and Mapbox. Queries for Mapbox and Google Maps are available on Github. And the data from there queries are here -- Mapbox and I'm still working on getting the data from Google Maps API usage. That is for another post!

Hope you will find HTTPArchive a useful tool to analyze data from the web. It certainly seems easy to use and with lots of interesting data for analysis! Happy exploring!

Location: East Cambridge, Cambridge, Middlesex County, Massachusetts, 02142, United States of America
Older Entries | Newer Entries