Recent diary entries
I recently needed to create a big georeferenced image from a mapnik style file, and found out no image exporting tool currently offers more than a direct interface to mapnik's options. That is, I could not get "image in 300 dpi for printing on an A5 sheet", I had to understand what scale_factor is, and what is the default resolution, and why lines had become so thick. And I'm a programmer — imagine a confusion of a regular user!
Today I have published Nik4. It makes everything easier.
Grab a 800x600 image at z13?
-c LON LAT -x 800 600 -z 13 — and no suprises like when the output image differs from osm.org (nik2img puzzled me with that one). Print a region in 300 dpi on A5?
-a 5 --ppi 300 --bbox X1 Y1 X2 Y2. You don't have to think about scale_factor ever. Make a very large image? No problem, use
--tiles 4 and wait a bit; you won't run out of memory.
See https://github.com/Zverik/Nik4 for an extensive description and installation instructions (
easy_install nik4 — there, no more instructions needed). Print more maps.
And for generating tiles there is just as simple polytiles.py.
If you tried rendering quality maps with Mapnik, you know its label placement is awful. Just a bit better than throwing labels on map at random. So for printing, postprocessing in a vector editor is mandatory.
If you tried editing Mapnik-generated SVG, you know it is completely unstructured. Which means, if you need to move a label, first you have to select all of its letters one by one, and then its casing, also letter by letter. And there will be a lot of frustration when you select roads underneath the label, or lose selection.
Yesterday I've made a small script that takes Mapnik SVG, finds labels and wraps them in groups. So when you open that SVG in Inkscape, you will move labels not by letter, but as a whole.
The only parameter, -d, controls the maximum width of a letter, including spacing between letters. The default works for me, but if your labels are tightly packed, or are printed in a very large font, you may need to use that parameter.
On this day last year I announced the Imagery Offset Database: a centralized storage for imagery offsets. It was planned as a way to provide every mapper, especially beginners, with a confidence that they are tracing correctly aligned imagery. And for those not knowing imagery can be misaligned, a way to not ruin a map. After the announce, tens of mappers started entering their offsets into the database, and I've never made a local offset bookmark ever since.
Aerial Imagery cannot be ideally georeferenced. Depending on precision of your measurement tools (GPX traces give 1 to 10 meter precision), you may notice that the imagery layer is positioned incorrectly, and use your editor's controls to shift it to the right place, so GPS traces follow roads and paths on the imagery. This is common knowledge among experienced mappers, and I hope beginner mappers learn that soon enough.
Sadly, looking at new editors I can't but conclude than either there is no misaligned imagery (which is probably false), or many mappers, including those who work on those editors, don't bother with aligning imagery to GPS traces. How many of you pressed those little arrows for shifting a background layer in iD editor? Does your favourite mobile editor, with which you upload POI nodes to the database, account for shifted imagery? Do you make bookmarks of such offsets so you can quickly restore them later, or on a different computer?
A week ago Simon Poole added IODB support to Vespucci, making it the second editor supporting it. This is great news, number of editors supporting the database has doubled overnight. Support in iD has been stalled, and I didn't expect Potlatch or iOS apps to support it. So basically, pay attention and don't be suprised when buildings are misaligned with imagery.
Now some statistics. As of now, there are 5180 non-deprecated entries in the database, of which 4078 (70%) are in ex-USSR countries (3180 in Russia). Mappers add on average 14 entries a day and have not skipped a day yet (see the graph above).
Of 5644 imagery offsets 4606 (82%) are of Bing layer, 714 of Russian and Ukrainian regional imagery, and around 150 offsets refer to some HOT-related imagery. Strangely, there are 44 offset of Google Maps, and 30 — of Yandex's. I assume those users were just testing the database and not uploading anything to OpenStreetMap.
I hope someday we won't need that database, but for now, when only a handful of coutries have precisely aligned imagery, it is one of the most important tools for mappers. It will probably be included in JOSM core later this year, and I'd be glad to hear of more editors supporting it.
And by the way, I always planned to publish the server's source code, but got to it only recently: see it on github.
Quite a lot of news sources have asked almost all cartographic providers if they will change Crimea's owner on their maps. No international provider answered positively. In Russia companies are more willing to comply: Mail.ru maps are showing Crimea as a part of Russia, and Yandex now has two separate maps, for Russians and Ukrainians, which show Crimea in different colors. Wikipedia contributors started a tiny edit war between images that show Crimea as a disputed territory or a part of Russia.
Members of Russian and Ukrainian OpenStreetMap communities have discussed the matter during the first few days of the Crimea situation, and have come to an agreement ten days ago. We declared a moratorium on touching administrative borders in Crimea, as well as name tags (since some mappers started changing Ukrainian names to Russian), active until 1st of June, 0:01 MSK. Until at least then Crimea stays a part of Ukraine on our map. Every edit altering name tags or administrative borders will be treated as provoking an edit war and reverted, users who do that repeatedly will be banned. There were some cases of that already.
Of course users are encouraged to map non-political things, like buildings and landuses, and add language-specific name tags: name:uk, name:ru and name:crh. Right now there is a mapping event in progress, focusing on improving Crimea's coverage on OpenStreetMap, which has attracted around 70 mappers.
For Russian patriots I've made a special political map which shows Russia's position on Crimea, Kosovo, Abkhazia and South Ossetia to users with Russian IP addresses, and a normal map with Ukrainian names for visitors from other countries.
Идёт пятый день эпичного картопроекта «#крым», в рамках которого мы — уже 65 человек — вместо своего двора открываем в JOSM крымские деревни и луга, и обклацываем по бингу и ортофотопланам всё, что видим. На сайте проекта всё написано, и есть карта, где нельзя не заметить разницу между пятничным и сегодняшним покрытием. Иногда мы собираемся сообща улучшить карту города, но впервые поставлена задача обработать целый полуостров, и мы справляемся с ней лучше, чем можно было ожидать.
В этом мероприятии нет политики, если вы не привнесёте её сами: до 31 мая действует мораторий на изменение административных границ и тега name в Крыму. Наша цель — просто собраться вместе и почувствовать, что сообщество OpenStreetMap — это картографическая сила, равной которой нет. Карта Крыма из OSM уже почти на всей территории лучшая в мире, но наша задача — довести её до такого состояния, чтобы мы сами могли ей гордиться.
Картопроект заканчивается вечером 31 марта. Чтобы обозначить себя как участника, в комментарий к одному из загружаемых пакетов правок добавьте хэштег #крым или #крим. Обсуждение идёт на форуме.
Olympic facilities being mapped really good in OpenStreetMap has been a subject of many articles recently, mainly with comparison to Google Maps (which are nearly empty, of course, unlike Yandex Maps). This has been good for OSM awareness, and Russian mappers welcome that. In Russia a lot of Olympics-related projects (including the torch relay) were using OpenStreetMap, and most of them even attributed us correctly.
But there are comments that bug me, which imply that, as it usually happens with humanitarian response, in case of Olympics some mappers just organized themselves on a weekend and mapped the whole area from Bing imagery and probably official data (of which there is none, by the way). That is completely untrue: we had no mapping parties or other events targeted at mapping the region.
This chart shows number of changes (identified by num_changes XML attribute) for the region containing Sochi, Adler and Krasnaya Polyana. Only small changesets were counted, which do not cover other regions. You can clearly see that the region was mapped mostly by agsochi (83% of all changes) four years ago (with 185K changes in March 2010), and then gradually updated to reflect current situation.
This kind of chart is typical for many regions with active mappers: someone maps everything when they first find the area, and then update it through the years. It just happened that Olympic Games were held in a well-mapped region, not the other way around.
Other contributors with more than 5 thousand changes are:
- Sergey Astakhov — 3.3%
- raw2008 — 2.2%
- garl — 1.4%
- -James- — 1.3%
- ulil — 0.77%
- СПТ Сочи — 0.77%
The last one is notable in that it is an account for Sochi's fire brigade, which since April 2013 has been constantly adding hydrants and fire stations in the region. You can see the outcome of their work on OpenFireMap.
Yesterday I've created another repository in MapBBCode project. It is a small script, which does great things.
MapBBCode Loader searches for BBCode on the current page, and if it succeeds, it appends Leaflet and MapBBCode libraries and replaces all [map] bbcode with maps. And if there is a specially marked button, the script adds a map editor to it. Basically, this script allows integrating MapBBCode into forum and blog engines which do not have native plugins for that yet.
I guess nothing can stop MapBBCode from conquering the internet. Even closed-source forum engines (I'm looking at you, vBulletin, which requested $250 license for access to documentation) and lazy administrators. The complete instruction of five simple steps for embedding MapBBCode anywhere is published in the GitHub repo.
The loader requires MapBBCode 1.2, which is now in development and may contains traces of bugs.
But that is not the point: the loader is not supposed to power such one-off demostrations. It serves as a plugin replacement for engines that are hard to modify. For example, shtosm.ru is powered by a closed-source engine, and only MapBBCodeLoader has allowed for maps in it.
Yesterday a new version of MapBBCode library was released, 1.1.1, and plugins for 4 forum engines were updated to use it. Full changelog is on github, here is a list of visible changes:
- Documentation was moved to the official website and can be translated. There are Russian versions of some pages. Also I plan to write several guides, including a user's guide and a programmer's tutorial.
- OpenStreetMap attribution link not only opens our website, but opens it at the location displayed on a map. A kind of permalink. It is the 8th leaflet plugin made for this library, and, like previous seven, can be used separately.
- Some users requested zooming with a scroll wheel. It was turned off because it disrupts page scrolling, but I was hinted at the solution. Now you can zoom with mouse wheel after clicking on a map.
- Adding proprietary layers, like Google's, has been made extremely simple. On some forums it's just a matter of installing an add-on or uncommenting a line in plugin's code. But since the library was made for popularizing OpenStreetMap, there is a restriction: non-OSM based layer cannot be made the default one.
During those two weeks, the MapBBCode plugins were installed on several forums, including the biggest offroading forum in Russia (Skif 4x4) and the second biggest cycling forum (Velopiter). I can't thank enough administrators of those forums. Shoorick and DKiselev have written simple unconfigurable plugins for WordPress and FluxBB, which are being tried by some people. In a week or two I plan to release a vBulletin plugin, which would allow granting maps to the biggest forums, like SkyscraperCity, Velomania and Drom.ru, with hundreds thousands of users.
It is now time to mark a milestone in my month-long programming marathon and announce the first release of the MapBBCode library: a way to integrate maps into almost everything, but primarily written for forums and blogs. There is a long introduction on the official site, but the problem it solves is very common and somewhat annoying, especially with biking and construction sites: they have no maps.
Mapping plugins for forums tend to focus on geolocating users, which is a dull and useless task. What if I want to draw a route and allow fellow bikers to download it as GPX before setting off? The world of embedded maps is dominated by monstrosities, like the one I've spotted in Shaun's blog (40€ for a map!). We have Leaflet, we have the best map in the world, and forum/blogging engines have API — let's get coding.
As an example of library's versatility, MapBBCode Share website was born. The idea was simple: put strings in a database — and you've got a map sharing service, simple and accessible, like pastebin or imgur. And you can copy raw code to a forum or a blog. This vision started to shatter when GIS-Lab community used the site for planning a mapping party by openly publishing an edit link. And then I added import and export to a dozen of formats, personal libraries (so you won't have to store bookmarks locally) and API. It turned out that sharing maps is only a fraction of the site's potential.
Now you can plan a trip together with friends, place markers, embed the developing map in a forum post, and a day before the trip you can download the route and waypoints in GPX format right from a forum page. You can import CSV and immediately download it as HTML with no external dependencies (besides Leaflet). No iframes — unlike every other service, MapBBCode Share produces full-featured HTML that works offline and can be included in a website by copying code. You can upload OSRM-generated routes, quickly edit poly files, inevitably export GeoJSON, — the site "competes" not only with geojson.io and uMap, but even with JOSM. Privacy is bulletproof: you sign in with OpenID (google/yahoo accounts will do), your ID is immediately hashed before storing, so even the administrator doesn't know who is using the website. The hash doesn't leave the server, so no one can link maps to users.
Today I am releasing the whole package: more than 200K of code, 60K of documentation, 7 new Leaflet plugins, everything published under the most permissive license of all, WTFPL. The development goes on, but now all changes are logged, and the author is responsible for keeping backwards compatibility. Please use MapBBCode Share and write plugins for embedding the code into forum and blog engines. Let's make the [map] tag a standard, and make sharing maps no harder than sharing images or code.
Someone may help if you are obviously not a programmer, but god forbid you write a single script — you're done, from now on <see above>.
(a summary of my sotm talk, proved again this week)
While participants are already home and preparing for State of the Map in Birmingham, organizers of SotM Baltics 2013 are still working, so the conference website doesn't get old and obsolete. We made a special page with links to all media mentions, photos and blog posts on SotM Baltics. If you blogged about it or forgot to publish some photos, now is the time.
We have our channel on Vimeo where new video recordings of talks are published daily. They are announced in the twitter and added to the schedule along with slides. At this rate you will manage to watch most of videos before the main SotM.
It's hard to make recommendations, because you would surely skip talks not mentioned, but if you have time only for one video, do watch "Jokes about Estonians", which will be published on Friday. It's not related to OSM, but we are glad it found a way into our schedule.
Between two grand conferences, states of the maps, there is a smaller one, just a bit more that a meeting of mappers, although from several countries. Hosted in Tartu this year, SotM Baltics will not only allow you to test and perfect your presentation for the main State of the Map event, but also get aquainted with mappers and OpenStreetMap users from "distant" countries like Latvia, Estonia, Finland and Russia. During two days of the conference you'll see the interior of the biggest university in Estonia, enjoy a summer city with all signs written in a strange language, and have a couple of fun events to attend while you are discussing the imminent fall of OSM with other community members.
So, this leads me to an announcement that today is the last day of the call for presentations! I know I should have been spamming all of you the whole week, like other conference organizers did. But still, you have the whole day to think up a title, it's not like we're requesting anyone to submit the whole presentation. There will be a lot less pressure on the stage compared to hundred-persons halls of bigger SotMs, so this is a prefect opportunity to get heard and discuss important topics with mappers who aren't as active in mailing lists.
State of the Map Baltics 2013 will be held on 3-4 of August in Tartu, Estonia.
On 27th Steve Coast wrote in his blog a big speech about people who take the sole responsibility for important elements of infrastructure in crowdsourced projects. He tells of points in OpenStreetMap which are controlled by indispensable people, and other projects members acknowledge that they have no idea what to do if those people would quit. It's time, Steve says, for them to leave, so we could «find out how we distribute the workload now while we only have 900,000 registered accounts instead of 9 million».
This is a familiar subject for reliability experts, the Single Point of Failure talk. In crowdsourced projects such points are people, of course. And indeed, several years ago only one man worked on Potlatch, another one made OWL, the mapnik style was controlled by third, and this chain of indispensable people went on and on. But Richard Fairhurst correctly notices in comments that today in all those control points communities have grown, and indispensable people are not indispensable anymore. Actually, I can name only one SPF in our project from the top of my head. And it's Steve himself.
Who is contacted by media when OSM has another victory over some proprietary cartographic service? Who is deemed visionary, who opens every State of the Map conference? Who our project is associated with by outsiders? Is it someone from the press contacts? No. It is the Chairman Emeritus of the OSMF Board and the Inventor of OpenStreetMap. What is his recent archievement in the project? On his site there is a list: "founded OSM" (2004), "started SotM" (2007), "co-founded Cloudmade" (2007 again), "kicked off mapstraction" (2010, take a look at djeo btw). Which are his latest two projects? FrontDoor, which allows to fix houses' entry points using Bing imagery. Last month James Fee asked Steve, where are the results. He evaded the subject. And OpenGeoCoder, for which Steve specifically chose not to use OSM because of Share-Alike license. His last edit to the map was in February 2011.
You know how most active people in our community respond to imports. Only recently there was a major flame war in mailing lists concerning French cadastre buildings. Few consider TIGER experiment in USA to be successful: last week's Operation Cowboy's main objective was fixing data imported five years ago. And what major problem does Steve identify in his yesterday note, what solution he proposes? Addressing, of course. US has too many buildings, and it is impossible for the mapping community to collect all the addresses. We just don't have enough time. And because there are vast emptiness in OpenStreetMap instead of addressing information, it is unsuitable for geocoding. And the situation is even worse in Europe and Japan. I guess no one ever succeeded in finding an address there. What could we do to save the project? Let's take TIGER data (which is, of course, horrific, but there is no other way), slap it into OSM and tell the community to fix it! Better than nothing, anyway.
Now I should remind you that Steve Coast still represents our project. Not Germans, Jochen and Frederic, whose articles I adore. Not Englishmen, Richard, Harry and Matt, messengers of reason. Not Simon Poole, current chairman of the OSMF Board, and not Richard Weait, an activist of Communications Working Group. Steve Coast. Only in October he was interviewed by James Fee, editor of the popular GIS-related Spatially Adjusted blog. In September he opened the State of the Map US conference, in November he told of his OpenStreetMap involvement to Czech GIS-portal. To this day Steve is a face of our project, despite being irrelevant. It was fun a year ago, but today I think he is starting to be harmful to project, slowing it down.
"We have points of control which are fully owned by single individuals. The people around them readily acknowledge that they don’t have a clue what we would do if they quit. I say that’s terrible. I say they should quit and we should find out what we have to do." Steve, who did you write this about?
(this is a translation of SHTOSM entry, and I'm sorry if something is translated inadequately)
Завтра вечером, с 22 часов, картографы во всем мире проведут несколько часов (а то и всю ночь) за рисованием карты. Петербуржцы нашли помещение, чтобы собраться для этого вместе, недалеко от метро «площадь Александра Невского». Всё про встречу написано в заметке: http://shtosm.ru/2012/02/06/1/
Приглашаем новичков! Если вам интересно рисовать со снимков, это один из очень немногих моментов, когда такое не только приветствуется, но даже поощряется. Опытные участники всё покажут и расскажут, от установки правильного редактора до различения типов растительности. Мы будем рады познакомиться с новыми осмерами из нашего города (а также с туристами, кому нечем ночь занять :)
Подходите на Старорусскую, 5 к 22 часам завтра, 7 февраля. Если сможете, запишитесь заранее (см. ссылку выше). От входа нужно позвонить GaM-у, +7 921 426-74-00, чтобы он поднял попу со стула и провёл вас в конференц-зал. Возьмите с собой ноутбук, мышку и какую-нибудь вкусность.
Спеша поправить карту, советую не пользоваться ссылкой «Правка» вверху страницы, потому что от того «простого» редактора возникают плохие привычки и мозговыносительные ошибки, не говоря даже о ежеминутном риске потерять часы работы просто из-за глюка flash или случайного нажатия «назад» в браузере. Воспользуйтесь отдельным редактором: http://josm.ru/
О, из всех мест, где тусуются осмеры, я как-то забыл про эти дневники. Если вам интересно, что происходит в русском OSM — читайте форум и сидите в чатике. И мониторьте вики. В отсутствие времени на всё это можно читать только про самое важное: http://whatosm.textual.ru/
Обойдя три квартала у метро, решил сначала по-быстрому зарисовать дворовые проезды, а как-нибудь потом — собранные POI. Сегодня закончилось это «потом».
И, должен сказать, удовольствия от такой работы немного. Сидишь битый час, возишься с каждой точкой, нажимаешь «отправить на сервер», а там — всего 50 жалких объектов. Из них, дай бог, треть будет рисоваться на карте, и то лишь тип и название.
Час за часом, по сотне точек на квартал, пытаясь ориентироваться среди плохо привязанных фотографий (городская же застройка) и по мелко исписанной бумажке, вводишь данные каждого магазина, каждой чёртовой нотариальной конторы. Пытаясь разобрать время работы в отражении на двери, думаешь лишь: «в гробу я видел этот квартал!»
Но пройдёт неделя — и руки сами возьмут планшет, а ноги поведут к очередному белому пятну на карте.
Первая картосуббота — надеюсь, положит начало традиции периодических сборов для отрисовки нашего города.
В чём отличие amenity=waste_disposal от amenity=recycling? На последних всегда написано предназначение (картон/стекло/пластмасса и т.п.), и они стоят вне пределов Российской Федерации. Потому что видел я такие баки в Петербурге, пихают в них всё подряд, независимо от надписи. Поэтому в России контейнеры для мусора всегда обозначаются как amenity=waste_disposal. Ну, или amenity=waste_basket для небольших баков на улицах.
Это я к тому, что не пользуйтесь amenity=recycling без абсолютно точных свидетельств, что контейнер предназначен для переработки и им действительно пользуются согласно назначению. Я вижу http://www.openstreetmap.org/?lat=59.9587&lon=30.32845&zoom=16&layers=M и понимаю, что recycling там понаставлены только потому, что мапник их рисует, а waste_disposal — нет.
Почти час рисовал разные дорожки, озёра и болота по IRS в потлатче. Переключаясь между окнами, случайно нажал Backspace. Опера услужливо вернула предыдущую страницу. Вся работа потеряна. Вот жопа!
Это к вопросу, чем жосм лучше потлатча.