Recent diary entries
Finally thhe Brazilian comunity have been able to extract address data from IBGE 2010 census data, covering millions of points all over Brazil. Following a discussion on the Brazilian mailing list, this data have been devided by states and voluntairs from different states have received their part of the data.
This data have a few limitations:
the data is mainly covering rural addresses
there are lots of duplication in the data
data sometimes refer to entrance of building, entrance to estate, and even center of building
With these limits the import of the data must be done with caution. The preferred way after the discussion is to use the IBGE data in a separate background layer, and compare it with already existing data and Bing overlay.
Further I have been in contact with VLIZ regarding importing their EEZ/Maritime boundary data. They like the initiative, but are uncertain of what impact it will have on their userbase. My plan as I states in my contact with them is to do a one time import of their data, or even part of their data if they are not willing to release it all. They have also responded that they have some other data packages under development and might be willing to share some or all of it with OSM. I will continue to playnthis ball to see if we can get some benefiting data out of it.
For the rest of May I do not know how much time I have for OSM. Generally when I am at home my time is limited to diary entries and mailing lists, and to some extent some gps surveying, though there is little I can survey anymore as most streets have been traced. If I had time to do some walking paper I could get some street names in place (where IBGE data doesn't cover it, i.e. urban parts of Guarapari.
The good thing about not having a specific project is that I can jump around on the map without feeling that I am delaying a specific import or project. This way I have now been able to add some residential roads in Marechal Floriano and Vila de Itapemirim, some random land cover in southern Espírito Santo, and gathered a bunch of SVG files I can use as base creating a complete collection of SVG icons for my INT-1 documentation.
The same SVGs are used in my INT-1 preset and mapcss for JOSM. I have by now all icons in chapter U according to British Admirality, US Hydrographic Institute, Brazilian Hydrographic Survey, French Maritime Administration, Canadian Coast Guard and Singapore Flag Administration. Mostly there are an agreement on usage of the INT-1 symbols from IHO, though there are some national variations. All national variations will be documented on the wiki, while I will chose the icons looking best for the preset and mapcss presentations.
In the future I will try to make a Mapnik style rendering based on INT-1 recomendations. I will follow the same base rule there as in my JOSM mapcss, chosing the icons looking the best.
At the moment I develop the preset and mapcss as well as edit new icons while at work (in my spare time) and upload changes while at home (due to somebody deciding to block ssh connections in the company firewalls). This makes for a long and slow debugging process, but hopefully the preset might be ready for an official listing in JOSM in not too long.
After completing the INT-1 preset I will start looking into making a Brazilian preset, where I collect tags and values agreed on by the Brazilian community, in order to assist in more consista t tagging in Brazil. This should IMO be done for several countries, not necessary to replace the default JOSM preset, but as an additional support to it.
All my .svg files are public domain, though some are derived from copyrighted material, all elements copied from other files are from PD licensed files, rest are drawed from scratch using only the copyrighted files as guide. No direct tracing of copyrighted files done.
Ok, so OSM is now under the new license, ODbL. Hopefully this doesn't mean much different for the contributors, and hopefully tha transaction to the new license havn't messed up too much.
In the last coupple of days I have gathered a few gpx tracks, and finally today got them uploaded to the server. I will within a couple of days look into getting some editing done, adding a few new features.
Among what have been explored is a german restaurant (Grossmutter), a tourist attraction (Fjordland) a zoo (didn't get the name) and a few roads. I also have the height warning of the viaduct in Linhares (the sign on the site says 5.20, but that is actually only valid for the lowest corner, I got to see more accurate measurements as it was in the process of approving a special road transport)
The tags I will use for the viaduct is maxheight=5.20 and maxheight:physical=5.20
With some of the roads I got around Pedra Azul I will be able to get a lot more out of Bing in the area, a large portion of one of the roads is so covered with trees that it is impossible to track from Bing. I hope the entire Rota de Largarta (thats the name of the road) can be traced now, previously only a few hundred meters have been traced.
As the license change is approaching and I have had litttle time to do any editing for a while (though collected a tiny bit of data for future edits), I have done a little thinking.
I have now contributed to OpenStreetMap for more than 4 years, I am active in translating wiki pages to Portuguese (though not keeping them up to date unfortienately), I have now harvested a little data for openBMap.org and got some 3G coverage to Brazil.
An idea for cloudmade/other routing program: Use openBmap data to give warnings about lack in mobile coverage on routes. (I know data still is somewhat limited, but that is constantly improving)
There have been some weeks now since Bing coverage of Brazil (and probably much more) have been extended, and many places in Brazil (and maybe elsewhere in the world) have turned into a editing frenzy. The map is improving quickly, some places so quickly that we start getting into conflicts because several users edits the same objects roughly at the same time. This is both a positive and negative thing. Positive because the map quickly improves, negative because it increases the possibility of having the same object duplicated sevaral times.
Another reasent development have been the switch of base map in FourSquare from google to OSM. At first several Brazilian cities turned into white spots in FourSquare, but some bright minds have taken unto them to correct this. At least one new active mapper have turned up because of this, helping putting more of Brazil on the map.
The effect this have had on the data is that the size of the XML files of Brazil have increased extremely. I am not able to do a check for all of Brazil (I neither have the bandwidth nor the discspace to do a comparison, nor have historical archives to show me the old sizes) but the state of Espírito Santo have close to doubled since chirstmas. If this is representative for the rest of Brazil than this might become challenging.
When that is said I have discovered another open project called openBmap (check out site) that collects harvested data regarding mobile phone (and wi-fi) coverage based on combining signal strengths with GPS possitions. I have installed their harvesting app on my mobile, and have collected a few points of ViVo coverage near Vitoria. I think I should have this logging running when driving around in the area, for the harvested ViVo data so far only shows 2G coverage, while there are quite lot of 3G coverage in the area. According to what I have read on the project it should be able to log 4G coverage, but to my knowledge this isn't available in Brazil yet.
I was positively surprised a little time ago when I opened my JOSM, and discovered that Bing! coverage had been greatly extended. I looked around Espírito Santo and saw that much of what I had been missing have been included, though there are still many areas of interest to me that are not yet included.
A mapping frenzie ensued from this discovery, and much have already been added to the map. I have come to a point where I soon cannot add much very useful information in my short edits anymore, but think I will look into polishing adjoining information as well as information within Espírito Santo (such as improving boundary relations around Brazil, such as municipalities in Rio de Janeiro state, and State and Federal highways in Espírito Santo.
It is desirable to see even further extension of Bing coverage, and guess that might be acomplished again next year (don't hand out too much candy at one time I guess). The map is slowly but surely improving, and already many places OSM shows much better map data then some comersial maps, and google.
Now, with the last improvement in coverage and resulting edit frenzie, the Brazil map is getting so big in OSMAND and other navigator software that maybe it is due time to cut it into state size portions. I browsed Geofabrik and CloudMade and noticed CloudMade already offer Brazilian states for Garmin and TomTom as well as state poli, shapefiles and osm extracts. OSMAND seems to base its chunks on Geofabrik who still only offer entire Brazil. Geofabrik seems to have daily updates while the last update of CloudMade seems to be 13th of December 2011, almost 3 months ago.
Finally last municipality in Espírito Santo have been imported, though the dataset might need some cleaning around.
It will be some work doing the cleaning, and with the limited time and internet connection I don't see that I can do that now. I hope to look into adding place= tags on those admin levels corresponding to a place=, but that will also have to wait for an opertunity of cleaning. Another option is that somebody else can do a cleaning though.....
 http://www.openstreetmap.org/browse/changeset/10468411 Last changeset
 http://www.openstreetmap.org/browse/relation/1843582 Espírito Santo Collection Relation
Good news: Internet is back, after the ship where I work have been without communication for a few days.
Bad news: Our internet line is so weak that I almost can connect to OSM. By almost I mean ALMOST, as in link timeouts and connection failures almost all the time.
This means there will be almost no more importation done this month. I have 18 municipalities left to close this import, but that might have to wait until March.
Another note of interest is that coming crew change will be in Belém or Fortaleza, and that my family are looking into coming to visit for the crew change. We don't get too much chance of being tourists in Brazil, and the northern regions is a little too far to do spontaneous trips.
The ship is working only 80 nautical miles from Cabo Orange, on the border towards French Guiana, I was almost hoping that we could visit Cayenne, but that would be too much to ask for as we are on contract with Petrobras.
I have returned home after a holiday in Portugal, and I managed to harvest some 12000 gpx points, though it seemed like most of it already was on the map. For certain I know yhere are 3 petrol stations and a restaurant to add, and maybe a hundred meter road. I might also be able to point out a post office and a grocery store (they called it a mini-market, a supermarket with two shelves....)
It looks like Portugal have done some imports in additional to maybe Bing data.
I will look at the data when I get time, for now, the GPX tracks are uploaded publicly...
Counting down for the holidays meaning I soon will have to suspend the import of IBGE boundaries of Espírito Santo. There are about 20 municipalities where internal divisions have not been imported yet. With regards to the work load left, and time needed to actually do the import, than I need about 10 slow workdays to complete the import.
I will now (soon at least) go away for holidays, meaning that there will be a good 10 days where I am not able to do much work on the map. Hopefully I get the rest of the import completed soon after I get back to working on it.
If it hadn't been for the various setbacks I have had during the import, I would have been finished by now, but unfortunately setback is a part of reality, even for OSM contributors.
Going away for the holidays means I might gather some GPX tracks and some POIs with my mobile. I have already downloaded the maps of Portugal, and are trying to download the maps of Spain to my mobile, so that we are covered for driving. (Spain is only optional if time allows for us).
Colatina was the last of the complex municipalities in the import, I was fearing São Mateus to be complex, but turned out to only have district boundaries, no suburbs. All remaining municipalities are expected to only have district boundaries.
I have now started encountering problems on my import, the data file is so large it starts to give serious headaches and runtime errors in the middle of a small edit session. How should I progress from here? Split the data in some way? I could obviously make smaller data files based upon the missing municipalities with their neighbouring municipalities. That way I could seriously reduce the size of the data file, but it will seriously slow down the import process (though not as much as these crashes I experience).
I will look into that this afternoon so that I can get the snowball rolling again.
BTW I have started to add place=suburb on relations for admin_level=10
Even with limited online resources, a internet connection that is failing at times, limited time, etc. I have managed to import a large portion of the IBGE boundaries data for Espírito Santo. I have now come to a new problem in my goal to import the entire data set. The files I work with are becoming so big that it just crashes on me. Merkaartor which I need to use to trace the source shape files doesn't want to upload anymore for some reason unknown to me, and seems to run out of memory and crash on runtime errors when I try to export the changed material to allow for upload through JOSM. I need to find another approach now.
Thankfully I have completed importing the areas with a large portion background data, so it should not be too much of a problem importing the rest of the boundaries onto downloaded data. Though it would be greatly appreciated if I could read the source data in JOSM, but I havn't been able to find any plugins allowing to read shapefiles in a satisfactory way. My limited resources prevents me from running perl, python, ruby and whatever other types of scripts there is to do various manipulation of spatial data, and the few java tools I have found seems to read the data erroneously.
Well, when I have my typical setbacks I do some tinkering on BING data in Guarapari. With the municipality boundary imported I can focus my edits inside the boundings of the municipalities, and have now been able to connect various parts of the municipality together without needing to leave the municipality. If somebody picks up my threads in neighboring municipalities than we can focus on making a much better map one municipality at a time, and this way we can quickly pass the quality of commercial maps.
BTW: Last year somebody making a visit in Guarapari expressed the lack of a map to purchase, well, they should now be able to get one from OSM, though there are still a lot of information missing.
The import of IBGE data continues, though I have made good progress the last few days. I have completed importing all municipalities, and in addition to creating the necessary relations on admin_level=8, I have also created all relations for mesaregions (admin_level=5), metropolitan areas (admin_level=6) and microregions (admin_level=7) for the entire state. I have now started to look into districts (admin_level=9) and suburbs (admin_level=10) for Vitória, the state capital, and will look at continuing this with the other municipalities of Grande Vitória metropolitan area. I think I will do all admin_level=9 first and complete with admin_level=10 after that.
I have however encountered a problem, the IBGE shapefile I use as source have also a tag SUBDISTRICT, in other words, something between admin_level=9 and admin_level=10. Maybe admin_level=10 for Brazil should be shifted to admin_level=11 in accordance with the German suggestion on boundaries=administrative, to make place for SUBDISTRICT as admin_level=10?
Currently the Brazilian tagging scheme for admin_level is:
admin_level=2 : National border
admin_level=3 : National regions (i.e. Nordeste)
admin_level=4 : States
admin_level=5 : Mesaregions
admin_level=6 : Metropolitan areas
admin_level=7 : Microregions
admin_level=8 : Municipalities
admin_level=9 : Districts
admin_level=10 : Suburbs (known as bairros)
admin_level=2 - admin_level=7 : as is
admin_level=8 : Municipalities
admin_level=9 : Districts
admin_level=10 : Subdistricts ::: squeezing this one in between Districts and Suburbs
admin_level=11 : Suburbs (known as bairros)
I will bring this one up on the Brazilian mailing list.
I have the last week or so started to import IBGE data for municipality borders of Espírito Santo state. I think this is the last state to do this kind of import. I have had a few problems doing the import, firstly I spent a lot of time finding out how to extract the data from the shapefile. The problem was that since I have no computer of my own, and have limits of what I can install on work computers, I was not able to get any convertion tools running (such as ogr2ogr, shp2osm, etc) and couldn't find any plugins to JOSM allowing to open the shapefile. But than I got the idea of trying out Merkartor, which are available in a portable version I can run from my portable hard drive. It allows me to open the shape file and do tracings directly from there.
The problem that I still am facing is:
1) Merkartor doesn't allow me to save for some reason, might be a bug in the latest portable version. This results in that I have to upload changes for each edit session.
2) Since I run from a portable disk, long edit sessions can end in a crash, like today, when I loose all data not already synced.
3) Since I am not familiar with Merkartor, I tend to find the program complicated, and want to return to JOSM as soon as I have completed this import. I also return to JOSM at times to clean or fix data.
4) Unstable internet connection from work. The internet line here is imbarresingly slow, and at times completely non-existing. In worst case I will not be able to upload an edit session, and a whole day work might get lost because of no connection errors for too long.
Despite all of this, I have already imported more than half of the borders, all state border municipalities completed, just missing a few coastal municipalities, and filling in the gap in the middle. I will complete it all by (in JOSM that is) adding relations for Micro and Mesa regions as well as Grande Vitoria metropolitan area. The map will look better when all of this is done. I'll probably make links to all relations checkers from the state wiki page.
I have done all the easy Bing tracings of Guarapari and surroundings, there are stil much to do, but things are getting more complicated. Items are either imensely huge, tiny, difficult to destinguish, etc. I am still doing some random Bing tracings, random places of interest to me.
I have also downloaded a series of Brazilian Nautical Charts, and as I get the .png convertions calibrated I'll use them for tracings. All this are in Public Domain under Brazilian law, provaided by the Brazilian Hydrographic Institute. All calibration is done against coastline, first towards existing coastline in database, than I check the existingncoastline against Bing, Landsat, and readjust the calibration until I have best fit between the Nautical Charts and at least one other source. After importing the desired objects into JOSM I control these objects towards Bing in order to get an as accurate possition as possible, before uploading the changes to the database. This have until now resulted in a lot of objects adjescent to the nautical objects been added from Bing also. The Nautical Charts have also to some extent additional information for existing objects, this information is of course added. Until now the following areas have been imported this way:
Punta de Ubu terminal with approaches,
Port ov Vila Velha, Vitoria and Tubarão with approaches,
Port of Macaé uncluding Mercado de Peixe and Imbetiba terminal, with approaches
Other areas to come, but than I need to calibrate the charts first. I want to do the remote islands and islets of Brazil this way, but it is very difficult to calibrate these charts, as they lack Bing data, have only low resolution Landsat coverage, and there are very few features visible in the database and in other sources. Calibrating the Rocha Atol with only the position of the lighthouse to calibrate against is hopeless, I need at least two more objects to get a satisfactory calibration.
I continue to work on my INT-1 documentation, preset and MapCSS, but are getting little progress. One reason is that I have documented all the information I have available from offline resources, and have had limited time and access to work with the online sources. Further I have made .svg icons of many of the easier objects, though are planning on making coloured buoy icons for map rendering. The preset is getting most attention while I do imports of nautical data, as I constantly find errors and missing detailes. I tempt to make the preset such that it reflects all imported data, though the SeaMark plugin is still necessary for buoys, beacons ans lights. The MapCSS is randomly tested in my JOSM, though I hope I can get a tileserver somewhere using it to give a graphical presentation of nautical charts.
As I mentioned in my previous post, a new mobile phone will allow me import more POIs using Mapzen, and I have started to gather geotagged photos for photomapping. Gathered my first gpx track in more than two years, and will look into that also. Maybe osmtracker also will allow me to start voice mapping and make geotagged notes.
Yesterday I also looked at a dirt bike, so maybe in the near future I can be seen on tracks and dirt roads around Guarapari gathering more gpx tracks.
Last time I was home I spent almost a fortune on a new car, a Peugeot 3008 Griffen, with a full panorama view and complete sun roof.
Just a few days ago I treated myself with a new phone. The new toy is a Samsung Galaxy Mini, complete with Android, built in GPS, 3MPix camera, WiFi, and more.
After installing a few necessary apps I found out it was time for a test drive, and picking up the kids in Kindergarten was just that opertunity.
The first app tested was osmdroid, but that didn't give me anything useful, need to find a better app for navigating with OSM maps, something that can compete with the pre-bundeled Navigation app that uses google maps..... The google map couldn't find my address, but I know that OpenStreetMap will, as I have added that ;)
Next try was osmtracker. Starting the app was easy, and browsing the predefined waypoint tags gave me a good impression that this will be helpfull on my travels, to gather various information such as traffic light signals, speed limits, etc. I needed to play some around to make it save the .gpx but managed in the end.
OsmGpxUploader was than tested to get the track onto OSM, and worked fine, was easy to use. Now next week I'll probably try loading that track in JOSM to evaluate the quality of the track.
I took a few photos with the track during the trip, to see if photo-mapping might be any helpful to me, later I try voice memo and text memo as well, but I guess this sort of experimenting is best done as passenger...
The other day I added the Praia do Morro post office on with Mapzen POI Collector, an app I have used on my iPad earlier, the iPad interface is much easier, but at least the android interface lets me add the POIs. I'll probably edit the information later either with the iPad app or in JOSM.
I am also looking into buying an offroad motorcycle, tracking dirt trails might be my next hobby.
Next on the program will be to buy the upcoming multimedia centre for Peugeot 3008, which will allow me to have an onboard satnav, hope it allows me to gather more tracks (maybe even passive, without starting the nav program), that way we will get a load of new tracks on osm soon. With some luck I will be able to install OSM maps in the car also, but have the feeling it will be some sort of propriary system.
I'm sitting in a hotel in Rio waiting for the third day in a raw, I have one more day of waiting after this. The waiting time have been spent sitting in the room watching BBC World about what happens in Libya, and walking around the area around the hotel gathering points with Mapzen POI Collector. Rio is getting well covered, but luckily for me it was a slightly blank spot around my hotel, so a few points could be added.
There are two things that could have been better when waiting like this.
1) I could have a mobile with internet capabilities, for example as I am thinking of getting myself a Nextel, I could have a Blackberry with internet subscription. That way I could upload Mapzen POIs on the go instead of returning to the hotel. That wouod allow me to do longer walks, and gather more points (and in fourscuare as well?)
2) I could get myself a new notebook computer again, allowing me to work effectively with JOSM and Bing tracings, edit .SVG icons, and actively edit on the wiki. The tablet is not a good platform for these kinds of work. JOSM don't run on my iPad, and as far as I know, no software for iPad allows editing .SVG, and wiki editing is difficult, specially on large sections (though possible)
I guess it will be some time before I can work more effective like this, but if I get to work focused extra like it looks now, than maybe I can start doing something about it in not to distant future?
I am soon heading back to work, that mean early monday morning. Last time I headed for the same vessel I was waiting almost a week on a hotel in Rio de Janeiro. The result was that I could put the hotel on the map using my Mapzen POI collector. If I'll be waiting on a hotel again, I'll try to get a few more points in.
Further, returning to work allow me time to do some map work again, since at home I am occupied with house maintenance, kids, financial mess, trying to start up a company and much more. At work I have my hours of work, a few hours of leisure, and my hours of rest, and with a descent internet connection and PortableApps I can get some work done, both through JOSM, and through maintaining wiki pages, a tagging preset and messing with MapCSS.
By the time I change vessel I hope to have uploaded all the symbols I have access to to INT-1, and make more headway on creating SVG icons for MapCSS and my preset.
Tracing is getting more and more difficult as I am running out of areas of interest for me to trace. I could do with better data in southern Espírito Santo, there are need of removing some cloud patches, and some more flyovers needs to be added, as I would like to trace Divino de São Lourenço and Guaçuí, among a few other places. Maybe I should continue adding natural= and landuse= around Guarapari?
For many people around, there are holydays now, meaning they break off from their routines (mapping somewhere else, getting more time for gathering on the ground information, or taking a time off). For me holidays doesn't exist, at least this year. I am in the process of changing employer, that is I started to work for my new employer before I ended my work at my previous. My new employer is also a little short of people with the right qualifications, so they have signaliced even before I started, that I need to work some extra in the coming time.
The good about that is while I work I get some time off, from time to time, that I can use to continue my work on INT-1 documentation on the wiki, with a JOSM tagging preset and a MapCSS rendering rule, and translate more wiki pages to portuguese.
I have reasently had a time at work with a very bad internet connection, resulting in many of the .png files for INT-1 symbols have been redrawn as .svg. I am at the same time redesigning the tagging preset and the mapcss to use these svg files, as I have managed with a trac ticket to get svg support implemented in JOSM (currently available only in latest experimental snapshot).
The result of this offline time is that I am starting to understand some ground principles in Inkscape, which allows me to draw simple objects, i.e. the simplest INT-1 icons such as E10 (church). I will soon start looking at more advanced symbols, such as chimney stack (E22), and look into how to solve more complex MapCSS rendering with combined icons (can I make rules that allow me to combine symbols for lighthouse, the light, a radar recon, and a fog signal? Or do I have to make a symbol for each case?) I will also need to look into if I can offset icons, turn them around, make images as border to a way, and so on.
When svg version of INT-1 icons are complete enough, I will post a link here, and will see to have the wiki documentation updated to use svg instead of current png files, but first the documentation need to be complete.
Why I work on many halfway projects instead of full attention to one? Probably because I work best that way.
There are increasingly many users registered in a 50km circle around my house, most of which located in Vitoria and Vila Velha. I've noticed that most of these never have done any edits, and have no information at all on their profils. It is probably thousands and thousands of these. Does the administration any way of checkingnlast login of users? That way one could install a filter, or in some way remove, users that only have created their accounts, but not contributed in any way.