OpenStreetMap

Using self-generated TMS compatible tiles with OSM

Posted by wallclimber21 on 30 September 2009 in English (English)

My question last week on how to use your own tiles in OSM didn't get an answer, so I had to figure out something myself. (http://www.openstreetmap.org/user/wallclimber21/diary/8019 )

This took a bit more effort than I expected:

Prologue: USGS topographical maps are a great source to map places that are otherwise difficult to map.
- They often contain features that are not available anywhere else (e.g. power lines)
- They are great for area's that are not visible on Yahoo Aerial tiles (e.g. too low resolution or because places where streets are covered by trees / forest).
- They often have feature names that can't be found anywhere else.
- They're in the public domain

(They're also very much outdated, so this won't work for areas with new construction etc.)

I've described here (http://www.openstreetmap.org/user/wallclimber21/diary/7750 ) how you can use those topo maps with Qgis.

Unfortunately, OSM support in Qgis is very basic. You really want to be able to use your tiles as a background in a real OSM tool.

The tiles generated by gdal2tiles.py follow the Tile Maps Service specification (TMS). But while they align perfectly with the tiles of Yahoo, Google, Virtual Earth etc, they don't follow the same standard in how to select a tile at a particular location. Google uses a zoom/y/x format, where (0,0) is at the top left of a flat world. Virtual Earth uses a recursive quadtree structure and TMS uses a structure similar to Google, but with (0,0) at the bottom left of the same flat world. Yahoo uses yet another slightly different convention...

The standard tiles of OSM are using the Google convention.

In addition, there are limitations to the way OSM tools can access custom background tiles: Potlatch only supports the Google convention (I believe), JOSM only supports the WMS standard, not the TMS. And Merkaartor supports both WMS and TMS, but with TMS only the Google convention.

I couldn't find an easy way to set up a WMS server, starting from existing TMS tiles. I'm sure it's possible with MapServer or TileCache or something, but I didn't have the patience to configure a whole setup.

So I decided to shoot for a Merkaartor based solution first.

Since this only supports the Google convention, one needs to hack gdal2tiles.py to modify the names of the directories and files that are created. You can find a patch for that here (http://gist.github.com/197749 ). Note that this patch only fixes the tile names. It doesn't make corresponding changes to the generated .html and .xml files, but since we don't need them, that's ok. A better patch would add a command line option to do all this. Feel free to contribute...

Now, go ahead an create some tiles from a topo map. You'll see that, after the patch, the y coordinate is different from before.

The next step is to get Merkaartor to use your tiles. Unfortunately, neither Potlatch, JOSM nor Merkaartor have support to use a local filesystem for this: they want to talk to a server to get the tiles.

No doubt, there are many HTTP servers that will just send you files, but I'm familiar with rails and didn't want to waste time, So I simply created a new rails project:
> rails topo_server
Then I created a symbolic link in ./topo_server/public directory to my tiles directory:
> cd topo_server/public && ln -s .../master_tiles master_tiles
Finally, I started the server itself:
> ./scripts/server

Finally, you need to tell Merkaartor to use your new tiles.
- Tools -> TMS Servers Editor (this dialog box is completely non-intuitive in how the buttons work...)
- Select one of the existing ones (e.g. Cycle Map) and click "Add"
- Modify the fields:
Name: USGS
Server address: localhost
Path: /master_tiles/%1/%2/%3.png
Tile size: 256
Minimum Zoom: 9
Maximum Zoom: 15
(Check you master_tiles directory for the minimum and maximum to fill in)
- click apply on the right size, then ok.

Now you can enable your tiles as a background by right clicking on MAP layer. A USGS option should show up under TMS adapter.

If all goes well, you're good to go!

Except, that is, if you're on a Mac...
Merkaartor will start to crash left and right and become completely unusable. This is because of a race condition in Qt (even the very last official release 4.5.2) with the way it handles Mac sockets. When you are using tiles hosted on your own machine (localhost), this bug will hit all the time and Merkaartor will crash hard.

The only way I've found to fix this is to download the Qt source code, apply the following patch (http://qt.gitorious.com/qt/qt/commit/84abdaa41e6c3bde6ac653e02bd72300b6681572?diffmode=sidebyside ), compile, wait a few hours and install. If you already have other Qt libraries installed, make sure you point to the new version by setting:
export DYLD_FRAMEWORK_PATH="/usr/local/Trolltech/Qt-4.5.2/lib::${DYLD_FRAMEWORK_PATH}"

(You don't want to know how long it took me to figure this all out...)

Unfortunately, my productivity in Merkaartor doesn't come close to Potlatch, which is nearly perfect in the way it is designed to minimize clicks and key strokes. Potlatch also has support for custom tiles. I assume it uses the same Google convention, but I haven't been able to get it to talk to my server on localhost. I'm still working on that. But, for now, the current solution will do.

Enjoy!

PS: It would be nice to have some info on the "New Diary Entry" page on how to add markup to a post. It's not rocket science to 'svn co' the full source code the OSM site and grep for the definition of 'htmlize', but others may not have the same kind of determination. (Edit: nope, didn't work either...)

Comment from TomH on 30 September 2009 at 07:51

The markup is just HTML so no need to go looking at any source ;-) It will be sanitised so some things may not work but basic markup should.

Hide this comment

Comment from Richard on 30 September 2009 at 08:45

That's brilliant. I've long wanted to get the USGS topo maps back into Potlatch (they were there briefly as part of Christopher Schmidt's now-defunct OpenTopoMap project) - agreed that they're a wonderful resource.

You'll need to specify http://127.0.0.1/my/tile/server/!/!/!.png rather than localhost. If you use localhost, Flash Player's dumb security model thinks "uh-oh, external site accessing local resources" and baulks. If you use 127.0.0.1 it doesn't notice.

If you have the time (and with the connivance of the admins), it'd be wonderful to set this up on dev.openstreetmap.org as a service - either downloading and reprojecting/tiling WMS on the fly, which is what OpenTopoMap did, or perhaps just offering a service for people to request an area then tiling it.

Hide this comment

Comment from wallclimber21 on 30 September 2009 at 09:01

Thanks for the potlatch tip: it's now displaying tiles.

However, it doesn't display all of them: there are vertical and horizontal white bands where no tiles are displayed. This must be a bug in Potlatch. They display fine in Merkaartor.

Hide this comment

Comment from Richard on 30 September 2009 at 10:03

Seems odd - it displays tiles from other sources (e.g. Walking Papers) ok using the same system, and I've never had a problem like that reported before. Given that you're using a Mac, can you open up Safari's Activity Viewer and see what URLs it's trying to access?

Hide this comment

Comment from Dalkvist on 30 September 2009 at 14:58

Great minds think alike =)
I have been doing the same things with a Old Swedish map series, ekonomiska kartan.

But I use python for testing, as staring a server is a one liner, just run "python -m SimpleHTTPServer 8085" in the tile directory root.

Loading tiles in potlatch works great, even when giving it the url "http://localhost:8085/!/!/!.png".

Hide this comment

Comment from Dalkvist on 30 September 2009 at 15:01

Oh and in http://geoserver.org/ it is not to hard to serve geotiffs as wms directly.

Hide this comment

Comment from wallclimber21 on 30 September 2009 at 16:54

It looks like Potlatch is requesting all tiles just fine, but it doesn't display them. However, I didn't mention in my little write-up that I created a script that runs gdal2tiles on individual geotiff files first and then merges them together after the fact. It's always tiles that are created from multiple tiles that have this problem.

Could it be an issue with alpha not being 255 for tiles on the boundary? Just guessing here.

I'll file a bug an attach a number of tiles (when I get back from work...)

Hide this comment

Comment from wallclimber21 on 30 September 2009 at 16:55

Also: localhost works fine... I just needed to add http:// in front.

Hide this comment

Comment from Richard on 30 September 2009 at 22:55

Are they progressive JPEGs, perhaps? I _think_ Flash Player doesn't load progressive JPEGs.

Hide this comment

Comment from wallclimber21 on 30 September 2009 at 23:09

No, they're just .png.

Hide this comment

Comment from Richard on 1 October 2009 at 08:49

Hm. It does sound like there's something about them that's causing Flash Player not to display them. If you could attach the tiles to a trac ticket that'd be cool.

Hide this comment

Comment from RussNelson on 2 October 2009 at 03:35

You *do* know about the Terraservice WMS, don't you? Recent versions of JOSM include them as one of the default values you can copy into your menu.

http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=drg&styles=&format=image/jpeg&

Hide this comment

Comment from wallclimber21 on 2 October 2009 at 08:27

Yes, I know about those.

But they are insufferably slow. The image quality is much inferior compared to the originals and they are completely broken in the High Sierra's. (For a nice example, you just have a look at Yosemite Village.)

They also don't work with Potlatch...

Hide this comment

Leave a comment

Parsed with Markdown

  • Headings

    # Heading
    ## Subheading

  • Unordered list

    * First item
    * Second item

  • Ordered list

    1. First item
    2. Second item

  • Link

    [Text](URL)
  • Image

    ![Alt text](URL)

Login to leave a comment