rayKiddy's diary

Recent diary entries

Better way of verifying the connectedness of relations?

Posted by rayKiddy on 12 July 2018 in English (English)

I have an application which I am developing. I am not sure it is of general interest, yet. I track some relations in it. I am interested in cities and counties in California and I am recording their ids and other information about them.

I just implemented code (in java) which reads the relations, and then recursively reads the ways and nodes of it and verifies that the lines therefrom are connected. The obvious implementation is taking a rather long time to do the job. I am hoping there is a better way.

Any suggestions?

I will post more about the application when I have it running on my colo.

thanx - ray

Is there a way to see diary comments inline with diary entries?

Posted by rayKiddy on 29 September 2015 in English (English)

I think it would make sense to see comments in-line with diary entries. This seems a bit obvious, no?

Right now, one has to go to the diary entry page, which breaks the flow of the document and occasionally messes up navigation in the diary entries, forcing one to start at the top again.

And I seem to be losing the thread of a diary entry. Was something put into a message? Or into a comment? And what is the difference between a reply to an entry and a comment on an entry? Is the difference here important enough that there seems to be separate UI for these?

Why does this seem to be more complicated than it needs to be. The UI for reading a list of threaded discussions, even date-ordered discussions, seems to be something that has been done. So, why are we doing this differently and in an unnecessarily complex way?

If I could get the diary in mbox format, a mail reader would be a much more convenient way to reading this. Is that possible? Or does one have to go through this web UI?

  • ray

What happened to the city of Sacramento?

Posted by rayKiddy on 29 September 2015 in English (English)

I have been fixing up boundaries of cities in California and I have found something odd.

Where is the city of Sacramento?

There is a city there, somewhere. There is a county. The county boundaries are at and that all looks good. And it is not a county/city hybrid thing like San Francisco. Yes? And I can find the cities of West Sacramento, Rancho Cordova and others nearby, But I cannot find boundaries for the actual city of Sacramento. Google has boundaries for it, but OSM does not?

Or is there some way I should be finding it that I am not doing? I guess it could be mis-spelled.

I am going to the area around the county in and doing this search:

 out body;
 out skel qt;

It finds lot of stuff, but no city. Any ideas?

admin boundaries: TIGER vs OSM...

Posted by rayKiddy on 27 August 2015 in English (English)

Wow. There are some interesting challenges in trying to figure this out.

I have figured some things out about how to find TIGER info in OSM. I can do things like look for "tiger:reviewed"="no".

I have downloaded a bunch of the TIGER data. I am going from the directories under One challenge I am having is finding the longitude and latitude data. I know it is there somewhere. Is it in the files under EGDES? No. Under ROADS or ADDR? No. Only 40 or so more to check.... But I am looking at only the dbf files in those zipped files, using the dbf2mysql utility for Ubuntu. Is the lat/long data only in SHP files? That would be obnoxious.

But the TIGER data does have TLID fields. So this should help me. Because OSM features have a "tiger:tlid"=num value. So, I should be able to pick a TLID from the TIGER data and see it in OSM, or see and id on OSM and find it in the TIGER data, right? Maybe not so much.

I have over 12,000,000 edges. I have all of OSM. Can I find one place where an id appears in both? No. How should I search? Randomly? I have done a bunch of random checks and so far I hit nothing. Which kinds of id values appear in both? Does OSM only include polygon objects? Or any of the other 20 or so types in TIGER? There are so many ways to slice this. I am sure when I figure this out, it will be sooooooo obvious..... :-)

We will see.

admin boundaries: school districts in CA, US

Posted by rayKiddy on 25 August 2015 in English (English)

My continuing effort to learn, before I start adding a bunch of data, the proper way to get relations for the school districts in CA into OSM is making me aware of some interesting issues.

Question: Is there a tool that takes a relation id (or presumably a node or way id) that allows you to add or edit a key-value setting? In the relational db, this would be a simple update statement. Is OSM, you download the entire relation, add the value and upload all (I think) the data. This to change or add one key-value pair. It seems a command-line tool for doing this would be an obviously useful thing.

I recently spoke with someone at the Santa Clara County Registrar and found out how they use GIS data. Or rather, how they do not use it. They have to run elections, such as school board elections, and also state or country races. The state gives them district boundary information as TIGER data. The county gives them something else. And school districts provide them, basically, with textual lists, like narrative maps. Apparently they list streets and which side, or both sides, in a completely non-standard ways. So, the Registrar has to manually sync this up with their data. Could these government entities be using OSM data? I think they could....

That being said, TIGER says this:

 ELSDLEA: 38460
 GEOID: 0638460
 NAME: Sunnyvale Elementary School District
 MTFCC: G5400
 ALAND: 35549740
 AWATER: 9341
 INTPTLAT: +37.3897355
 INTPTLON: -122.0247588

Obviously there is TIGER meta-data in the OSM map. But locating the ways defined via TIGER for "GEOID" = "0638460"? I am still trying to figure out how this works. Maybe the Overpass syntax for this will one day be obvious. (sigh). I hope so. For now, I can re-use and extend queries that people are handing me, but I do not grok enough to actually write any query that works as I expect.

It would be good to be able to go to government entities and say: "If you verify your geographic boundaries in OSM, you will be able to use this set of lines defined by this tag and track/verify/ignore changes to them so that your internal systems can use OSM." I can't say that yet. We will see.

administrative entities, consistency?

Posted by rayKiddy on 13 August 2015 in English (English)

In my previous, I captured all the relations in California and around it with:

 result = api.query('relation'
               'out body;')

I had been wondering about some of the things I was seeing in the "is_in" tag. This explains some of that. There is obviously a bit of inconsistency here.

 $ grep '^is_in ' rel01.txt | sort | uniq -c
       2 is_in --> California; CA; USA
       1 is_in --> Irvine
       2 is_in --> Los Angeles County
       2 is_in --> Santa Barbara County
       1 is_in --> USA
       4 is_in --> USA, Arizona
     185 is_in --> USA, California
      68 is_in --> USA, Nevada
       1 is_in --> USA, Utah
       1 is_in --> Ventura County

Well, maybe just using "code" tags makes it better.

 $ grep '^is_in:country_code ' rel01.txt | sort | uniq -c
       2 is_in:country_code --> MX
     258 is_in:country_code --> US
 $ grep '^is_in:state ' rel01.txt | sort | uniq -c
       4 is_in:state --> Arizona
     185 is_in:state --> California
      67 is_in:state --> Nevada
       1 is_in:state --> Utah
 $ grep '^is_in:state_code ' rel01.txt | sort | uniq -c
       4 is_in:state_code --> AZ
     179 is_in:state_code --> CA
      67 is_in:state_code --> NV
       1 is_in:state_code --> UT
 $ grep '^is_in:country ' rel01.txt | sort | uniq -c
       2 is_in:country --> México
       1 is_in:country --> United States of America
     256 is_in:country --> USA

I think that is a "yes". And just to see all the keys. I have no idea what some of these mean, but here they are:

 $ awk '{print $1}' rel01.txt | sort | uniq -c
       2 $
       1 3
       1 addr:country
       2 addr:even
       2 addr:odd
       1 addr:state
     537 admin_level
      87 alt_name
       3 alt_name:vi
       3 amenity
      79 attribution
       1 board_type
     219 border_type
     591 boundary
       1 census:population
       2 construction
      58 county:abbrev
      58 county:ansi
      58 county:name
       1 county:name:vi
      39 created_by
       1 currency
       3 ele
       1 FIXME
       2 flag
       2 gnis:Class
       2 gnis:County
       2 gnis:county_id
       2 gnis:County_num
       2 gnis:created
       3 gnis:feature_id
       2 gnis:id
       2 gnis:ST_alpha
       2 gnis:state_id
       2 gnis:ST_num
       1 history
       1 import_uuid
       1 int_name
     267 is_in
       1 is_in:continent
     259 is_in:country
     260 is_in:country_code
       1 is_in:county
     259 is_in:iso_3166_2
     257 is_in:state
     251 is_in:state_code
       2 ISO3166-1
       2 ISO3166-1:alpha2
       2 ISO3166-1:alpha3
       2 ISO3166-1:numeric
       8 ISO3166-2
       1 land_area
       3 landuse
       1 layer
       1 military
     573 name
       1 name:ab
       2 name:ace
       2 name:zh-min-nan
       2 name:zh-yue
       2 name:zu
      98 nist:fips_code
      94 nist:state_fips
       9 note
       6 odbl
      12 official_name
       1 official_name:vi
       1 old_name
       1 old_name:vi
       1 owner
     364 place
      97 place_name
      26 population
      21 ref
       6 ref:fips
       1 relations
       1 route
       2 Script
       2 short_name
       1 short_name:vi
     332 source
     236 tiger:CLASSFP
     238 tiger:CPI
     237 tiger:FUNCSTAT
     237 tiger:LSAD
     236 tiger:MTFCC
     238 tiger:NAME
     238 tiger:NAMELSAD
     238 tiger:PCICBSA
     237 tiger:PCINECTA
     238 tiger:PLACEFP
     238 tiger:PLACENS
     238 tiger:PLCIDFP
     267 tiger:reviewed
     237 tiger:STATEFP
       9 timezone
     591 type
       5 website
      16 wikidata
     464 wikipedia

Simplified access for administrative entities? (perhaps solved)

Posted by rayKiddy on 13 August 2015 in English (English)

Yikes, this stuff is just not terribly ... intuitive. I received a helpful response from maxerikson. He said:

Here's an Overpass Turbo query that returns all boundary=administrative that
are inside the current view:

 out body;
 out body;

Now, just for giggles, google a question like "specify bbox in overpass?" It is a bit too abstract. It did point me to the OP documentation but I already knew where that was. What I wanted to know was: How do I replace that {{bbox}} above with a bbox defined by coordinates.

It turns out to be:

 relation["boundary"="administrative"](37.3612,-122.082,37.4326,-121.965);out body;

It is not:

 relation["boundary"="administrative"](-122.082,37.3612,-121.96537.4326);out body;

This is wrong even if the documentation insists that the bounding box is "usually" specified in the order: min long, min lat, max long, max lat. Which clearly does not work. Er.....

Well, and then one has the question of how one is going to access the data. I wanted to use python, so I installed overpy.

I finally got this.

 $ cat
 import overpy

 api = overpy.Overpass()

 # bbox = min Longitude , min Latitude , max Longitude , max Latitude 

 # My box:

 #    "minlat": 37.3612,
 #    "minlon": -122.082,
 #    "maxlat": 37.4326,
 #    "maxlon": -121.965

 # result = api.query('[out:json][timeout:25]'
 #    '[bbox:{{bbox|-0.489|51.28|0.236|51.686}}];'
 #    'relation["boundary"="administrative"];'
 #    'out body;'
 #    '>;'
 #    'out body;')

 #result = api.query('[out:json][timeout:25]'
 #    '[bbox:{{bbox|-122.082|37.3612|-121.965|37.4326}}];'
 #    'relation["boundary"="administrative"];'
 #    'out body;'
 #    '>;'
 #    'out body;')

 # result = api.query("node(50.745,7.17,50.75,7.18);out;")

 # result = api.query('node'
 #     '["name"~"^Holtorf"](49.7,7.1,50.8,7.25);out body;');

 result = api.query('relation'
                    '(37.3612,-122.082,37.4326,-121.965);out body;')

 print 'relations %d' % len(result.relations)

 for relation in result.relations:
     print ''
     for tag in relation.tags:
         print '"%s" --> "%s"' % (tag, relation.tags.get(tag, "n/a"))
 print ''

Wow. That was certainly ... easy. Yeah, that's the ticket. It was easy.....

Simplified access for administrative entities?

Posted by rayKiddy on 9 July 2015 in English (English)

Is there any single way to get information about administrative entities? It seems to me that there is a class of map users who are interested in admin entities. Perhaps they are interested in city boundaries. In the US, there may also be school districts, water districts and open space districts.

Once one has an id of some relation, it seems ok to get information about that entity.

But I have tried a couple of times to discover how to find these entities and I keep hitting a wall.

I can construct OverPass queries for these things, but they do not work and it is hard to see why they do not work. It is sort of like that old joke: How many prolog programmers does it take to screw in a lightbulb? false.

Here are queries I have tried:

relation ["boundary"="administrative"] ["is_in:country"="USA"] ["is_in:state"="CA"] ["admin_level"="8"] ["border_type"="city"]; out body;


 <?xml version="1.0" encoding="UTF-8"?>
 <osm version="0.6" generator="Overpass API">
 <note>The data included ....</note>
 <meta osm_base="2015-07-09T21:00:02Z"/>


Hm. Not too helpful.

relation ["is_in:country"="USA"] ["is_in:state"="CA"] ["admin_level"="8"] ["border_type"="city"]; out body;

relation ["is_in:country"="USA"] ["is_in:state"="CA"] ["border_type"="city"]; out body;


relation ["is_in:country"="USA"] ["is_in:state"="CA"] ["name"="Ramon Road"]; out body;

relation ["is_in:country"="USA"] ["is_in:state"="CA"] ["name"="Sunnyvale"]; (._;>;); out body;

what are these oddly shaped circular features?

Posted by rayKiddy on 17 October 2014 in English (English)

I am editing in HOT task "#672 - Ebola Outbreak, Port Loko, Bombali districts, road network", and I am seeing these odd somewhat circular features on the ground. This screenshot is about 600 ft (180 m) wide. Groups of them look a bit like honeycombs. What are these?

I suspect that they are man-made and are something agricultural, but I am also not seeing buildings nearby. So, it is not clear. The line is a tertiary road and probably not paved. Any guesses?

several groups of unknown circular features

School Districts Boundaries in California

Posted by rayKiddy on 3 December 2013 in English (English)

Starting on an interesting effort. It should be able to map out CA's school districts on the map. I think that all the pieces are somewhere, nearby.

There are 1003 school in this state. There is TIGER data for the boundaries of all of them. If I can find an easy way to access the shp files, that would be nice, but I am on a Mac and I do not buy Arc stuff. But the info is there. And there may even be a way to locate the lines in the current map that come from TIGER but I do not see it yet. Would something like ':tiger:geoid:0638460' work as a search term? No, it does not. We'll see.

Just stumbling around at this point. I have a mess of data sitting here. And a map with a lot of connected things. It seems as though they should be connectable. We will see.

cheers - ray

Location: de Anza Square Shopping Center, Sunnyvale, Santa Clara County, California, 94087-2248, United States of America

Schools: are they areas or points?

Posted by rayKiddy on 6 October 2013 in English (English)

I am finding a lot of these. It may be that I am misunderstanding things. If you look at the ITO Map layer for schools, it shows a lot of schools that do not have names attached. But actually what I am seeing is that there is a school that has been defined as a point and there is a lot of info on that, such as the name and gnis codes and such, and then there is an area and it has amenity=school and absolutely nothing else on it.

I am thinking that it is better to have the area so I have been transferring the information from the point to the area and deleting the point. But is this right?

And if so, is there an easier way to copy all of the values from one thing (the point) and apply all those values to another thing (the area)? It seems that there should be some magic shift-elbow-click operation that does this. Or maybe not.

And I am using iD and maybe I need to step up to a more capable editor. We will see.

cheers - ray

probably noob questions...

Posted by rayKiddy on 3 October 2013 in English (English)

Very excited to see how this all works. And mostly it works very well. A few things seem to be harder than they might be, or seem to be missing. Perhaps I have not found the proper documentation. I see lots of wiki style pages, but it is a mix.

First, when I am in edit mode, I see a photo overlay, perhaps from satellite images. I go out of edit mode and all I have access to are line modes. What happened to the images? How can I turn on the images in View mode?

Second, I thought I could search for something, see that it returns 10 results and then, you know, radical thought here, see all 10 results on the map. I only see how to select one search result by clicking on it. So, pick one and lose all the other results? But it is a map and can show more than one search result, yes? But how?

Third, I have an idea and wonder if there is a forum to go to or some way to find interested people. I want to map retail food sellers. I am in the US so I would start there and understand the US markets better. My sense is that there are differences in "square feet (or meters) per person" for grocery stores, fast food restaurants, convenience stores and so forth. It has been suggested that people in poorer areas of the US have more access to unhealthy processed food and less access to basic food stuffs, like produce. Certainly many urban areas have almost no grocery stores, but lots of bars, convenience stores, pay-day loan stores and such as that. It would be interesting to see how this maps out.

Please point me to the correct forum for these kinds of questions. Otherwise, I would appreciate your patience with me and your forbearance.

thanx - ray

Location: de Anza Square Shopping Center, Sunnyvale, Santa Clara County, California, 94087-2248, United States of America