Search

OpenlyLocal Council map, missing data and schemaless XML

I should be able to provide a single XSLT  to transform the full OpenlyLocal list of councils to KML, but I'm thwarted by another problem with XSLT on our eXist platform, namely a problem in setting the output media type, so it was back to XQuery to do the job.



declare function local:council-to-placemark ($council) {
let $description := 
<div> 
   <ul>
     <li>OpenLocally</li>
     <li>Home</li>
     <li>Party Markup</li>
   </ul>
</div>
return
<Placemark>
       <name>{string($council/name)}</name>
       <description>{util:serialize($description,"method=xhtml")} </description>         
       <Point>
          <coordinates>{string($council/lng)},{string($council/lat)},0</coordinates>
       </Point>
</Placemark>
};

declare option exist:serialize "method=xml media-type=application/vnd.google-earth.kml+xml  indent=yes  omit-xml-declaration=yes";
let $x := response:set-header('Content-Disposition','attachment;filename=councils.kml')

let $councils := doc("http://openlylocal.com/councils.xml")//council
return
<kml xmlns="http://www.opengis.net/kml/2.2">
   <Folder>
      <name>Councils </name>    
        { for $council in $councils$councils[empty(*:lat/@nil)]
          return local:council-to-placemark($council)
         }
   </Folder>
 </kml>


KML GoogleMap  OpenStreetMap

The XML is a pleasure to work with and in my view a great example of the value of XML in OpenData.  I did have one stumble, when trying to fliter out councils with missing geo-coding. Handling missing data in XML is a choice of convention:  [nothing] or <lat ></lat> but Chris has chosen <lat nil="true"/>. The later form is trickier to filter for - e.g,. $councils[empty(*:lat/@nil)]

A while back I had a exchange with NOAA over one of their weather data feeds since, contrary to the attached schema which rightly defined data types and optionality, the data included missing values in the form <temp_c>NA</temp_c>. They have since corrected the feeds which  are now valid e.g. Newark and its schema.The explanation the NOAA representative gave me at the time was that consumers were confused when data was missing because they assumed the feed must be faulty.  I presume users are have become used to handling XML feeds now.

The case with the OpenLocal data seems slightly different. This XML contains datatype attributes in the elements e.g. <lat type="float">-3.5247</lat>, as well as the empty elements. Type information and the presence of the missing elements (sometimes with the nil attribute set, others with empty content)  reduces the need for schemas (there is no schema attached)and is sufficent to transform to HTML pages and just enough to to guide basic data entry. This is an interesting compromise which fits well with Native XML databases such as eXist. Food for thought.

 

The nil=true attribute for null fields is the default style of Ruby on Rails' XML serialisation. It's good practice to specify NULL explicitly as it's not the same thing as an empty string. Rails generates default API code automatically if you're using the scaffolding feature, which, as you can see here is artbitrary, "schemaless" XML. XML is designed to be self-describing so as long as you've chosen your table and column names appropriately it's extremely usable.

Thanks for a good example of using XQuery to transform this kind of data.

(Oh, and it should be OpenlyLocal not OpenLocally in the title).

@adrianshort Thanks for the explanation and the correction.