Tangled in the Threads
Jon Udell, March 27, 2002XSLT Explorations
When I started developing for the Web, I was astonished to see how easily I could conjure applications seemingly out of thin air, by transforming patterns of text into other patterns of text. I'm still amazed by this phenomenon, though perhaps it shouldn't be so surprising. Life itself springs from a textual data structure (DNA) which transformative processes turn into other textual data structures (proteins). A programming language designed expressly to transform structured text is therefore, by definition, a really powerful tool. So despite my reservations about Extensible Stylesheet Transformations (XSLT), which to my mind awkwardly straddles the worlds of procedural and declarative programming, I find myself using it more and more often for things I used to do in parser-equipped scripting languages such as Perl or Python.
In his End Tag column for XML Magazine, Adam Bosworth -- former Microsoft XML evangelist and now BEA Systems' VP of Engineering -- called for a post-XSLT solution:
We need a language that can natively support XML as a data type and yet can gracefully integrate with the world of objects (Java or otherwise) and can take advantage of the self-describing nature of XML by supporting querying of its own variables. This language as used by humans will look like a programming language, not an XML grammar.
This sounds good to me. But I lack the imagination to see clearly what that language should be. If it's going to exist, we'll have to bootstrap our way into it. Using XSLT for all it's worth is probably the best way to do that. In that spirit, here are some things I've done lately with XSLT, and some reflections on doing them.
The Muench method is the Schwartzian transform of XSLT
I recently had to transform a data set like this:
<recordset> <record> <id>153</id> <label>USA</label> </record> <record> <id>155</id> <label>USA </label> </record> <record> <id>156</id> <label>VENEZUELA</label> </record> </recordset>
Into this result:
USA 2 VENEZUELA 1
Here is the XSLT solution:
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:key name="countries" match="record" use="label" />
<xsl:template match="recordset">
<table>
<xsl:for-each select="record[generate-id(.) = generate-id(key('countries', label)[1])]">
<xsl:sort data-type="number" order="descending" select="count(key('countries',label))"/>
<tr>
<td><xsl:value-of select="label"/></td>
<td><xsl:value-of select="count(key('countries',label))"/></td>
</tr>
</xsl:for-each>
</table>
</xsl:template>
</xsl:stylesheet>
This is called the Muench method, for Oracle's Steve Muench, author of Building Oracle XML Applications. Following the analogical style of my last column, I'd have to say that the Muench method is the Schwartzian transform of XSLT. On the Mulberry Tech mailing list, Muench writes:
I started thinking, "How does a database make grouping and sorting fast?" Well, it does that by creating indexes. Then it dawned on me that creating an XSLT key like:
<xsl:key name="tid" use="tracker-id" select="."/>
was conceptually similar to the SQL DDL statement:
CREATE INDEX tid ON currentDocument( tracker-id );Namely, it asks the system to "waste" a little storage to keep track of some extra data structures to make access to this *specific* piece of information -- which presumably is happening frequently for a given task -- much faster.
That's a lot like Randal Schwartz's famous Perl transformation:
@sorted_files =
map { $_->[0] }
sort { $a->[1] <=> $b->[1] }
map { [ $_, -s $_ ] }
<*>;
Both of these idioms effectively create indexes that prevent a lot of repetitive scanning. I won't try to describe here how they work. If you Google for Schwartzian transform or Muench method, you'll find better explanations than I can give. It's the Muench/Schwartz analogy itself that I find most thought-provoking. Both idioms rate as major discoveries in their respective communities. This is a testament to the unpredictable power of programming languages, and to the inventive spirit of the people who use them. That said, there's a sense in which these methods reinvent the wheel. As Muench notes, his discovery was inspired by classic database technique. This suggests, to me, that our hypothetical next-generation XML programming language should perhaps evolve in parallel with the emerging technologies for XML storage.
Blog locally, rewire globally
Blogging has been a major theme in my last few columns, and last week my weblog intersected with XSLT. Dave Winer had cooked up a new driver architecture for the RSS aggregator in Radio Userland. Its purpose was to empower the aggregator to read non-RSS formats. A few days later, I read about an application of the driver that reformats a non-RSS Nasa newsfeed for use in Radio's aggregator. I thought this was cool, but wondered whether an XML transformation couldn't do the same job in a more general way -- solving the problem not only for Radio, but for any RSS reader.
Radio doesn't include XSLT support, and while IE and now Mozilla do, you still can't really make generally available a service that relies on client-side XSLT processing. Fielding a server that's equipped for XSLT -- using IIS/MSXML, Apache/Xalan, or something else -- requires a fair amount of effort. Fortunately, as I learned from XML wunderkind Aaron Swartz, there's a public XSLT service that you can use on the URL-line in a pipelined fashion. This was just what I needed. I wrote an XSLT stylesheet (details here) that turns Nasa's format into RSS, and then I subscribed to that channel in Radio, thereby publishing its URL on my Radio channelroll -- that is, the auto-generated list of feeds I'm receiving in my aggregator. Here is the URL:
http://www.w3.org/2000/06/webdata/xslt?xslfile=http://radio.weblogs.com/0100887/gems/liftoff.xml&xmlfile=http://liftoff.msfc.nasa.gov/Content.xml&transform=Submit
The red text is the URL of the W3C service; the blue text is the URL of the XSLT file I uploaded to my weblog; the green text is the URL of the Nasa newsfeed. Although there is no XML-RPC or SOAP or WSDL anywhere in sight, I suggest that this is a beautiful example of the kind of service composition that will characterize the next phase of the Web services movement.
A couple of days later, a Nasa feed popped up at NewsIsFree. When I looked at it, I was reminded of an unresolved problem with my solution. The Nasa feed actually contains three different channels, like so:
<newsfeed>
<channel id="Liftoff" name="Liftoff to Space Exploration"
url="http://liftoff.msfc.nasa.gov/">
...items...
</channel>
<channel id="Science" name="Science @ NASA"
url="http://science.nasa.gov/">
...items...
</channel>
<channel id="SpaceCal" name="NASA Space Calendar"
url="http://www.jpl.nasa.gov/calendar/">
...
</channel>
</newsfeed>
The NewsIsFree feed didn't differentiate among them. My transformation did, by selecting one of the three channels:
<xsl:template match="channel"> <xsl:if test="@id='Liftoff'">
But I hadn't accounted for Science and SpaceCal. Fixing this was a profoundly easy thing to do. I cloned the XSLT stylesheet twice, and changed 'Liftoff' to 'Science' and then 'SpaceCal'. As soon as I saved those files, I was done. Radio upstreamed them to my server, where they became available to the W3C transformation service. Publishing another blog entry containing the three URLS was all it took to advertise these newly available services to the whole world.
In fact, there was no need to clone the XSLT file and change one word in each copy. Rather, I should have parameterized the stylesheet, then cloned the URL twice and changed one word in each of those. My first effort to do that had failed because the W3C's service, based on James Clark's XT transformation engine, uses an older syntax than I'm familiar with. But when I tried again just now, I prevailed. Here is a parameterized stylesheet, and here are the Liftoff, Science, and SpaceCal channels derived from it.
Rael Dornfest coined a phrase I like very much: blog locally, publish globally. The notion that blogging can also be a way to compose and publish services, as well as content, is terrifically cool.
Pushing the envelope of client-side XSLT
Although client-side XSLT isn't yet sufficiently standard to support generally-available services, there's an lot of transformative power under the hood of modern browsers. My final case study involves a UI prototyping exercise that I've done using Internet Explorer. The task required me to create a live demo that explores alternate ways to present results from a search engine, based on an experimental XML API. The engine (which as you might guess, is Safari) organizes results according to books and chapters. I'm exploring an alternative mode that organizes results, instead, by individual sections. Doing that requires fairly major restructuring of the output. This presented me with another of those "Ob-Platte" puzzles I mentioned last time, namely:
What is the Perl hashtable of XSLT?
In other words, in Perl I'd attack this problem by scanning the result set, building up a hashtable of documents with their IDs and relevance scores, sorting the hashtable, and then traversing it to produce output. What kind of intermediate data structure is available to XSLT? As it turns out, you can use the document itself for this purpose. The <xsl:variable> construct, which assigns a simple value to a variable, can also (when it contains XML content) assign document structure to the variable. Suppose you are given something like this:
<results>
<book id="17">
<chapter id="5">
<section id="273">
<hit score="92">...</hit>
<hit score="73">...</hit>
</section>
</chapter>
<chapter>
....
</chapter>
</book>
<book>
...
</book>
</results>
Here is an XSLT construct that creates a new, section-oriented structure, and assigns it to $savedSections.
<xsl:variable name="savedSections">
<xsl:for-each select="/results/book/chapter/section">
<xsl:element name="savedSection">
<xsl:attribute name="score">
<xsl:value-of select="@score"/>
</xsl:attribute>
<xsl:attribute name="sectId">
<xsl:value-of select="@id"/>
</xsl:attribute>
<xsl:attribute name="chapId">
<xsl:value-of select="../chapter/@id"/>
</xsl:attribute>
<!-- gather more ancestral info here -->
<xsl:for-each select="./hit" >
<xsl:element name="savedHit">
<xsl:copy-of select="."/>
</xsl:element>
</xsl:for-each>
</xsl:element>
</xsl:for-each>
</xsl:variable>
When it came time to use $savedSections, though, I ran into another analogy problem. The variable does not contain a node-set which can be iterated over using <xsl:for-each>. Rather, it contains a result tree fragment -- that is, a flattened, stringified representation of the content. As many others have discovered before me, there are special functions to convert from one representation to the other. Using MSXML, it's <msxsl:node-set>, and it works like so:
<xsl:for-each select="msxsl:node-set($savedSections)/savedSection"> <xsl:sort data-type="number" order="descending" select="./@score"/> <!-- etc. -->
The same thing is done, in Xalan, with <xalan:nodeset>. This conversion happens so often, in fact, that XSLT 1.1 aims to eliminate the need for it, by enabling node-sets to be created directly.
There was one more conceptual hurdle to overcome. When constructing $savedSections, I had originally written:
<xsl:for-each select="./hit" >
<xsl:element name="savedHit">
<xsl:value-of select="."/>
</xsl:element>
</xsl:for-each>
When I viewed the output based on this code, I could see the new structure I had built. But I couldn't write a template that would match the <savedHit> element. Thus I learned, once and for all, the difference between <xsl:value-of> which (again) produces a flattened and stringified representation, and <xsl:copy-of> which produces a node-set.
Although <msxsl:node-set> is incredibly handly, it meant that I couldn't use this stylesheet to pipeline search results through the W3C service. Using another server to do the same thing was an option, but I was impatient to deploy something and didn't have access to an public instance of IIS configured for XSLT scripting. So I decided to try a client-side solution. Here it is:
<html>
<head>
<script>
function transform()
{
var searchtext = document.searchForm.searchtext.value;
var xsl = new ActiveXObject("MSXML2.DOMDocument");
xsl.async = false;
var xslurl = 'XSLURL'; // not public yet
xsl.load(xslurl);
if (xsl.parseError != "")
{ alert("cannot load xsl: " + xsl.parseError.reason + ", " + xsl.parseError.line); }
var xml = new ActiveXObject("MSXML2.DOMDocument");
xml.async = false;
var xmlurl = 'XMLURL?text=' + searchtext + '&level=hit'; // not public yet
xml.load(xmlurl);
if (xml.parseError != "")
{ alert("cannot load xml: " + xml.parseError.reason + ', ' + xml.parseError.line; }
var theString = xml.transformNode(xsl);
document.write (theString);
}
</script>
</head>
<body>
<blockquote>
<form name="searchForm" method="post" onSubmit="javascript:transform()">
<input name="searchtext">
<input type="submit">
</form>
</body>
</html>
This worked fine for me locally. But when I uploaded the HTML/JScript file and its counterpart XSL stylesheet to a server, it failed. After much fruitless headscratching and Googling, I finally realized why. MSIE, laudably, doesn't trust a file loaded from a remote location (namely, the aforementioned HTML/JScript file) to in turn load a file from a different remote location (namely, the search engine). The setting that controls this behavior is found here:
Tools->Internet Options->Security->Internet Domain-> Custom Level->Miscellaneous->Access data sources across domains
It's disabled by default, as is proper. You can enable it or, as I suggested people do for the duration of this test, you can set it to prompt for permission.
That solved the problem, and enabled the prototyping exercise to go forward. I can't heartily recommend this approach, though. There are too many versions of MSXML floating around, and when you start trying to sort out the differences among them, you're on a slippery slope. Mozilla's XSLT engine, meanwhile, is a horse of a completely different color. So for the time being, XSLT is best exploited server-side. Still, it's impossible to ignore the power and appeal of client-side XSLT. There are already today whole applications built around this technology -- notably, the offline version of Salesforce.com's customer relationship management software. I continue to think that the fullest realization of the Web services architecture will be a peer-to-peer network whose nodes are consumers, transformers, and producers of XML. Distributing the transformative power throughout the network seems an obvious and inevitable next step.
Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/.
This work is licensed under a
Creative Commons License.