Tangled in the ThreadsJon Udell, August 15, 2001
The power of the URL-line
RPC-style services don't replace the humble URLThere is often more value in getting XML out of a web service than in sending XML into a service. In such cases, the human-friendly URL-line offers distinct advantages.
For years I used the phrase "URL command line" to describe the browser's Location (Netscape) or Address (MSIE) window -- that is, the place where you type or paste URLs. My friend Rael Dornfest has more recently coined the more elegant term URL-line.
Like the traditional command line, the URL-line is available for use both by humans, who can type it or paste it or click on it, and by programs, which can invoke it. Unlike the traditional command line, the URL-line tends to be self-documenting -- a key point which I think is often overlooked. When you read the man page for a Unix command, the description is given in formal and abstract terms. You might find some examples at the end of the man page, but you might not. The URL-line's syntax, on the other hand, is quite often demonstrated as a consequence of normal interactive use of web pages. This documentation effect occurs because web pages can be, at the same time, documents and software. The blending of these two styles meant that first-generation web APIs were:
That's true, anyway, for GET requests. POST requests, which do not reveal syntax on the URL-line, made API discovery harder. See my earlier column, Website API Discovery, for some tips on how to expose URL-line syntax that's been hidden from view.
One of the key drivers of the web's explosive growth was its radical transparency. Like the Pompidou museum in Paris, the web wore its infrastructure on the outside, where you could see it. The prominent URL-line was one major example of this transparency, the availability of HTML (by way of the browser's View Source command) was the other.
The ability of an URL-line to travel in an email message adds a whole new dimension to distributed computing. Just this morning, I arranged a meeting with someone in a place neither of us has ever been. Once I navigated a map server to the right level of detail, I only needed to capture that URL and include it in a message, sparing the other person the effort of repeating that sequence of actions.
The notion that URLs can represent frozen and distributable application states is tremendously powerful, and perhaps still not understood as well or exploited as effectively as it should be. Here, for example, is the URL that I sent:
http://maps.lycos.com/mapresults.asp?CMD=MAP&FAM=lycosxml&SEC=geo &GC=X:-71.466594|Y:42.762509|LT:42.762509|LN:-71.466594|LS:20000| c:Nashua|s:NH|z:03060|d:506|p:USA&IC=42.762509:-71.466594:8:&W=400 &H=250&LV=2&CAT=&GAD2=water+street&GAD3=nashua%2C+nh&GAD4=USA
I've inserted newlines here to simulate what will often happen when such an unwieldy URL is included in an email message. Some mailers will manage to render such things as clickable hyperlinks, but many won't. It seems like a silly thing, but it's really quite a serious issue in URL-line design. People are less likely to include an URL-line in messages when they suspect the recipient will find the URL-line to be broken. People who receive broken URL-lines will be less likely to use them -- that is, to assemble the parts into a whole that can be pasted into the browser and invoked.
There are two somewhat contradictory ways to resolve this dilemma. One is to compress the URL to its minimum length, while retaining all the data. Sometimes that's possible, just by squeezing script names and parameter names down to single characters or short abbreviations. Here, it's probably not possible. The other approach is to encapsulate the URL's data into an opaque transaction code, like this:
I've used this method several times to compress "continuation" URL-lines -- that is, links embedded in email messages that verify an email address as part of a site registration procedure. The problem here, of course, is that such an URL is opaque, and fails to document the URL-line syntax and its underlying web API.
In such cases, it may make sense to use both styles. That is, expose the full URL-line by default, but include a "send this link to a friend" feature which offers an alternate and compact URL-line guaranteed to survive email transit unbroken.
A few years back, before the frenzy that now surrounds the phrase "web services," I wrote an article called Measuring Web Mindshare in which I explored how the URL-line syntax of two websites -- AltaVista and Yahoo -- defined two sets of services which could be combined, using script, to create a new service that selected a group of similar sites (as defined by the contents of a Yahoo category) and ranked their web mindshare (as defined by inbound links counted in the AltaVista index).
From a scripter's perspective, the web's URL-line is very like the macro recording capabilities of many popular applications. When you want to automate some behavior in Microsoft Word, you won't likely start by cracking a manual. Instead, you'll turn on the macro recorder and drive the application through a sequence that exemplifies what you want to automate. Then, you'll look at what got recorded. This generated code demonstrates the relevant APIs that you'll need to manipulate in script. And it jumpstarts the process of generalizing from the specific actions recorded to the more abstract capabilities that you want to implement.
URL-line vs web services
When I wrote that article, the idea of web services as a formal discipline, based on something more rigorous than URL-lines and HTML pages, was staring to gain wide currency. One of the key tenets of the web services movement is that first-generation web APIs were fragile. My web mindshare script, for example, depended on HTML screen-scraping, and was easily thwarted by even trivial design changes applied to the HTML pages of the sites it scraped.
In another article written around the same time (fall 1999), I called attention to XML-RPC. Like its big brother SOAP, XML-RPC formalizes the notion of web APIs, by defining a rigorous way to structure both the inputs to and outputs from web services. The example used in that article was MailToTheFuture, a web service that Dave Winer continues to offer. You can use MailToTheFuture interactively, by way of HTML and CGI, or programatically using XML-RPC.
When you use a web form to ask MailToTheFuture to schedule delivery of an email message at some future time, here's the HTTP POST request that's used:
headers: POST /addMessage HTTP/1.0 Host: www.mailtothefuture.com Cookie: email@example.com%09xxxxxxxxxxx Content-type: application/x-www-form-urlencoded Content-length: 133 data receiverMailAddress=udell%40monad.net subject=nashua+meeting messageBody=water+st dateTime=8%2F16%2F2001%3B+12%3A00%3A00+AM
Here's that same request expressed in XML-RPC:
<?xml version="1.0"?> <methodCall> <methodName>mailToTheFuture.addMessage</methodName> <params> <param> <value>firstname.lastname@example.org</value> </param> <param> <value>xxxxxxxxxxx</value> </param> <param> <value> <struct> <member> <name>dateTime</name> <value>8/16/2001; 12:00:00 AM</value> </member> <member> <name>messageBody</name> <value>water+st</value> </member> <member> <name>receiverMailAddress</name> <value>email@example.com</value> </member> <member> <name>subject</name> <value>nashua+meeting</value> </member> </struct> </value> </param> </params> </methodCall>
Clearly the XML-RPC flavor of the request is more complex. I've biased things slightly here, by not pretty-printing it, but you might well wonder what's gained by encoding a handful of name/value pairs in XML.
It's interesting, in retrospect, that I didn't ask myself that question at the time. It's also interesting that I didn't demonstrate the request in my article, but rather the response:
Consider the response that comes back from MailToTheFuture's XML-RPC interface when you send the wrong password:<?xml version="1.0"?> <methodResponse> <fault> <value> <struct> <member> <name>faultCode</name> <value> <int>4</int> </value> </member> <member> <name>faultString</name> <value> <string>The password is incorrect.</string> </value> </member> </struct> </value> </fault> </methodResponse>
Like the request, this response is well-formed -- and thus automatically parseable -- XML. Every XML-RPC service will use this same pattern. It's true that, for each application, you'll need to decide how to handle faultCode 4. But you won't need to guess that the output is an example of a methodResponse, or that its value is a fault object containing a struct made up of a faultCode and a faultString.
What I subliminally knew then, but have since become more aware of, is that while XML-RPC and SOAP are inherently symmetric, there is often more value in getting XML out of a service than in sending XML into a service. What's more, the URL-line remains, for lots of reasons, a really useful way to request services that may or may not emit XML, and if they do emit XML, may or may not emit SOAP or XML-RPC packets.
Here's just one example. A project I'm working on has a web-based reporting system. All of its features are exposed as URL-lines, and as you use the system you discover that these are constructed in a really powerful way. Here's an example:/stats/report?REPORT=1&DATE_1=08/01/2001&DATE_2=08/31/2001&ACCOUNT_ID=93
This is a thinly-disguised SQL passthrough. A script that wants to gather data about accounts 1 through 92 for this date range, or about account 93 for another date range, can easily re-parameterize the SQL query that's embedded in this URL-line. And, while an HTML table can be a quite regular and parseable representation of SQL data, it lacks the nicely self-descriptive qualities of an XML representation. So, in this reporting system, every report pages comes with a Download XML button that invokes a parallel URL-line which invokes an XML results package:/stats/reportXML?REPORT=1&DATE_1=08/01/2001&DATE_2=08/31/2001&ACCOUNT_ID=93
It is certainly possible to XML-ize these queries. But is it useful to do so? The input/output symmetry of XML-RPC and SOAP matters when you are passing around complex nested data structures. In a great many situations, though, people (and programs) prefer to issue simple requests that may yield complex results.
Here's another example that takes us full circle. Rael Dornfest's RSS viewer, Meerkat, offers a rich URL-line API that is also available by way of XML-RPC. Rael's article on Meerkat's XML-RPC API demonstrates a PHP XML-RPC request asking Meerkat for 3 days' worth of XML or Java stories from channel 724:$f = new xmlrpcmsg("meerkat.getItems", array( new xmlrpcval( array( "channel" => new xmlrpcval(724, "int"), "search" => new xmlrpcval("/XML|[Jj]ava/", "string"), "time_period" => new xmlrpcval("3DAY", "string"), "ids" => new xmlrpcval(0, "int"), "descriptions" => new xmlrpcval(200, "int"), "categories" => new xmlrpcval(0, "int"), "channels" => new xmlrpcval(0, "int"), "dates" => new xmlrpcval(0, "int"), "num_items" => new xmlrpcval(5, "int"), ), "struct" ) ) );
Here's the same request as an URL-line:http://www.oreillynet.com/meerkat/?_fl=xml&s=XML|[Jj]ava&c=724&t=3DAY&_de=200
If you subtract the _fl=xml from this URL-line, you'll produce a normal interactive HTML page. If you put _fl=xml back, you'll select Meerkat's XML flavor. Results then depend on your browser. In Netscape 4.x, you'll be prompted to download an XML file. In MSIE, you'll see the XML directly. This is exactly equivalent to my reporting example: asymmetric use of a compact URL-line to request a richly-structured response.
REST vs RPC
I have been aware of this asymmetry for some time but, as I've said, only subliminally. What prompted me to write this column was an entry in Tim O'Reilly's weblog, entitled REST vs RPC. He refers to discussion, on the decentralization and FoRK mailing lists, about how RPC (remote procedure call) technologies like XML-RPC and SOAP relate to REST (Representational State Transfer), which Roy Fielding and Richard Taylor, in their paper Principled Design of the Modern Web Architecture, say is "the unpublished rationale behind the modern Web's architectural design."
Here's some of what Fielding and Naylor say about REST:
A distributed hypermedia architect has only three fundamental options: 1) render the data where it is located and send a fixed-format image to the recipient; 2) encapsulate the data with a rendering engine and send both to the recipient; or, 3) send the raw data to the recipient along with metadata that describes the data type, so that the recipient can choose their own rendering engine.
REST provides a hybrid of all three options by focusing on a shared understanding of data types with metadata, but limiting the scope of what is revealed to a standardized interface. REST components communicate by transferring a representation of the data in a format matching one of an evolving set of standard data types, selected dynamically based on the capabilities or desires of the recipient and the nature of the data.
REST's architectural style, which the paper elaborates in detail, was chosen to satisfy a specific goal: the construction of a distributed hypermedia system -- "a shared information space," as Tim Berners-Lee has said, "through which people and machines could communicate."
What people have begun asking, lately, is whether the RPC-based architectural style at the heart of the web services movement involves too much overhead, and whether it's in fact the wrong approach.
I don't think that's the case. But I do think we should pay attention to the asymmetry I've noted here. HTML screen scraping is a really stupid way to consume web services. There's no question in my mind that most of the hard-wired HTML that represents the current population of web services would be better recast as XML, rendered as HTML either server-side or client-side when needed, or else delivered straight for downstream pipelining. Some of those services should also get the full RPC treatment, either because complex inputs as well as complex outputs are involved, or because communication is more of the machine-to-machine flavor than the machine-to-person or person-to-machine flavor. This isn't, though, an either/or dichotomy. As Meerkat proves, a rich URL-line API is fully compatible with an RPC-style API. The truth is that we need both, and ideally our development tools should make it trivial to support both.
You don't have to read a dissertation on web architecture to know this. You just need to send somebody a map URL that pinpoints the location of a meeting. Such fluid interaction among people and machines, when it works right, demonstrates what Berners-Lee meant the web to be.
Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/
This work is licensed under a Creative Commons License.