Jon Udell: The Event-Driven Internet

Tangled in the Threads
Jon Udell, November 26, 2001
The Event-Driven Internet

What worked for the GUI should work for the Internet too

In the web services paradigm, software speaks when spoken to. With pub/sub, software speaks when it has something to say.

When it comes to defining the emerging next-generation Internet, we are all blind men groping an elephant. Back in July, in a column called Kenamea's Application Network, I highlighted some of the agreed-upon features of the beast. These included peer autonomy, reliable and transacted communication layered over HTTP, network-global (versus per-application) access control, and a publish-and-subscribe event notification scheme.

Recently, I talked with the folks at KnowNow, whose technology touches many of these same bases. Event notification has been a longstanding research interest of KnowNow's founders, Rohit Khare and Adam Rifkin, so it's not surprising that pub/sub is a central feature of their system.

Like Kenamea, the KnowNow system arranges for a persistent HTTP connection to the browser, and routes network events to JavaScript handlers in order to create realtime effects. A simple example, adapted from one of the KnowNow samples, shows how the notification can work. Here's the setup:

The KnowNow event router (downloadable from KnowNow) is running on a machine, and listening on port 8000.

The router's kn_apps folder has an application subfolder -- let's say it's called Dommy -- with an index.html page. On that page, there can be one or more event-enabled DOM elements.

The index.html page refers to some JavaScript that does the event-enabling -- that is, it wires the DOM elements to the event router.

Given this setup, here are the effects:

The content of every event-enabled element is associated with a router-managed pub/sub topic, and stored persistently in the router's cache.

When the page loads, it updates the content of each event-enabled element in response to a message received from the router.

If an element's topic does not already exist, the router creates it.

When the topic's data is modified, the page receives an event and updates itself in realtime.

The page can itself modify the topic's data, by way of an HTML form that's wired to the topic.

Here is an example of an event-enabled page. It has two event-enabled elements: a paragraph, and a form.
<html>
<script src="/kn/?do_method=lib"></script>
<script src="../jscomponents/form.js"></script>
<script src="dommy.js"></script>

<body onLoad="init()">

<p>This element is subscribed to /what/dommy/test:</p>

<p id="kn:/what/dommy/test">test</p>

<p>This form publishes to /what/dommy/test:</p>

<form action="kn:/what/dommy/test" method="POST">
<input type=text name="kn_payload">
</form>

</body>
</html>
When first loaded, this page creates the topic /what/dommy/test. By convention, KnowNow topic names are hierarchical. The first level describes the general type of a topic (e.g., who, what, where), the second names an application, and the third names a specific element.

Suppose the URL of this page is http://SomeServer.com:8000/kn_apps/Dommy/. I load that page into my browser, which happens to run on the same box as the router, and you also load the page into your browser which is running somewhere else. Now, I type "it changed!" into the form and hit Enter. The paragraph element, which was formerly "test," becomes "it changed!" -- in both our browser instances.

That the element changed, in my browser, might not seem too surprising. Obviously, JavaScript wired to the DOM can effect such a change. What's more, the router is local. But things aren't as they might seem. The router might as easily be on your machine, or anywhere else. And JavaScript is not talking directly to your DOM. It is talking to the event router. Your DOM is listening to the event router (and so is mine). As a result, you are going to be quite surprised to see the element change in your browser.

What will particularly surprise you, in this situation, is that you haven't downloaded or installed any special software. This is one of the interesting differences between Kenamea and KnowNow. In the case of Kenamea, the only way to receive events is to run a local instance of Kenamea's version of an event router, which is called the harness. The Kenamea harness is a robust piece of software that does lots of useful and important things, including secure transacted messaging and single sign-on. But it doesn't flow invisibly and effortlessly to the user. One of KnowNow's clever innovations is a JavaScript-only microserver, which is served up by the event router to visiting browsers, and which immediately joins such browsers to an event network.

Programming the KnowNow microserver from JavaScript and Python

Here's the JavaScript code referenced from the index.html page:
data_points = new Array();

// rip through the document and find all elements with topics as ids. 
// create an object for each such element, and push it to the data_points array

function init() 
    {
    for (i=0; i < document.all.length; i++)
        {
        if (document.all[i].id.substring(0,4) == "kn:/")
            {
            newId = i + document.all[i].id,
            data_points[data_points.length] = new DataPoint
                (
                newId,
                document.all[i].id.substring(4)
                );
            document.all[i].id = newId;
            }
        }
    
    // now that we have identified all of the datapoints, enable them

    for (i=0; i < data_points.length; i++)
        {
        topic = data_points[i].topic;
        kn.subscribe
            (
            topic,
            function() { handleEvent(arguments[0], topic)}, 
            {do_max_n : 1}
            );        
        }
    }

// constructor for a datapoint.
// datapoints are parts of the page that have been specified as 
// live and should receive events.

function DataPoint(id, topic)
    {
    this.id = id;
    this.topic = topic;
    }

function handleEvent(e,topic) 
    {
    for (i=0; i < data_points.length; i++)
        {
        if (topic == data_points[i].topic)
            { document.getElementById(data_points[i].id).innerHTML = e.kn_payload; }
        }
    }
This is pretty interesting stuff! The microserver was sourced into the index.html page by this line:
<script src="/kn/?do_method=lib"></script>
The microserver creates hidden frames in the page through which to receive and post events. And it exposes the pub/sub API to JavaScript code running in the page. I particularly like the web-friendly style of this approach. Topics just look like URLs. You read them, and update them, by means of simple conventions that are well-known to web developers.

There are, naturally, other ways to access this pub/sub API. On the KnowNow site you can find a Python implementation of the microserver. It suggests ways to build event-aware apps that talk directly to an event router (that is, to its HTTP API). These apps need have nothing to do with interactive JavaScript/DOM trickery. Here, for example, is a Python script that will publish a change to our sample topic:
from pyKnowNow import kn

if __name__ == '__main__':
    k=kn(host='SomeServer.com',port=8000,user=None,password=None)
    k.start()
    k.publish('/what/dommy/test',{'kn_payload':'changed from python!'})
    k.stop()
If you and I happen to be monitoring that topic in our browsers, the paragraph element will change interactively. But nothing says that the parties interested in that topic are always, or only, people running browsers. Of more general interest is the notion of a computing fabric in which processes subscribe to events, are notified of changes, and then take appropriate actions. Here's the skeleton of such a process, based on the Python microserver:
from pyKnowNow import kn

def callback(event):
    for item in event.rawEvent.items():
        print item[0], ':', item[1], '\n'

if __name__ == '__main__':
    k=kn(host='SomeServer.com',port=8000,user=None,password=None)
    k.start()
    k.subscribe(kn_from='/what/dommy/test',cb=callback)
    while k.connected():
        pass
    k.stop()
This process will be notified whenever an event is posted to /what/dommy/test, and will echo the event. The notifier might be a DOM-wired form on a web page, but could as easily be some other process which (as in the first Python example) is talking to the event router.

Variations on the pub/sub theme

Khare and Rifkin believe passionately that Internet-scale networks of event-aware processes will be one of the key architectural innovations of the next decade. I think they probably right. If you think about it, there is nothing very new about the current web services craze. It feels more like consolidation than innovation. The web has always been about HTTP endpoints exchanging packages of formatted data. You could brutally but perhaps not unfairly say that web services only extends this model to communication among processes as well as among people.

There is nothing in the architecture of web services, per se, which breaks free of the conventional client/server, or call/response, model of data exchange. Publish/subscribe, as an architectural principle, promises something quite different. It leads to classes of applications that do not simply speak when spoken to. Rather, they speak -- to other applications and to people -- when they have something to say.

If this event-driven style is going to be an important feature of the Internet landscape, we should expect to see other examples of its use. And that's exactly what I am seeing. Over at UserLand Software, for example, Dave Winer's crew have added pub/sub capability to the xmlStorageSystem, and also to Radio UserLand. In the latter case, a publisher of an RSS channel can specify, using a cloud element, a registration procedure that a reader of the channel can use to sign up for change notifications. Here is Dave's example:
<cloud domain="data.ourfavoritesongs.com" port="80" 
  path="/RPC2" registerProcedure="ourFavoriteSongs.rssPleaseNotify" 
  protocol="xml-rpc"/> 
If I want to find out about changes immediately, rather than on the usual hourly schedule, I'll call ourFavoriteSongs.rssPleaseNotify, and ask it to alert me -- by means of XML-RPC, SOAP, or HTTP POST.

The pub/sub style is also woven deeply into Hailstorm. All of its services are kinds of data storage, and many of their XML-based APIs support registration for change alerts. Here too, web services protocols are the building blocks, but the real action takes place at a higher level of abstraction. To register for change alerts in Hailstorm, you use its data manipulation language (HSDL) to update a service document -- for example, the one that supports the .NET Inbox service. You'd add a /myInbox/subscription node, with a /myInbox/subscription/trigger that specifies an XPATH expression defining the set of nodes in the service document that are sensitive to change.

Internet software will, undoubtedly, make growing use of events to minimize polling, and ensure faster and more complete sharing of information among cooperating processes. Will these event mechanisms converge on a standard? Perhaps, though I'm not sure that moving this stuff up a level in the protocol stack is a huge priority, at least not yet. With Hailstorm, in particular, issues of identity and ownership of personal data loom larger. If you want to offer a service that reacts to changes in my .NET Inbox, we'll both need to live in the realm of Passport authentication, and I'll need to grant you permission to install the subscription trigger that watches for changes. If we can arrive at an open standard for doing these things, with multiple implementations from the commercial and open source worlds, then we can have a meaningful discussion about standardizing an event mechanism.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/

This work is licensed under a Creative Commons License.