Tangled in the Threads

Jon Udell, February 02, 2000

Email and XML

This week, Gavin Brelstaff kicked off a long thread with this posting:

Has anyone investigated automated downloading of web-mail boxes onto local disk? Perhaps parsed into a standard mailbox format?

I guess for current hotmail.com you'd have to negotiate cookies and passwords across redirected servers and domains.

Alan Shutko pointed out that gnus, the infinitely flexible Emacs message reader, can already do this. "Of course," noted Alan, "that depends on the HTML not changing."

This is a subject near and dear to my heart. On the one hand, it's a marvelous thing that every user-accessible Web application is simultaneously a script-accessible service. On the other hand, HTML screen-scraping is a fragile mechanism. Wouldn't it be nice if Web-based services presented more stable APIs? This is, clearly, a good use of XML. An early player in this game, webMethods, took an evolutionary approach. Its B2B Server comes with a toolkit that you can use to wrap an XML interface around an existing HTML-oriented service. The wrapper is still vulnerable to changes in the underlying HTML, and must track those changes, but the interface that the wrapper presents to its clients is stable.

For webMethods, this wrapper approach was intended only as a stopgap measure -- a way to leverage legacy HTML-oriented services. The idea was always that a newly-written service would present XML as its primary interface, with HTML as a derivative of that.

Not so fast, said Fred Pacquier:

Jon, you assume sanity and technical rationality, but these are not a given.

I would say that most Webmail free services depend on people having to manually check in with their browsers, seeing the ads and other paid services, etc. Otherwise, why bother with XML, they could just as well offer POP and IMAP access as an alternative to Webmail : some do, like MyRealBox, but most don't.

Point taken. However, just because you've defined an XML interface, and use it internally to produce the HTML for a free public service, doesn't mean you have to give it away along with the HTML service. You could certainly regulate access to the underlying XML service. Integrators might pay for such access for exactly the same reason that they today pay for packaged component software that helps them build solutions. In a world full of network services, I don't think it's wise to assume that your service will live or die based solely on the number of eyeballs it can attract and hold. A service ought to be prepared to function as a pluggable component that can work easily and effectively in many different contexts.

Fred Pacquier continued:

I get your drift. Still, on the specific topic of webmail, I fail to understand the need to tack an XML interface on what basically is email: POP3 and IMAP4 are native "underlying services" that could be regulated in the same way, no ?

Michel Pelletier responded:

I am currently writing an IMAP server in Python and I have written an IMAP client for Zope and I have serious beefs with the IMAP protocol.

IMAP makes many many assumptions about the underlying architecture your application will have. Often it seems that rfc2060 is actually dictating application behavior. Because the protocol is so complex it is very difficult to write a protocol library for it.

I think a much much more useful protocol for mail would be to use something like XML-RPC, THEN design any application-level assumptions in a higher-level protocol which is what IMAP would become. This would make the definition of client requests and server responses programmatic, no need to write gobs of code just to parse weird inconsistent, unstructured strings going back and forth. For example, IMAP server replises are usually "<TAG> <REPLYCODE> <DATA>" but sometimes they are "<TAG> <DATA> <REPLYCODE> <DATA>", and even more frustrating is that optional command elements come *before* required elements, so "<TAG> <CODE> <OPTIONAL1> ... <OPTIONALN> <REQUIRED>" is the result, which is a pain to deal with because you must work backward from the end to get the required data, which is usually more useful than the optional data anyway.

Exactly. Internet apps were doing simple, text-oriented, client/server, request-response protocols long before the Web made this approach cool. But like UNIX, the Internet has tended to spawn a proliferation of protocol syntaxes, each of which requires custom parsing logic. As Tim Bray pointed out in his keynote speech at the Perl conference several years ago, this is nuts. Tim's the co-editor of the XML spec, and he brought down the house with his satirical tour through the menagerie of weird UNIX config-file formats -- for inetd, fvwm, and more. His point was simply that it is a colossal waste of time to invent new syntaxes, then write (or require other people to write) custom parsers to handle those syntaxes. It's just a bad habit, and it applies equally to config-file formats and to protocols. Let the XML parser recognize what's in a stream of structured text. And then just focus on the code that does the actual work with that data.

XML for custom messaging?

In proprietary mail products such as Notes and Exchange, the concept of a custom message is well developed. You can, for example, define a message whose type is 'Status Report for Project X', with a well-defined structure and a specified set of metadata values (e.g. 'in progress', 'late', 'completed').

Because this custom-messaging capability is a central feature of workflow applications, I often lament the lack of it in Internet-standards-based email applications. The problem has several dimensions:

Although XML is a frequent knee-jerk reaction when the subject of metadata arises, email headers are extensible and are already used to carry all sorts of message metadata. It's the second problem that limits our ability to deploy message templates of the type 'Status Report for Project X.' Although bundled with browsers, the standard Internet mail clients don't support programmable electronic forms the way browsers do. I think that's a shame. There are many days when more information flows through my mailreader than through my browser. We live in email, and the need to better manage that flow of information is intense. Why can't we extend our Internet mailreaders -- other than gnus, that is -- to deal more intelligently with this flow?

The problem is certainly solvable in the web realm. There we're free to invent whatever custom messaging protocols we like, and the XML technology that can help manage those protocols is readily available. To me, though, webmail seems like a poor alternative to the rich user interface of a native GUI mailreader. But I realize that there are two sides to this story:

Michel Pelletier:

It might be a flawed assumption that webmail environments are the minority. Even if so, they are catching up fast. Hotmail had something on the order of millions of subscribers in less than a year or so. E-mail is the most useful function of the internet, and webmail is a way to bring that technology down to the level of people who have a hard time with touch-tone phones.

Alan Shutko:

Yes, isn't the world getting better? You now have less security, minimal admin accountability, fewer features, and advertisements! Sure, you can access your mail anywhere in the world... as can everyone else!

On the plus side, this will ensure more phone communication, since webmail can't handle the 3-400 messages I get a day and everyone will be encouraged to avoid email whenever possible.

Oh, but I forgot! Mailings lists will go away, replaced by web bulletin boards! Instead of having postings come to you, you'll be granted the privilege of checking 20 websites a day with 20 different interfaces, all of which manage to be complete and unmitigated crap. Most of which having itsy-bitsy text input windows and half of which don't wordwrap! Text input forms were not originally meant to input lengthy messages.

It's nice that I can swap the instant response of my laptop for a tiny snippet of time on a machine halfway across the planet. The network delay on every action I take gives me time to meditate on zen koans. Or rather, I would meditate but those flashing advertisements are so enthralling!

It's amazing that people would throw away all the UI improvements of the last 20 years for a 3270 with graphics. Never thought I'd be using screenscraping in the year 2000, but you do what you gotta do.

When contemplating a richer, more functional messaging environment, we shouldn't have to make this choice between highly-usable but fixed-function clients (today's mailreaders) and less-usable but highly-extensible clients (today's browsers). What do you think? Should messaging migrate into the browser, on the assumption that browsers need anyway to natively support the kinds of rich UI machinery that makes email clients so effective? Or should the browser simply become the kind of rich UI engine that precludes the need for a special-purpose messaging client?

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He's now an independent Web/Internet consultant, and is the author of Practical Internet Groupware, from O'Reilly and Associates. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

Creative Commons License
This work is licensed under a Creative Commons License.