Tangled in the Threads

Jon Udell, May 1, 2001

The First Mass Web Extinction

Web services are great, until they disappear

In an essay posted to the newsgroup, John Faughnan stresses the need for portability and "exit strategies"

Last week, John Faughnan -- a longtime correspondent -- posted a wonderful essay to my newsgroup. In his essay, John imagines himself in the year 2060, remembering the "first mass web extinction":

It was in the spring of 2001, 20 years before the first AI. I remember trying to find a send-fax web service. Every one I tried was merged or showing signs of advanced web site decay. Then I tried HotOffice.com and got a bankruptcy notice. Another site said they weren't accepting new registrations but was trying to stay open for their business customers. The free sites were gone, and the pay-for-service sites were all dying.

Yes, we'll be able to tell our grandchildren (or grandbots) about this one. In a short period of time vast arrays of web services are gone. The ugly, bad and good alike. Some of the best fee-based services were killed by free competitors sustained by crazy money in a lunge for market share -- when the climate changed the free services died, but the weakened fee-based services went too.

We'll eventually recover. There a lot of open niches now, and new species will inhabit them (AOL, MSN, Yahoo? sigh). But we shouldn't miss the very interesting lesson that HotOffice and other ASP services are teaching us.

If you use DBASE for your office, and Ashton-Tate dies, you still have the code and the data. Eventually you'll need to migrate, but you know the data model, you have the data, it's manageable.

If you use WordPerfect Office, and Corel dies, migration is harder. You can still use the application though, you have time.

If you're using HotOffice.com or another bankrupt or withdrawn ASP service to run a business, you're dead. You can't get at your message repositories, you can't get at your data, and you don't have the services. Users have focused on network reliability as an ASP weakness (and it is a big weakness with our immature networks), but the real issue is the inability to migrate from one ASP service to another, or from ASP to desktop.

It's almost as bad with licensing software (.NET), though if your data is in XML there's a greater potential for migration.

There are technical solutions to this problem. Obviously Notes/Groove and P2P applications point to one approach - data replication/synchronization on the client. Standard data structures (XML) mean that migration is more plausible. Contracts and organizational/legal innovations will help too.

My concern is that we learn these lessons about migration and exit strategy as soon as possible and as thoroughly as possible. Let's make the next extinction a bit different. Let's say NO to non-public file formats for leased software.

Do we feel like these lessons are being learned? Or should we just sit back and wait for the next extinction?

Well said, John! Like most of you, I've been feeling the effects of the first extinction. Last year, for example, I published an online report entitled Internet Groupware for Scientific Collaboration. One section of the report was a case study on how to use TimeDance.com's group scheduling service to coordinate meetings. Then, just days after I published the report, TimeDance.com went dark. Oops!

Towards portable discussion services

The report was itself connected to another web service -- QuickTopic -- which is (so far, thankfully) alive and well. QuickTopic enabled me to discuss, with readers of the report, the fate of TimeDance. But when QuickTopic's creator, Steve Yost, read John's essay, he thought about the transience of all web services and posted this reply:

Steve Yost:

A friend alerted me to this thread. I was amazed by the way eCircles went dark, and Zaplets has changed their model, both saying essentially "please copy and paste to save your data".

I plan for Quick Topic to be around a good long while, but to assuage users' fears, it would be great if we web apps at least provided a rich means for users to capture their data. I'd like to allow users to download entire threads as XML files (surprisingly, nobody has explicitly asked for this). Let's pick a standard for thread archiving and do it! I'd like to hear from other web app providers about this.

Yes, by all means, let's have such standardization. I've built a few discussion converters in my day -- from NNTP to various web formats, and vice versa. It would be easy to represent the basic objects -- a message, a thread -- in XML. Why aren't users demanding this? It's a classic chicken-and-egg problem. Users won't demand that one discussion service export XML until they can see that other discussion services will import it. In this particular case, though, I don't think we need or want to crank up a W3C-style standards process. We just need Steve and a few of his colleagues who maintain other web-based discussion services to agree on a format.

While they're at it, I'd love to see them agree on a handful of XML-RPC and/or SOAP APIs -- just the basic primitives needed to navigate a message base, and post messages. Currently, these services hardwire the features they support into the user interfaces they express through HTML. Of course every service should try to be special in its own way. But special behavior should be layered on top of core functionality. Users want to bind monogomously to UI, while at the same time promiscuously flitting from one service to another. Why shouldn't I be able to use the Ultimate BBS interface, or another web interface, or indeed a "fat client" GUI interface, to converse in a QuickTopic discussion, or any other kind of discussion?

There are two general objections to this argument:

Sometimes, the universe sends you a message you just can't ignore. As if the need for portable discussion services weren't already plain enough, last week I also heard that BIX (the BYTE Information Exchange) -- which had been running in an unsupported way for months -- has finally passed away. The strength of BIX, both during and after its association with BYTE, had little to do with its primitive UI or features, and everything to do with its loyal community and civilized discourse. These excellent qualities may yet survive in a new service, Noise Level Zero, which aims to recreate a BIX-like environment. NLZero will apparently be compatible with Galahad, the popular GUI shell that made BIX palatable to those not charmed by its basic teletype interface.

A while back, I talked with some BIXen who were considering how to migrate BIX to a new substrate. It was a daunting project. If NLZero succeeds -- and I hope it does -- it will be thanks to extraordinary effort on the part of dedicated people. From now on, new services ought not to require such effort. Data formats, and APIs, should pave the way for smooth exits. Otherwise, users will rightly be reluctant to sign up.

The RSS fiasco

The wave of extinction also affected those of us who use the Rich Site Summary (RSS) format to promote websites. I first wrote about this in a June, 1999 column on Netscape and Microsoft channels. Ever since then, I've used RSS to promote my own website and others I manage, including Linux Magazine.

Dave Winer, at UserLand Software, has recounted the history of RSS. Briefly, it started as a way to syndicate headlines from Scripting News. Netscape later proposed the format that became RSS, and implemented an RSS aggregator and viewer at my.netscape.com. There have always been other aggregators and viewers, including My UserLand, Moreover, and more recently, Meerkat and News Is Free. And it's a lucky thing because, last week, Netscape pulled the plug on My Netscape's RSS channels. Jake Ochs was the first newsgroup correspondent to be rudely surprised by this:

Jake Ochs:

It looks like Netscape killed the custom channels (RSS) feature. Bummer. They really broke a golden rule on this one. No notification that the service would be terminated, no backward compatibility with the new My Netscape. I am not pleased.

Shortly thereafter, Meerkat creator Rael Dornfest weighed in on the matter, noting that Netscape had sinned even more egregiously than was first apparent. It wasn't just that My Netscape users, like Jake, couldn't use their custom channels any more on the My Netscape page. Far worse, and almost unbelievably, Netscape had withdrawn the RSS 0.91 DTD from the web!

Like a lot of RSS users, I had for several years been reflexively starting every RSS files like this: 

<?xml version="1.0"?>

<!DOCTYPE rss PUBLIC "-//Netscape Communications//DTD RSS 0.91//EN"
      "http://my.netscape.com/publish/formats/rss-0.91.dtd"
>

When it yanked that DTD file, Netscape threw a wrench into the whole RSS mechanism. As Rael pointed out, the "staccato availability" of the DTD had provoked some RSS aggregators to ignore it, and use non-validating XML parsers. But many continued to use validating parsers, and these aggregators broke when the DTD went away.

What you can do immediately -- and what I have done -- is remove that DOCTYPE declaration from your RSS files, as Rael recommends.

For the long haul, I'll be switching to RSS 1.0. (The address behind that link -- http://purl.org/rss/1.0/ -- is of course another example of a web service that would benefit from interoperability features and what John Faughnan aptly calls an "exit strategy.") RSS 1.0 uses XML namespaces to allow modular extensions along various dimensions of metadata. But in its simplest form it is similar to, and compatible with, RSS 0.91. Importantly, RSS 1.0 does not depend on a single centrally-located DTD. As Rael's article notes, that's a recipe for disaster.

To add insult to injury, by the way, Netscape has yet to decommission its RSS aggregator. Somewhere in the bowels of My Netscape there lives the memory of my own site's RSS channel. And so I am now receiving machine-generated messages from a process that is failing to parse my RSS file: 

Dear JudellJ, This is to remind you that we have been experiencing
problems retrieving your Rich Site Summary File. Please refer to the
earlier email we sent you for detailed information about the error 
and ways to resolve it. 

If you would like help in correcting the error, please visit 
http://my.netscape.com/publish/help/mnn20/troubleshooting.html 

If you have been unable to resolve your error, please feel free to 
reply to this email or email us at mnn_support@netscape.com.

You do not need to contact us when you resolve the problem; our systems 
will automatically check for a corrected file. Thanks for your help and
continued participation in this program.

The My Netscape Team

I sympathize with John Munsch, who posted an impassioned statement to the Yahoo Groups syndication list saying, in part:

"Hey, after all, we're Netscape, if we're not going to use RSS any more, then nobody else will either, right?"

Sigh. I'd appreciate it, anyway, if somebody at Netscape could at least track down that poor aggregator and put it out of its misery.

Lessons learned

As John Faughnan points out, some products -- notably Groove -- anticipated the first extinction. When I interviewed Ray Ozzie last year, he was explicit on this point. Groove can use a centralized relay service, but it doesn't have to.

Ray Ozzie:

It wasn't merely for our own amusement that we decentralized Groove. I had just come off the Notes experience where, in sales situations, we'd get hit with the argument: "We can never get into a situation where you're a point of failure for our organization." Now I'm looking at this trend toward ASPs, putting services onto websites. If you're running your company on a piece of critical software, the ASP can specify its contractual obligations, but ultimately you're constrained as to how you use the service. If you're building something that you want to just get out there and work for people, you want to design it so you're not going to court because the failure of your site caused businesses to fail, or people to lose lives. Your system needs to be failsafe in its ability to do things in a distributed fashion. So while people will depend on us for working software, nothing about the Groove software relies on Groove Networks, Inc. We can go out of business, and Groove will still be usable, to the extent that the software people have continues to run.

Have we come full circle, then? Yes, but it can't stop there. As a software platform, the web's greatest asset remains its zero footprint. That's not going away, nor should it. The next generation of purely web-based services, though, will have to be portable, offer redundant and failsafe data and services, and guarantee graceful exits.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

Creative Commons License
This work is licensed under a Creative Commons License.