Tangled in the Threads

Jon Udell, June 20, 2001

Collaborating about collaboration

Local SMTP proxying looks like a promising technique

How do you inject metadata capability into email without changing the client? A different twist on the old pipeline idea might do the trick.

Although my book on groupware has now gone out of print, its themes keep coming up again and again. Newsgroup discussion this week circled back to tools and techniques for collaboration.

Dan Diephouse:

We are a non-profit organization that does lots of collaboration with outside people and companies. We want to be able to share files, ideas, schedules, and anything else you can think of with our partners.

Given the following criteria...how would you guys collaborate?

  • It must be easy for the end-user. Preferably no downloading of special clients. Java applets and programs that launch out of the browser are OK also. But, probably should be avoided. That also means cross-platform compatibility. Unix/Windows/Mac. Which limits us to the browser. BUT, I am open to other suggestions.

  • We must be able to share files. Preferably each file should have a description stored with it and other appropriate metadata.

  • Scheduling within the organization, and with people outside the organization.

  • Messaging.

  • Security. We have about 100 jobs/year that we work we work on with other people. Preferably they should only see the files for their own projects, the messages for their projects, etc.

I have looked at a few options already:

  • Newsgroups. Creates a good forum to message and share files. Although it makes it a bit of a hassle to retrieve files I would think. Then we could write an intranet application that did scheduling. Also interfaces could be written for the intranet that posted, to the newsgroup, scheduling information.

  • Develop an intranet extension using WebDAV. Nice, but the technology is so new. And, how do we deal with metadata? The DAV interface supports metadata, but the clients don't.

  • Use somebody else's solution, such as groove. There are other products out there. I'd be interested to hear people's opinions on them.

In the past, I've tended to agree that avoiding a new client is a high priority. That's why I wrote a book that shows what you can accomplish using common mailreaders and newsreaders, in tandem with the universally-deployed browser. I got great results using the techniques outlined in my book, but stitching these parts together in useful configurations is a skill that doesn't come easily or naturally to most people. So my conclusion at the moment is that this no-new-client requirement probably needs to be relaxed. There going to have to be some new client software that helps us integrate document sharing and messaging. It should be no heavier than a current browser download/install (preferably much lighter), it should be able to run everywhere, and it should be everywhere.

There are always plenty of opinions on both sides of this question, of course.

Joshua Boyd:

What is with the obsession of not downloading and installing programs? Just insist that whatever program you have to download and install doesn't require anything more than JUST downloading it, and creating a shortcut on the desktop. Everything else should be statically linked. Or, a zip (or tarball or binhex) that can just be unzipped to a directory, with no other changes.

The reason I'm sick of the "no installation" cry is because limiting yourself to web only, or web and java applet only, really cripples your capabilities. All of a sudden, we are limiting our fancy new P4 machines to less than we had with Vt-100 terminals.

Peter Thoeny:

I am avoiding client side programs and favor web based apps wherever possible. Reasons: 

   * Installation hassles (as mentioned)
     ==> Help desk cost

   * Configuration hassles
     ==> Help desk cost

   * Platform questions: You need to create and maintain
     client programs for all major platforms like 
     Windows, Mac, Solaris, HP-UX, etc. (unless you want 
     to limit your users).
     ==> Development and QA cost

   * Upgrade issues: Need to update client software for
     each release. This can be a major issue (cost & 
     downtime) if you have, say, 2000 client machines 
     distributed over many locations. At work we are on
     the way of upgrading a plain old defect tracking 
     system, where all clients *must* be upgraded before 
     you can use the system on the server...
     ==> Help desk cost, productivity cost

We all know that the UI is limited, but many times it is possible to create most of the functionality you need with plain old HTML (served dynamically) and JavaScript.

Rich Kilmer:

What about the help desk costs of using the browser? I find it amusing that we have these thin client (read: 80 meg w/plug-ins) browsers on most desktops. Developing web applications that work across browsers or platforms (even Mac and Windows) with any decent level of sophistication is not an easy proposition either. Think of how many web sites (intranet or internet) demand IE 4 or above for Windows. I know there are great small browsers like Opera, but not many (US) corporations use them.

It's a frustrating situation, to be sure. The web model, for all its limitations, has one overriding advantage: universality. It's true that the essence of the platform -- HTML and JavaScript -- is harder to discern amidst a welter of incompatible extensions. But the core still exists, and it continues to enable a very large and important class of applications, which gain wide adoption in an almost frictionless way. So, even as we plan for, and develop, a next-generation client technology, we have to use the current stuff for all it's worth.

The annotated repository

Collaborative tools have always been, of necessity, hybrids that exploit web/email and client/server synergies. I thought I'd explored all the ways of combining these parts but, this week, the discussion led to an idea that seems compelling to me: local mail proxies.

Recall Dan's requirement that each file in a shared repository "should have a description stored with it, along with other appropriate metadata." Of course, we already do that when we wrap email messages around attachments. For many people, email inboxes are, in effect, the file system. They don't look for things by subdirectory, but by email folder -- and within folders, by attributes that the file system doesn't know about, like "Subject:" and "From:".

The instinct to do this runs deep, and we need to find ways to work with it rather than against it. Better capture and use of metadata seems, to many people, a vital next step:

Bill Nicholls:

Part of the problem is in the email client. What is needed is an email client that requires at least the minimal DCMI (Dublin Core Metadata Initative) amount of identification, with additional tags as appropriate.

chromatic:

If e-mail clients are heading towards integration with address books and calendars and tasklists, why not push things slightly further and make them aware of projects?

For example, if I send an e-mail to Brian or Laura, it's probably about the book, and ought to go in the appropriate folder. If I send something to jlp, it's probably about Jellybean, and ought to go there.

Picture another screen in your e-mail or newsgroup client that shows five to ten of the Dublin Core items, Trove categorizations, or some other metadata classification system. Even simple rules (if it's to Laura alone, it's probably an editorial question, if it's to Brian and/or Chris, it's probably a technical question) can help the classifier fill in some of the blanks itself. A human would have to be able to change defaults or add new schemes, but the software could pick up new rules from there.

chromatic alludes to another kind of hybridization: human/machine. Metadata that matters is metadata that's been refined through the prism of human intelligence. Machines can't do this at all; humans can't do it efficiently; human/machine cooperation is what's needed.

Mark Wilcox:

Part of your process -- the identification of critical documents -- requires a human editor. If an application can automatically mark a document critical, then we're talking reasoning, then you've got discussion for after-hours drinks & AI conferences, but not something likely to happen for a long, long time, if at all.

What the machine is good at, and would be better at if given some more help, is pattern matching.

Local SMTP proxies

At this point, Rich Kilmer jumped in with a simple, powerful, and innovative suggestion: why not use email proxies to inject software assistance into email communication?

Rich Kilmer:

Here are some ideas:

What if there were an in-machine SMTP proxy that passed all sent mail through it and "asked" questions about the mail being sent. If it knew, for example that "udell@monad.net" worked with you on Project X and you sent an email to him, it may ask if the email was about the project and tag the message accordingly.

or...

What if people, groups and projects were represented on the screen and we could drag objects on them. For example, if I create a meeting in my calendar and then drag that meeting on a group, it asks if I would like to "invite the group" to the meeting...if so, it would form an email message accordingly and attach appropriate metadata. If I drag a file to a person, it may pop up a list of "tasks" I may ask that person to perform with the file, such as "please review" or "the file you requested" and attach metadata to allow the receiving system to automatically categorize them.

the point is...

We need to find ways that the metadata is allowed to be attached to the messages as seamlessly as possible. Mark is right in indicating that machines are not good at reasoning about content, people are, thus we must empower people with the best approaches we can think of to impart their reasoning into their machines.

I just love this local proxy idea! I've written before about the power of local HTTP proxies, notably Proxomitron. This kind of technology has all sorts of surprising and wonderful uses. But maybe we shouldn't be surprised. It's just the same old, but endlessly powerful, idea of pipelined components. And it could be a great way to inject metadata into the email channel.

To test the notion, I looked around for some Perl code that would implement the necessary framework: a simple SMTP server, to point your mail client at, and a simple client, to relay traffic to your real SMTP server. Nothing on CPAN seemed just right, but a Google search turned up Bennett Todd's smtpprox, which he describes in a readme file thusly: 

smtpprox is a trivial transparent SMTP proxy, an SMTP server and
client combination. It uses its own SMTP server and client modules
which are designed to expose every step of the protocol dialogue to
the calling program, which provides for the greatest flexibility in
hooking in envelope and content controls and scanning.

It works as you'd expect. You issue a command like: 

smtpprox localhost:25 smtp.yourisp.com:25

Then, you redirect your email client from smtp.yourisp.com to localhost. At this point, easily-modified Perl code has its hands on your outbound message traffic. smtpprox is written Unix-style, and relies on fork(), which used to mean problems if you wanted to run on Windows. And that, of course, is where most email clients do run, and where most local SMTP proxies would want to be. Fortunately, ActiveState Perl now provides fork() emulation, and with that I was able to use smtpprox on an NT 4 machine with no trouble.

In principle, it's straightforward to use this framework to implement some of the ideas discussed here. In practice, there's a lot of hard work to do. So far, I've just scratched the surface, by experimentally appending slugs of XML to outbound messages. But message rewriting can get hairy if you need to adjust the structure of MIME attachments, and really hairy if one of those attachments is an S/MIME signature (as is always true for me).

In any case, these are just mechanical problems. The real issue is that, as several people noted, not much value can be added by a purely mechanical process. You'd want to involve the user in approving, or adjusting, proposed metadata. That means UI, and we're back to the same dilemma. Push the UI into a browser? Deliver a cross-platform GUI? I suspect the latter is what's needed. In which case, this approach nicely decomposes the problem. We can imagine an email-aware metadata assistant as a relatively simple helper application, independent of the email client, and not difficult to port.

I'm sure variations of this fascinating idea have been tried. I'm tempted to try too, but before plunging ahead I'd love to hear from anyone who's already been there. If you want to join the discussion, please use the BYTE.com NNTP server because the web view of the discussions is undergoing an overhaul.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

Creative Commons License
This work is licensed under a Creative Commons License.