Through the lens of the podcast transcription service CastingWords, I've observed Amazon's MTurk from a few different perspectives. In a podcast conversation with CastingWords' Nathan McFarland and Mycroft's Ben Hill, we explored network-enabled ways of harnessing collective intelligence. That podcast was then transcribed by CastingWords, and I reviewed the results. More recently I became a satisfied CastingWords customer.
Last night, to complete the picture, I joined the global workforce of Turkers -- that is, MTurk-enabled workers. It's a weird subculture that Katharine Mieszkowski explores in her excellent Salon article I make $1.45 a week and I love it.
One of my talks last week focused on the idea that blogging can open windows into the world of work. That's certainly true of Turk Work. Blogs such as Turk Lurker, Mechanical Turk Monitor, and CastingWords Turker chronicle the experiences of Turkers as they churn through HITs (Human Intelligence Tasks), rack up pennies, exchange tips for finding and optimizing work, and reflect on the often bizarre nature of tasks that, at least for now, do not yield directly to automation.
Here are some of the HITs available to me at the moment:
The description of the third HIT reads:
To make one's mother to stand on one foot (with the other one lifted) and to picture this outstanding scene. Name of the mother, her age and the location are needed. (TIME!!!) The fee - $2 per chosen picture.People in online sweatshops who are paid to to play games in which they create virtual artifacts, earn virtual currencies, and build virtual reputations have fallen down one kind of rabbit hole. Many Turkers have fallen down another, as shown most strikingly by the Aaron Koblin Sheep Market project described in Mieszkowski's Salon article.
CastingWords, of course, is a very real and very effective business. So I did a bit of transcription in order to see what it's like from the Turker perspective.
Podcasts are chopped up into six-minute segments. Last night, the segment I transcribed was worth $1.02, which will be credited to my Amazon.com account (and can then be transferred to a bank account) if my work passes several quality review checks. These checks are themselves implemented as HITs.
This morning, segments from that same batch are worth $1.28. (I should have waited!) At that rate, a Turker who could transcribe accurately and in realtime would earn $12.80/hour for himself and $1.28 for Amazon. I can't do accurate realtime transcription, of course, so I'd be lucky to approach minumum wage.
How'd I actually do? To be honest, I lost track. The segment I chose at random turned out to be an interview with Mauria Aspell, a childhood friend of Bill Clinton who dated him in high school and college, and who was evidently being interviewed for a history of Clinton's relationships with women. Thirty seconds into the recording I guessed what it was about, but when the name finally dropped it was still a shock. Bill Clinton's romantic past, in 6-minute segments, parceled out to the Turk Nation for transcription at $1.28 a pop. Whoa. Where's my blue pill?
Celebrity dirt notwithstanding, my encounter with Turk Work wasn't very satisfying. There was no context, no orderly progression, no sense of collaboration, no awareness of (or pride in) a finished product. The Salon article suggests that these are essential qualities of the experience. And as I look more closely at how MTurk is structured, I suspect that's right.
If you're running an MTurk-enabled business, you have to focus on throughput and efficiency. That drives you toward assembly-line tactics. But as Nathan McFarland noted in our conversation, you'd also like to reward excellence:
We'd be willing to pay certain workers more and other workers less, but that's not an option right now.So for now, that's a hard constraint. Here's another. From my perspective as a CastingWords customer, there are clearly some transcribers who have a better feel for my material than others. I don't know who they are, though, or who's transcribed which parts of various podcasts. If the goal is maximum throughput, that may be necessary. But that's not my goal. Transcribing 620 minutes of audio for $260 in six days was darned impressive, but two months later I'm still only halfway through the final polishing, which I'm tackling in fits and starts. If this winds up being a long-term relationship, I'd rather identify the transcribers who do well on my stuff and give them whole tasks. And some transcribers would likely prefer that too. I sure would. Apart from the question of whether it's in CastingWords' interest to allow such relationships to form, the current MTurk architecture precludes the possibility.
There are, of course, other architectures. Last week in Ann Arbor, for example, I met Bill Tozier, who among other things works for Distributed Proofreaders, an adjunct to Project Gutenberg. "We're the prior art for MTurk," Bill told me.
MTurk is one model for the online coordination of work. There will be many others, some commercial and some open, each constituted in its own way.
One final thought on MTurk. When the tsunami hit, and then Katrina, I used Amazon in both cases to contribute to disaster relief. It wasn't very satisfying, though. In retrospect I'd rather have contributed work to the Katrina People Finder. Next time around, it'd be cool if Amazon used MTurk to coordinate that kind of work, and really cool if it found ways to create social capital in the process.
Former URL: http://weblog.infoworld.com/udell/2006/09/20.html#a1527