Clay Shirky recently published a post on Corante, entitled "Against Well-designed Reputation Systems (An Argument for Community Patent)" that I've been meaning to respond to, for awhile. His thesis, which I agree with wholeheartedly if I understand it correctly, can be summed up as 'Don't do it. Don't even think about doing it.'
Clay, rather than respond to your post with a post, I will instead respond with a working implementation of an alternative approach to social evaluation (same ends...different means). Due out in alpha in the next few weeks, I believe that Claimspace demonstrates my agreement with your observations and reservations about the risks inherent in developing a traditional ranking and reputation system. Claimspace speaks louder than words. It may even prove useful as a social evaluation platform for community patents.
A favorite passage from Clay's post:
"The obvious conclusion to draw is that, when contemplating the a new service with these characteristics, the need for some user-harnessed reputation or ranking system can be regarded as a foregone conclusion, and that these systems should be carefully planned so that tragedy of the commons problems can be avoided from launch. I believe that this conclusion is wrong, and that where it is acted on, its effects are likely to be at least harmful, if not fatal, to the service adopting them.
There is an alternate reading of the Slashdot and eBay stories, one that I believe better describes those successes, and better places Community Patent to take advantage of similar processes. That reading concentrates not on outcome but process; the history of Slashdot’s reputation system should teach us not “End as they began — build your reputation system in advance” but rather “Begin as they began — ship with a simple set of features, watch and learn, and implement reputation and ranking only after you understand the problems you are taking on.” In this telling, constituting users’ relations as a set of bargains developed incrementally and post hoc is more predictive of eventual success than simply adopting any residue from previous successes.
As David Weinberger noted in his talk The Unspoken of Groups, clarity is violence in social settings. You don’t get 1789 without living through 1788; successful constitutions, which necessarily create clarity, are typically ratified only after a group has come to a degree of informal cohesion, and is thus able to absorb some of the violence of clarity, in order to get its benefits. The desire to participate in a system that constrains freedom of action in support of group goals typically requires that the participants have at least seen, and possibly lived through, the difficulties of unfettered systems, while at the same time building up their sense of membership or shared goals in the group as a whole. Otherwise, adoption of a system whose goal is precisely to constrain its participants can seem too onerous to be worthwhile. (Again, contrast the US Constitution with the Articles of Confederation.)"
Dr. Credlove, How I Learned to Stop Worrying and Love the Reputation System
Last year, after we launched CodePlex, I sat down with my new manager, Bob Rebholz to discuss what my next big challenge at Microsoft might be. Bob was busy planning the overall strategy for project "Athens", which would come to be known as Microsoft.Community, internally. To his credit, Bob gave me a lot of freedom to chart my own course. But one thing he insisted upon was that I assume product management/planning for a project he had in mind to 'do something about reputation and recognition.' Alas! Reputation! What better than the frickin' Holy Grail of Social Computing to keep me stimulated and gainfully employed for years to come? I wanted it but I hid my eagerness behind a veil of resistance. People DIE when they touch the Holy Grail, right? That or they go insane trying to find it. Why in the world would I accept the charter to build a ship that might never sail, of its own power, or drive me crazy, in the process? Did I really want to perish like all those crazy Nazi dudes in Indiana Jones. NO!
For nearly two weeks, I tried to convince Bob to unload "Reputation" on one of my two peers, Dave or Brian. But Bob was resolute and frankly, I'm drawn to challenging v1 products like a moth to flame.
To take a bit of the sting out of ugh, "reputation", I soon prevailed upon my new teammates to adopt the kindler and gentler approach and code name: "Rapport", for our fledgling project. I then focused my attention on defining and delivering on two objectives:
- Enable technical professionals to gain recognition, on their terms, wherever they go on the Web.
- Enable people to rapidly ascertain and socially evaluate the probable credibility of any online resource whose author has included it in Claimspace: a placeless market of opinion.
"Hm?" you ask, "Isn't credibility sorta related to reputation?" Well, yes. Yes it is. But it is not the same thing. Wikipedia's entries on reputation and credibility are illuminating and um, confusing. I have toiled, perhaps overmuch, to convince my team and indeed, anyone who would listen that building a traditional, point-based "reputation system" (like Slashdot's karmic system) is a dangerous and costly proposition. I even developed a handful of little slogans to drive my anti-Reputation crusade, such as "REPUTATION IS THE ONLY FOUR-LETTER WORD IN THE ENGLISH LANGUAGE THAT HAS MORE THAN FOUR LETTERS."
The central points in my anti-reputation argument were threefold.
- If you build a traditional reputation system, users will "game" it, meaning that success will come at a steep and continuing cost, both in terms of dollars and user acceptance.
- It is dangerous for a company to assign a value to its customers. Doing so is bound to alienate more customers than it delights.
- Reputation is a personal and subjective calculation of a person or group's probable behavior or reliability or credibility, in a specific context or situation. The measure of a person or group's reputation cannot be reliably expressed, in a simple and universally meaningful way because the context and personal objectives of the subjects and their evaluators are ever-changing. For example, consider this well-known photo and fill in the blank.
In the context of the Yalta Conference, Joseph Stalin's reputation was one of _____. On a scale of 1-10, Stalin had a reputation of _____? On what basis? To whom?
Interestingly, it is possible that Claimspace--or something similar--might someday be used as the basis for an entire generation of reputation systems. Funny how things work out, isn't it, Bob?