PageRank for Answers?

A few of the comments I’ve gotten about my reputation system are telling me that I’m on the right track, but I’m really not doing enough to measure answer quality, not just quantity.  Recently, the community lead for the C# team told me that she had been finding some of their newer moderators (who are excellent) by using the “Most Helpful Rated” column–not the “Top Answerers” column that I evangelize so much.

It’s really too easy to just pick off low-hanging fruit and marking simple answers as answers.  If somebody posts something that’s truly great in the forums, shouldn’t they get more credit than a random Microsoftie that closes a thread with a link to the newsgroups and marks that as an answer?

Of course, the logical answer is “Of course”, but it’s just not that simple.  It adds a layer of moderation on top of nearly everything in a system that’s already fairly taxing for moderators (for the answer marking to work well, you really need moderators that care about answer marking.)  So, how do you judge answer quality without asking people to do extra moderation?

One idea I’ve toyed around with was using a combination of page view counts and “trackbacks” on a given post.  The former we already measure–every time somebody looks at a post on the forums it increments a little counter.  Sure enough, the best, most-helpful posts have the highest view counts by far.  The latter–“trackbacks” to posts, might not be entirely feasible with our current forum implementation, but the idea is simple (and the key to Google’s billions)–the more people that link to your post, the more likely it is that your post was useful and of high-quality.

This is some fairly half-baked brainstorming, and I’m very aware of some of the holes in the system–it’s easily gamed, there’s no real assurance that a given post isn’t just being read quite a bit because it appears helpful until you read it, and so on, but I’d really love to brainstorm with everyone…what do you think?

How can you gage answer quality without asking a moderator to do it?

Comments (4)

  1. BlakeHandler says:

    SPAM – Unwanted Email

    SPIM – Unwanted Instant Messages

    SPLOG? – The problem I’ve found is that people post comments to blogs for the ONLY purpose of having "links" to THEIR sites. Many bloggers are forced to turned off public comments.

    PORKBACKS? – I personally had SO many spam trackbacks from websites that simply had NOTHING to do with my postings, that I no longer allow public trackbacks.

  2. Fredrik says:

    >> How can you gage answer quality without asking a moderator to do it?

    Ask the users to! After all, who better to gauge whether an answer was helpful than the ones who try to make sense out of it? – for example a simple "was this answer helpfull to you?" vote with a thumbs up/down (akin to amazon). If you have some sort of system also keeping track of a users standing within the community (ebay) you can also then weigh each vote on this, so that more respected users have a higher influence on an answers rating etc. You’ve got a huge number of active and very talented members on these forums – make sure to take advantage on it and not limit yourself to the chosen few who are moderators 🙂

  3. Will Parker says:

    >> >> How can you gage answer quality without asking a moderator

    >> >> to do it?

    >> Ask the users to!

    That’s only part of the equation, because it doesn’t offer any inherent measure of the subject’s breadth or depth in either technical expertise or — how do I say this politely? — people skills.

    In a prior (career) lifetime, I spent twelve years as a tech support engineer. I’ve had every kind of tech support conversation there is — not in content, you understand, but in form. I’ve also committed — or overheard — almost every sin that can be committed by people in an allegedly helping role.

    Believe me, neither a ‘most correct answers’ nor a ‘highest rated’ count is going to be enough to tell you which individual is truly the best asset for your forum — and I say this as someone who rated pretty high on both marks.

    I think what you’re missing is some method of determining which people truly love the subject matter so much that they will happily teach the subject to all comers, and make the extra effort to insure that their ‘students’ truly understand the concept under discussion.

    Work from this principle: Not all friendly loudmouths are good teachers, but all good teachers are friendly loudmouths.

    First, go spend some quality time with your developer tech support people. Watch what they do and listen to how they do it. If possible, sit in on some customer calls. Repeat that process with some of the front line consumer tech support people. Interview front line and lead support people (but ignore their managers).

    Second, go ahead with your plans to collect quantitative data via the methods already discussed here. That’s enough to get your initial pool of candidates. Be sure to capture the subject areas each candidate has covered.

    Finally, do a search for any members of the candidate set who are frequent commenters on the same subject(s) elsewhere on the web, either via forum comments or via their own blogs.

    The people you want are the ones who just won’t shut up.

  4. I recently had the opportunity to attend the Online Community Camp run by Forum One Communications in…