July 25, 2012
When Crowd Wisdom Works and Doesn’t Work.

Sometimes it’s easy to oversimplify an hot, complex idea. Case in point: crowd wisdom. David Leonhardt,in his article for the New York Times, points out exactly why and when crowd wisdom isn’t so wise.

The article provides an excellent background overview on the rise of faith in crowd wisdom. But the most important part is where he highlights the successes and failures of it. On the successes:

“The early successes of prediction markets were notable. To take a small personal example, my wife and I, not exactly frequent moviegoers, twice won money in a large Oscars pool simply by hewing to the British odds. Much more significantly, Intrade [the online prediction market where people can bet on real-world events] was a more reliable guide to the 2006 midterm election than cable networks. On election night, its odds showed that the Democrats had become the favorites to retake the Senate, while television commentators were still telling viewers it was unlikely.”

And on its failures:

“In the days leading up to the Supreme Court’s health care decision, rumors began to circulate in Washington that the justices had decided to uphold the law….

With the rumors swirling, I began to check the odds at Intrade several times a day. The odds had barely budged. They continued to show about a 75 percent chance that the law’s so-called mandate would be ruled unconstitutional, right up until the morning it was ruled constitutional. 

The market — the wisdom of crowds — turned out to be wrong.”

As, David rightly points out, crowd wisdom isn’t applicable in all situations. 

“If the circle of people who possess information is small enough — as with the selection of a vice president or pope or, arguably, a decision by the Supreme Court — the crowds may not have much wisdom to impart. ‘There is a class of markets that I think are basically pointless,’ says Justin Wolfers, an economist whose research on prediction markets, much of it with Eric Zitzewitz of Dartmouth, has made him mostly a fan of them. ‘There is no widely available public information.’”

That’s an critical point for anyone to remember when considering crowd wisdom as a model:

When everyone has access to the same information and is trying to make sense of it, crowds can be useful. But when only a few insiders have the information - as in a unannounced Supreme Court ruling, for example - crowds are useless. 

It’s not, as Dave Copeland points out, the “more is more” strategy that makes crowd wisdom work. Getting anyone and everyone into the a respondent pool doesn’t make it smarter. The pool must be curated with people with knowledge relevant to the question at hand.
It all goes back to one of the principle of designing businesses for the networked: The company with the smartest consumer community wins