Do Online Ratings Affect Your Judgment?
August 9, 2013

Herd Effect Slants Online Ratings

redOrbit Staff & Wire Reports - Your Universe Online

When a person “likes” an online comment, article or product, others become much more likely to follow suit, according to a new study from researchers at the Massachusetts Institute of Technology's Sloan School of Management.

The research suggests that high online positive ratings may not be a true indicator of an item's value or quality, but are instead the result of a "herd effect” that causes people to “like” something that is already well liked by others.

The MIT study involved a five-month experiment conducted on a major news-aggregation web site. Researchers randomly manipulated the favorability ratings given to certain comments on the site, and then observed how readers evaluated those same comments when they were given different ratings.

This approach was necessary because in most circumstances, "it's hard to distinguish the effect of high quality from the effect of social influence bias,” said Sinan Aral, an associate professor at the MIT Sloan School of Management, and one of three authors of the study.

“It could be that past positive ratings have snowballed to create a high score, or it could just be that those items likely to get high scores are just of high quality."

The results of the experiment revealed that comments whose ratings were altered in a favorable direction saw their popularity swell, receiving a 25 percent higher average rating from other site users.

Comments manipulated to have positive ratings were also 32 percent more likely than unaltered comments to receive a favorable rating from the very next viewer, and 30 percent more likely than untreated comments to obtain a very high favorable rating.

"This herding behavior happens systematically on positive signals of quality and ratings," said Aral.

Interestingly, the negative ratings did not appear to exhibit the same effect, in part because some people tended to "correct" negative opinions with positive ratings.

Aral said that while comments given negative ratings did attract more negative judgments, that increase was cancelled out by a "correction effect" of additional positive responses.

"People are more skeptical of negative social influence," Aral said. "They're more likely to 'correct' a negative vote and give it a positive vote."

While this phenomenon of social positivity sounds nice enough on the surface, Aral warns that there are downsides to it as well, such as the manipulation of online ratings by marketers, political operatives or anyone who stands to profit by creating an inflated appearance of popularity.

"These positive ratings also represent bias and inflation," Aral explained.

"The housing bubble was a spread of positivity, but when it burst, some people lost their savings and their houses went underwater. Stock bubbles represent a positive herding, and they can be dramatically bad in the wrong context."

The study also revealed topical limitations in herding. For instance, stories involving politics, culture, society and business generated positive herding, while stories posted under the topics of economics, IT, fun and general news did not.

Aral suggests that people be as analytical as possible when it comes to harnessing collective judgments.

"We have to be careful about the design and analysis of systems that try to aggregate the wisdom of crowds.”

Aral acknowledged that the experiment "opens up as many questions as it answers,” and said that further research is needed into the psychology of the correction effect on the negative side as a way of understanding how collective judgments are formed.

"Our message is not that we should do away with crowd-based opinion aggregation," he said.

"Our point is that you need solid science under the hood trying to understand exactly how these mechanisms work in a broad population, what that means for the diffusion of opinion, and how can we design the systems to be fair, to have less incentives for manipulation and fraud, and be safe in aggregating opinions."

The study is published Thursday in the journal Science.