People often search the Web for reviews of products they might buy and merchants from whom they might purchase goods and services. It’s easy to lose track of time reading reviews on sites like Amazon, where people seem to enjoy sharing their opinions about almost anything. It’s not so easy to find online reviews of merchants surrounding me in a somewhat rural community.
Reviews are interesting when it comes to how they might be treated by search engines, how they could possible impact local search rankings, how a search engine might identify review spam, and the potential impact of those online reviews upon word of mouth and the reputations of businesses and the sales of goods and services.
For instance, Google Rich Snippets allow snippets to show the number of stars a place might have received in reviews from a particular resouce such as Yelp:
Google Place results that appear in a Google Web search may also show an average rating and a number of reviews for a number of businesses:
Do those ratings and reviews influence in some way whether or not particular businesses show up in Google Places search results? Do they influence searchers choices in where to visit, after seeing the ratings and the numbers of reviews? How much influence might reviews have, and how important is it for search engines to handle those reviews intelligently?
I ran across a Google research paper which collected and studied information about product reviews, merchant reviews, and Netflix ratings for movies.
The product reviews data contained over 8 million ratings of 560,000 products, gathered from 230 sources, and reviewed by 3.8 million authors. The merchant reviews data included 1.5 million ratings for 17,000 merchants collected from 19 sources and written by 1.1 million authors. The Netflix reviews of 17,700 movies consisted of 100 million user ratings submitted by 480,189 authors.
The paper, presented at the Fourth International AAAI Conference on Weblogs and Social Media, is titled Star Quality: Aggregating Reviews to Rank Products and Merchants, and it details a joint Google/Carnegie Mellon University study by Mary McGlohon, Natalie Glance, and Zach Reiter that asks and attempts to answer a number of questions about online reviews of products and merchants such as:
- Given a set of reviews of products or merchants from a wide range of authors and several reviews websites, how can we measure the true quality of the product or merchant?
- How do we remove the bias of individual authors or sources?
- How do we compare reviews obtained from different websites, where ratings may be on different scales (1-5 stars, A/B/C, etc.)?
- How do we filter out unreliable reviews to use only the ones with “star quality”
There are some interesting observations found by the authors, such as when a reviewer has written only a single review, that author is disproportionately more likely to have given a 5 out of 5.
In looking at reviews, the authors looked at whether the reviews were submitted anonymously, how prolific reviewers might be, and if the reviews could be helpful in determining the quality of the goods or services being reviewed.
As I mentioned above, I have been surprised by how few reviews there are for merchants in my community, but I expect the numbers to grow in the future. What kind of influence might those have upon business in the area?
To a degree, that’s going to depend upon how search engines like Google might make that review information available to anyone who tries to find it. The authors of this paper tell us that this is a subject that hasn’t had much research behind it at this point:
This is the first work, to our knowledge, over aggregated reviews from different sources. We observe that there are often biases of different sources and authors– different authors and review communities will often have very different behavior. We compare reviews coming from these different review sites and investigate how this may help deduce the true quality of an object rated.
Should a review from Yelp be given as much weight by Google as one from a local newspaper? Or from another review site? How effective might the search engines be in identifying spam reviews?
Are there more authoritative sources of data that the search engines might tap into, to find helpful ratings – for instance, the paper mentions the possibility of looking at Better Business Bureau ratings to get some “insight into which merchants are most reliable?”