When can I trust an average rating on Amazon?

Question 1

Review System Types

Answer 1

• rating
  
  • 1-5 number system
• review
  
  • text
• reviews of review
  
  • thumbs up or down
  • useful or not

Question 2

Review System Challenges

Answer 2

• gate keepter
  
  • anybody or verified purchase?
  • anonymous or id based?
  • if anonymous, how to avoid random, competition, or personal reviews?
• scale
  
  • 1-10?
  • 1-3?
  • -5-5? = psychologically really bad
• number of reviews
  
  • how big is big enough?
• what’s the performance metric?
  
  • subjective or objective?
  • movies vs electronics

Question 3

Galton’s Experiment

Answer 3

• 1906
• A farm in Plymouth, UK
• 787 guesses of an ox’s weight
• average = 1197 pounds
• actual = 1198 pounds
• a wisdom of crowds?

Question 4

Which of the following statements is FALSE?

• Galton’s famous 1906 experiment shows potential for “a wisdom of crowds”.
• Amazon’s review system allows users to rate products on a scale of 1 – 5 and enter reviews, but doesn’t allow review of reviews.
• Two major challenges in designing a review system are determining who is allowed to review, and what rating scale to use.
• In addition to a product’s average review, a sophisticated recommendation system should take into account the number of reviews.

Answer 4

• B

Amazon allows people to give a thumbs up or down in whether they found a review helpful or not.

Question 5

Wisdom of Crowds Key Factors

Answer 5

• definition of the task
  
  • objective
• unbiased and independent estimates
  
  • not influenced by other people’s guesses
• enough people participating

Question 6

Amazon Review General Knowledge

Answer 6

• natural language processing
  
  • filter out biased reviews based on word usage
• statistics
  
  • people who hate it or love it are more likely to review
  • bimodel distribution
• signal processing
  
  • timing of reviews
• voting
  
  • the idea of people ranking all products
  • impractical for Amazon

Question 7

According to statistics, which of the following is the most likely distribution of the ratings that would be entered for a product on Amazon?

• Most reviews are either very low (~1) or very high (~5).
• The number of reviews is the highest around 3, and taper off on both sides.
• Most reviews will be very high (~5).
• The number of ratings at each of 1 – 5 will be roughly the same.

Answer 7

• A

According to statistics, products tend to have a bimodal distribution with centers around the most negative and most positive reviews, since people extremely satisfied or dissatisfied are more likely to care enough to submit ratings. This is even more probable when some fabrications work and others don’t.

Question 8

The Wisdom of Crowds Positive Side

Answer 8

• independence
  
  • wrong in different directions
• what about knowing who’s the expert?
  
  • doesn’t matter
• what about scale?
  
  • N = 2 or N = 1000
• multiplexing vs. diversity gain
  
  • (1 - (1 - p)^N)N

Question 9

Suppose a group of 100 people are guessing the number of grapes in a jar. At the end, we are told that the average of the individual guess’ errors is 10. What is the error in the average if nobody collaborated and everyone submitted their guesses in a sealed envelope? How about if everyone copied one person?

Answer 9

• 0.1, 10

If everyone placed their guess in a jar, they would likely be independent, and so the error in the average would be reduced by a factor of 100 from the average of the individual errors. On the other hand, if everyone copied the same person, the guesses would be completely dependent, so the error in the average would be the same as the average of the individual errors.

Question 10

Bayesian Ranking

Answer 10

Given a set of products with # and average ratings:

~r_i = NR + n_ir_i / N + r_i

• N = total # of reviews
• R = summation of (# of reviews * rating) / N
• n_i = # of reviews for product i
• r_i = average rating for product i

Question 11

Websites who use Bayesian Ranking

Answer 11

• IMBDb
• Beer Advocate

Question 12

What happens as the number of ratings for a product becomes larger?

Answer 12

• The Bayesian rating of the product becomes closer to its personal “naïve” average.

Question 13

Amazon Rating Factors

Answer 13

• Bayesian ranking
  
  • # of people
  • N = N_minor N_avg
• too few or too outdated reviews penalized
• very high quality reviews help a lot
• major issues push ranking down a lot
  
  • electronics breaking down

Question 14

Suppose there are three brands of a product on Amazon, with 5, 10, and 15 ratings, respectively. The naïve averages for the products are 3, 2, and 1. What is the Bayesian rating of the second product?

Answer 14