Side-by-side

Question 1

Rating philosophy

Answer 1

1. users should find what they are looking for
2. users should get the best results in the most helpful order
3. users should have options and choices
4. results should feel right for the users’ locale

Question 2

Step 1

Answer 2

Understand the query

Question 3

Step 2

Answer 3

Assign utility and PQ ratings to all results on each side

Question 4

Step 3

Answer 4

Assign P, F, and D flags when appropriate

Question 5

Step 4

Answer 5

Mark duplicate results

Question 6

Step 5

Answer 6

Find the best results in the result sets shown (note them)

Question 7

Step 6

Answer 7

Decide which set of results is better for users in the rating locale

Question 8

Step 7

Answer 8

Choose an overall SxS rating based on the results

Question 9

Step 8

Answer 9

Write a comment to explain your rating

Question 10

Rating quality is evaluated on

Answer 10

Utility ratings
PQ ratings
overall SxS rating
reasoning behind your ratings

Question 11

result set for a Go query

Answer 11

should have the target result at the top

Question 12

result set for a Know query

Answer 12

should have a nice mix of resources to find good information

Question 13

result set for a Do query

Answer 13

should give users a lot of high quality options to accomplish their goal

Question 14

diversity

Answer 14

results in a set should work together to address different aspects of query interpretation/user intent and provide a good collection of resources for users

Question 15

three types of diversity

Answer 15

1. query interpretation/user intent
2. information sources/websites
3. result type

Question 16

about the same

Answer 16

both sides are equally bad or good

Question 17

much better

Answer 17

users would have a much easier time finding what they are looking for on one side

Question 18

Potential reasons for a much better rating

Answer 18

higher placement of target results
much better order of the results
really good unique results

Question 19

Rating comments are not required when you select

Answer 19

about the same

Question 20

Rating comments are required when you select

Answer 20

slightly better, better, or much better

Question 21

comments about individual results

Answer 21

primarily for your sue in assigning a SxS rating