Side-by-side Flashcards
Rating philosophy
- users should find what they are looking for
- users should get the best results in the most helpful order
- users should have options and choices
- results should feel right for the users’ locale
Step 1
Understand the query
Step 2
Assign utility and PQ ratings to all results on each side
Step 3
Assign P, F, and D flags when appropriate
Step 4
Mark duplicate results
Step 5
Find the best results in the result sets shown (note them)
Step 6
Decide which set of results is better for users in the rating locale
Step 7
Choose an overall SxS rating based on the results
Step 8
Write a comment to explain your rating
Rating quality is evaluated on
Utility ratings
PQ ratings
overall SxS rating
reasoning behind your ratings
result set for a Go query
should have the target result at the top
result set for a Know query
should have a nice mix of resources to find good information
result set for a Do query
should give users a lot of high quality options to accomplish their goal
diversity
results in a set should work together to address different aspects of query interpretation/user intent and provide a good collection of resources for users
three types of diversity
- query interpretation/user intent
- information sources/websites
- result type
about the same
both sides are equally bad or good
much better
users would have a much easier time finding what they are looking for on one side
Potential reasons for a much better rating
higher placement of target results
much better order of the results
really good unique results
Rating comments are not required when you select
about the same
Rating comments are required when you select
slightly better, better, or much better
comments about individual results
primarily for your sue in assigning a SxS rating
