Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Comp 3020 > Evaluation > Flashcards

Evaluation Flashcards

1
Q

What is the high level overview of evaluation?

A

detailed discussion on specific topics

  • analytical evaluation
  • usability evaluation
  • experiments
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Why evaluate?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What are the evaluation considerations?

A

There are a range of evaluation methods and settings

Your choice of method should be informed by the following:

  • Why
  • What
  • Where
  • When
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Evaluation considerations: Why? Why evaluate?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Evaluation Considerations: What

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Evaluation Considerations: Where

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

In general, when should evaluation occur?

A

Throughout the design process, from the first descriptions and sketches through to the final product.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are the other evaluation considerations?

A

And Stackeholders:

  • Management
  • Actual Users
  • Maintainers
  • Others impacted
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What are the general evaluation types?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

How do evaluation types complement one another?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What are the participants rights?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Once we’ve completed the research, what do we do with it?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Why does Data analysis and Interpretation get tricky?

A

If you don’t plan appropriately, your data can be meaningless, or worse, you may interpret it incorrectly, e.g., that your interface works when it doesn’t

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What are the Evaluation overview key points?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

In general, what are Analytical Evaluations?

A

Evaluations without involving users

Experts assess the system using structured techniques known to be effective at uncovering usability flaws

theoretical models of human performance can predict actual use

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What are the Analytical Evaluation Types?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

What are the Analytical Evaluation Inspections?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

What are Cognitive Walkthroughs

A

“Cognitive walkthroughs simulate a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action”

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

What is the process for Cognitive walkthroughs?

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Why is the process of Cognitive Walkthroughs tedios?

A

force you to consider every component, every step, from the user’s perspective

avoid “glossing over” the details because you get it

tried-and-true process for overcoming “designer blindness” – you can’t see the problems with things you’ve made

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

How do you get results form conginitive walkthroughs?

A

Through the process, information is recorded on:

Assumptions about what would cause problems and why (involves explaining why users would face difficulties)

Notes about side issues and design changes

Results are summarized
design is revised to fix problems identified

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What is a Heuristic Evaluation?

A

define: heuristic

a rule of thumb—a principle that is a shortcut for solving a problem or making decisions

not always right/true, but cognitive shortcuts

design heuristics

broad usability guidelines that can guide a developer’s design efforts

derived from common design problems across many systems

several researchers and practitioners have developed different sets of heuristics (e.g. domain specific)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

What is the general process for Heuristic Evaluation?

A

Systematic inspection of an interface design to see if an interface complies with a set of usability heuristics, or usability guidelines.

General process:
» 3-5 inspectors (usability engineers, experts…)

» inspect interface in isolation (~1-2 hr for simple interfaces)

» results are aggregated afterwards
single evaluator catches ~35% usability problems 5 evaluators catch ~75%

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

How does the proportion of usability problems found work?>

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

In general what is Nielsen’s Heuristics?

A

Nielsen’s Heuristics: Background

Developed by Jacob Nielsen in the early 1990s

Based on heuristics distilled from an empirical analysis of 249 usability problems

These heuristics have been revised for current technology

Heuristics still needed for some emerging technologies (e.g., mobile devices, AR, etc).

Design guidelines form a basis for developing heuristics

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

What are the actual Nielsen’s heuristics?

A
27
Q

What are the heuristics recommended for corporate web site evaluation?

A
28
Q

What are the Heuristic Evaluation Advantages?

A

A few guidelines identify many common usability problems

Fewer practical and ethical issues to deal with – no participants

Cheap and fast: a few guidelines identify many common usability problems

Provides common evaluation template (to compare approaches, systems)

29
Q

What are the Heuristic Evaluation Problems?

A

principles may be too general subtleties involved in use

designer may not be able to overcome being defensive / experts may disagree

you may actually have the wrong design altogether can be hard to find experts
false positives: does the rule always apply?
not complete: will miss problems

not a replacement for user testing

30
Q

In general, What is performance modelling?

A

Using models of human behavior to generate quantitative predictions of certain interface actions or sequences of actions

31
Q

What is the formula for fitt’s law and what is it?

A
32
Q

What does Fitt’s law tell us?

A

Fitts’ Law predicts how long it will take users to acquire targets once they know which target to select

33
Q

How do we measure decision time?

A

Hick-Hyman Law models the time it takes users to decide between n familiar alternatives

34
Q

What is the Hick-Hyman Law when items are equi-probable?

A
35
Q

What is Hick-Hyman Law when certain items are more likely to be chosen than others?

A
36
Q

What is the application for the Hick-Hyman Law?

A

Deciding on menu depth vs. breadth

37
Q

Hick-Hyman Law models decision time, but not searching time?

A

Models decision time, not searching time

If the user is not familiar with the interface elements, they need to investigate each one. It’s not a pure search task:

  • Time to search through n items is linear, not logarithmic
38
Q

What is the Keystroke Level Model?

A

Given a task consisting of a sequence of steps

  • How long will it take the user to perform those steps given a specific interface?

Keystroke Level Model (KLM)

  • Models performance given a sequence of steps for an expert user
39
Q

In KLM how is performance calculated?

A
40
Q
A
41
Q

What are performance modelling advantages?

A

Can evaluate components of interface prior to building it

Good for comparing different interface possibilities

Can get the kinks out of interface prior to full user testing/experimentation

42
Q

What are the performance modelling problems?

A

Difficult to model complex tasks

  • E.g., consider designing a KLM for your complete project

Most models consider only expert behaviour

For really accurate predictions, coefficients (those as and bs) need to be determined empirically

43
Q

What is the analytical evaluation summary?

A

Analytical evaluations are a set of structured techniques for evaluating interfaces without (necessarily) involving end users

Often used to address major usability issues and inefficiencies prior to, and in parallel to involving end users

Should always complement rather than replace evaluations with end users

44
Q

In general, What is a Usability Test?

A

A usability test is a “formal” method for evaluating whether a design is learnable, efficient, memorable, can reduce errors, meets users’ expectations, etc.

  • users are not being evaluated
  • the design is being evaluated
45
Q

How do you get users for a usability test?

A

Bring in real users

Have them complete tasks with your design, while you watch (ideally with your entire team)

Measure and record things

  • task completion, task time, error rates
  • satisfaction, problem points, etc.
  • use a think-aloud protocol, so you can “hear what they are thinking”
46
Q

How is the data used from a usability test?

A

Use the data to

  • Identify problems (major ones | minor ones)
  • Provide design suggestions to design/engineering team
  • Iterate on the design, repeat
47
Q

What are the important considerations for a usability test?

A

Usually takes place in a usability lab or other controlled space

Major emphasis is on

  • selecting representative users
  • developing representative tasks

5-10 users typically selected
Tasks usually last no more than 30 minutes
The test conditions should be the same for every participant Informed consent form explains ethical issues

48
Q

What are user testing environments?

A

Best environment depends on pragmatic considerations, as well as what you’re looking for

  • Do you want your whole team to be able to view? Do you want to be able to review a test?
  • How important are interruptions?
  • What are you resources?
49
Q

What are pilot studies and why are they challenging?

A

Especially important for usability testing Make sure your plan is viable

All the corners are checked (your script, questionnaires, tasks, etc., all work)

It is worth doing several to iron out problems before doing the main study

Ask colleagues if you can’t spare real users

Challenging

EVERYONE thinks they don’t need to do them. It all fits in your head, what can go wrong?

From nearly 15 years experience of running studies my advise is..

  • just pilot. OK? You need to. Something will go wrong
50
Q

How should you use tasks with a usability test?

A

A task is designed to probe a problem

Tasks should be straightforward and require the user to find certain items, or do certain operations

They can be more complex such as solving particular problems

Sample tasks for a weather network web site:

  • What is the forecasted weather for Winnipeg?
  • What is air quality in Los Angeles today?
  • What is the level of humidity in Winnipeg?
  • What is the forecast for Ottawa for the upcoming weekend?
51
Q

You are developing a user test for a new CS web page. Identify 6 tasks for the test:

A

You are developing a user test for a new CS web page. Identify 6 tasks for the test:

Task 1: Identify the instructor for Comp 3020

Task 2: Find the e-mail address of the Comp 3020 prof

Task 3: Find the admission requirements for the M.Sc. Program

Task 4: Find out the first day of classes next term

Task 5: Locate the requirements for being a Co-op student

Task 6: Identify whether the graduate Graphics course is a “fundamentals” course

52
Q

How many participants is enough for usability testing?

A
53
Q

How can questionaires be used for usability testing?

A
54
Q

How can one use Likert scales for usability testing?

A
55
Q

How do you observe people in usability testing?

A
56
Q

How do you indirectly track activities?

A
57
Q

What are unobtrusive vs obtrusive oberservations?

A
58
Q

ASK vs LOOK for being obtrusive?

A
59
Q

What is the observation recap?

A

main point: minimize and think about the impact of your observations and any interventions!

unobtrusive – minimize interactions

obtrusive – try to maximize learning (at what cost)

60
Q

What is qaulitative vs quantitative data?

A
61
Q

What should you analyze and report?

A

Report on times to complete task, number of errors

Provide simple statistical measures: mean, median, std dev.

Describe interaction patterns
e.g., four ways that people may use the interface

62
Q

How do you present results?

A
63
Q

Give the usability testing summary

A

Usability testing is a form of applied experimentation

Users are brought into controlled environments to complete focused tasks

Focus of the testing is on having representative users and tasks

Typically both objective (e.g., task completion times, errors) and subjective data is collected (e.g., questionnaire data)

Important end goal: draw meaningful conclusions about your system’s current strengths and limitations

64
Q

What are the two main types of evaluation and how do they differ?

A

Formative Evaluation: Is done at different states of development to check that the product meets user’ needs

Summartive evaluation: assesses the quality of a finished product

Comp 3020 (14 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions