Experimental Designs Flashcards
What is the independent variable ?
I predict that increasing recommended offers in Amazon will result in increasing products sales.
Recommended offers.
What are the types of research ?
Descriptive (y) , Relational (x <—> y), Experimental(x -> y)
What type of research :
Describes a situation or a set of events.
X happens.
Descriptive
What type of research :
Identifies relation between multiple variables.
X is related to Y.
Relational
What type of research :
Identifies causes of a situation or a set of events.
X is responsible for Y.
Experimentalcontextual setting (enternal validity, you can apply it or based in the real world / less control)
Descriptive Research uses what types of methods ?
Qualitative Methods (e.g., interviews)
Quantitative Methods (e.g., usage analytics)
______________ assignment means assigning individuals at random to groups in an experiment, so that each individual of the population has the same chance of becoming a participant in the study
Randomizing
The non-random arrangement of experimental units into groups (blocks/lots).____________ reduces known but irrelevant sources of variation between units and thus allows greater precision in the estimation of the source of variation under study.
Blocking
_____________It is an inductive estimate of the degree to which conclusions about causal relationships can be made, based on the measures used, the research setting, and the whole research design
Internal Validity in a controlled setting (higher internal validity , you can be sure x is causing y , you have full control)
______________It concerns the extent to which the (internally valid) results of a study can be held to be true for other cases, for example to different people, places or times. In other words, it is about whether findings can be validly generalized
External Validity in a contextual setting (higher external validity, you can apply it or based in the real world and you have less control)
Usability is described as the extent to which a product can be used by specified users to achieve specified goals with __________________________ in a specified context of use
effectiveness, efficiency, and satisfaction
Effectiveness is
accuracy and completeness with which users achieve specified goals
Efficiency is
resources used in relation to the results achieved Typical resources include time, human effort, costs, and materials.
Satisfaction
the extent to which the user’s physical, cognitive and emotional responses that result from the use of a system, product, or service meet the user’s needs and expectations
____________ are when the chef tastes the cake
finding and fixing usability problems. You can quantify the problems in terms of frequency and severity, track which users encountered which problems, measure how long it took them to complete tasks, etc.
Formative Tests
___________ are when the guests taste the cake
describing the usability of an application using metrics. This can be, for instance, a
comparison of different versions of a product or comparison of competing products.
Summative Tests
What are the types of evaluation:
1.Controlled settings involving users
Users’ activities are controlled in order to test hypotheses and measure or observe certain
behaviors. The main methods are usability testing and experiments.
2. Natural settings involving users
There is little or no control of users’ activities in order to determine how the product would be
used in the real world. The main method used is field studies.
3. Any settings not involving users
Consultants and researchers critique, predict, and model aspects of the interface in order to
identify the most obvious usability problems.
Qualitative research helps us understand:
- Behaviors, attitudes, and aptitudes of potential and existing product users
- Technical, business, and environmental contexts - the domain - of the product to be designed
- Vocabulary and other social aspects of the domain in question * How existing products are used
Give controlled or contextual user environment examples :
Controlled environment - Lab usability tests, Focus groups , Participatory design Interviews
Medium contextual - Remote usability test,Intercept study , Interviews
Highly contextual - Field Studies, Diary Studies, Interviews
__________ can be extremely flexible and vary between Unstructured, Semi-Structured, and Structured
Structured, closed questions limit users to a small number of predefined choices. Examples include yes-no questions, multiple choice, true-false, and Likert-scale questions, asking for ratings on a scale of 3, 5, 7, or more possibilities
Interviews
___________ reliability refers to the extent to which two or more individuals agree.
Interrater
A mix of direct observation and user interviews to better understand context-of-use
__________________ is based on a master-apprentice model of learning: observing and asking questions of the user as if she is the master craftsman, and the interviewer the new apprentice
Contextual inquiry
___________ is a visualization that tells a narrative of the end user trying to accomplish a specific task or goal
Persona development is focusing on end-users of the software being developed, and Narrative Journey Maps capture their interactions and help define goals/motivations and pain points.
Journey Map
____________ is a technique to understand how users organize information and concepts into categories. It can be a valuable tool to uncover one aspect of a user’s mental model, the technique assumes that the subject has refined organizational skills and that how they sort a group of abstract topics will correlate to how they will end up wanting to use your product.
It can also be moderated or unmoderated.
Card Sorting
__________ are quite useful for gauging initial reactions to a product’s form.It can also gather reactions to a product that the respondents have been using for some time.
Focus groups
_____________ is an expert review method in which interface experts simulate users, “walking through” a series of tasks,
Because of the exploratory nature of a cognitive walkthrough, it can give an understanding of how users might interact with an interface the first time that they attempt to use it
A cognitive walkthrough
In a___________, an expert takes a set of heuristics (rules of thumb) and compares the heuristics to the interface in question
They are short sets of usually no more than 10 interface rules. To be truly effective, the expert must be very familiar with the heuristics and have previous experience in interpreting them.
Heuristic review
_________ is important when choosing a dependent variable. A measure is perfectly ______ if you get the same result every time
you repeat the measurement under identical conditions.
Reliability, Reliable
What is Task completion time ?
Time from users who completed the task successfully
______________ is a scientific discipline concerned with the construction of assessment tools, measurement instruments, and formalized models that may serve to connect observable phenomena (e.g., responses to items in an IQ-test) to theoretical attributes (e.g., intelligence)
Psychometrics
______________ is a body of related psychometric theory that predicts outcomes of psychological testing such as the difficulty of items or the ability of test-takers
It is a theory of testing based on the idea that an observed or obtained score (X) on a test is the sum of a true score (T) and an error score (E)
Classical Test Theory (CTT)
Consistency of results when you repeat the same test on the same sample at a different point in time. Use Pearson or Intraclass Correlation Coefficient (ICC)
Test-retest Reliability
The extent to which two or more raters (or observers, coders, examiners) agree. Different from Intra-rater reliability (i.e., agreement among repeated administrations of a test by a single rater). It can also use ICC, percentage agreement, or kappa tests.
Inter-rater Reliability
_________ can be inferred by testing the internal consistency of items. For that, researchers use a measure of internal consistency known as Cronbach’s α
Reliability
_____________ is a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables
IRT models are often referred to as latent trait models
The term latent is used to emphasize that discrete item responses are taken to be observable manifestations of hypothesized traits, constructs, or attributes, not directly observed, but which must be inferred from the manifest responses.
Item Response Theory
__________ is the extent to which a test accurately measures what it is supposed to measure
It refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests
Test validity
Examples of post-study questionnaires:
- SUS - System Usability Scale (10-item)
- QUIS - User Interface Satisfaction (6 to 27 items)
- SUMI - Software Usability Measurement Inventory (50-item) * PSSUQ - Post-Study Usability Questionnaire (13 to 19 items) * UMUX - Usability Metric for User Experience (4-item)
- UMUX-LITE (2-item)
___________ is shown to provide a reliable measure for ease of use when applied after each task
Single Ease Question (SEQ)
The scales of the _________________________cover both classical usability aspects, also known as pragmatic usability (efficiency, perspicuity, dependability), and user experience aspects, also known as hedonic usability (novelty, stimulation, attractiveness)
User Experience Questionnaire (UEQ)
