PL1010: Research Design and Methods Flashcards
Your Deck Mentor for the Research Design and Methods deck is Bogna. You can email bogna.frykowska@forward-college.eu with any questions/suggestions about the flashcards in this deck.
Categorical variable
Variable with scores that are not on a numeric scale
Descriptive statistics –
Summarise samples – giving someone the main points in a simple form To describe data, we will use graphical and numerical (statistical) techniques
Inferential statistics –
Examine patterns in the data and consider how much data we have You can then draw conclusions about a population based on the analysis of a sample. -> conceptual replication
Summarising
collecting and summarising data
Standardisation
creating a system in which data can be assigned values (high vs low vs average)
Statistical inference
the ability to draw general conclusions from samples
How many times does a particular score occur?
Percentages/Averages Scores for a particular variable (Frequency statement)
Do scores for one variable correlate with scores for the other variable?
Statement about association
How strong is the correlation or association between two variables?
Statement about association
Do I trust that there is a “genuine” association (relationship)?
Statement about relationship between two variables
Frequency Distribution?
show scores in order and their frequency of appearance in the sample
Negatively skewed
Positively Skewed
When not to describe the skew of data?
When we cannot put our scores in order , from lowest
to highest so when we are describing a categorical
variable with unordered categories
Unimodal?
One major peak
Bimodal
Two major peaks
Approximately symmetrical
The Sample Variance formula
the n-1 leads to an unbiased estimate
How do outliers and the mean relate to each other?
Outliers are extreme values that differ from most values in the data set. Because all values are used in the calculation of the mean, an outlier can have a dramatic effect on the mean by pulling the mean away from the majority of the values.
What happens to the mean, median and mode in a skewed distribution
in normal distributions, they all take on the same number
frequency distribution
pattern of frequencies of a variable
Why are histograms good?
effective visual summary of a variable’s central tendency and variability
Why is the IQR a good measure of variability?
Because it’s not affected by outliers like the range is as it only includes the middle 50% of data?
Difference between central tendency and variability?
Central Tendency just tells you where the most important point lie while variability sums up how far apart and thus generalisable they are.
standardised scores (z-scores)
Z-Score Formula
Falsifiability
capacity for some proposition, statement, theory or hypothesis to be proven wrong (through systematic empiricism) a basis provided by the null hypothesis
null hypothesis
states the contrary of the experimental or alternative hypothesis
falsifiable hypothesis
can be logically contradicted by an empirical test that can potentially be executed with existing technologies .
A linear correlation describes
Two variables that are either proportionate or anti proportionate
Correlation coefficient
A number also represented by “r” describes positive (r=1), negative (r=-1) or no correlation (0=r)
Face Validity
basic form of validity demonstrated when a measurement procedure superficially appears to measure what it claims to measure
Concurrent Validity
demonstrated when scores obtained from a new measure are directly related to scores obtained from an established measure of the same variable.
Predictive Validity
demonstrated when scores obtained from a measure accurately predict behaviour according to a theory.
Construct Validity
requires that the scores obtained from a measurement procedure behave exactly the same as the variable itself.
What needs to be taken into consideration for construct validity
based on many research studies that use the same measurement procedure and grows gradually as each new study contributes more evidence.
Convergent Validity
demonstrated strong relationship between the scores obtained from two or more different methods
Divergent Validity
demonstrated by showing little or no relationship between the measurements and two constructs
Test-retest reliability
established by comparing the scores obtained from two successive measurements of the same individuals and calculating a correlation between the two sets of scores.
Inter-rater reliability
degree of agreement between two observers who simultaneously record
measurements of the behaviours
Split-half reliability
obtained by splitting the items on a questionnaire or test in half, computing a
separate score for each half, and then calculating the degree of consistency between the two scores for a
group of participants.
What are the two types of measures of reliability
Successive and simultaneous measurements
Construct
hypothetical attributes or mechanisms that help explain and predict behaviour in a
theory
Operational Definition
procedure for indirectly measuring and defining a variable that cannot
be observed or measured directly. An operational definition specifies a measurement procedure (a set of operations) for measuring an external, observable behaviour and uses the resulting measurements as a
definition and a measurement of the hypothetical construct.
What is naturalistic observation
A form of non-participant observation where a researcher is in a natural setting in which behaviur usually occurs without interupting
What is In participant observation
researcher engages in the same activities as the people being observed
in order to observe and record their behaviour.
What are naturalistic observation usually used for
used to describe non-human behaviour or children
What are the benefits and disadvantages of naturalistic observation (5)
external validity: real world setting behaviour is not manipulated overcoming ethical barriers –> instigating spanking vs. observing spanking time-consuming: having to wait till behaviour occurs research is prone to interruptions
When is participant observation needed
When simple observation is not possible. e.g. studying cults or gangs because their presence would alter the behaviour
What are advantages and disadvantages of participant observations (5)
access to information and observation unavailable to mere outside observation high external validity because of naturalistic setting time consuming observation potentially dangerous for researcher (sensitive nature) observers presence might alter participants’ experience –> objectivity?
What is Structured observation?
or contrived observation is the observation of behaviour in settings arranged
specifically to facilitate the occurrence of specific behaviours so they don’t have to wait for them to happen
Advantages and Disadvantages of structured observations
can be held in laboratory or other controlled settings to percipitate the behaviour that they want to observe –> good for developmental psych can be held in what is perceived a sa natural environment (field setting) less time consuming how natural is the behaviour?
During a study using observational methods, it is common to have two observers record behaviour
simultaneously. What is the purpose for this procedure?
objectivity of the measurements
In an observational study of children diagnosed with autism spectrum disorder, you record how
much time each child spends playing alone during a 30-minute observation period. Which method
of quantifying behaviour is being used?
duration
When researchers use behavioural observation techniques to measure behaviours in movies, what
is the measurement process called?
content analysis
What are behavioural tasks
usually computer-controlled, structured tasks measured across multiple repeated trials that researchers use to collect behavioural measures such as
response times and task accuracy measures.
What are many behavioural tasks structured around?
human information processing like cognitive tasks assessing attention, memory, language and decision making
What can also be measured by behavioural tasks
attitudes, preferences aside from cognitions
What is the main measure of interest in behavioural tasks
not the usually correct answer but speed of the response (response or reaction time)
What is a task paradigm
task originally constructed to investigate a particular hypothesis is used and adapted to examine others subsequently providing a standard model for line of research
What are a few physiological measures commonly used
monitoring
heart rate or blood pressure, measuring
galvanic skin response, imaging
techniques positron emission tomography
(PET) scanning magnetic resonance imaging
(MRI) electroencephalogram (EEG) magnetoencephalography (MEG).
What are two prerequisites of behavioural observations
behaviour is not disturbed observations are based in subjective judgments and intepretations which pose a threat to reliability so need for more than one observer
Archival research
involves looking at pre-existing records (archives) to measure behaviours or events
that occurred in the past
Content analysis
measuring the occurrence of specific events, actions or statements in written
text (e.g., literature, press reports, transcripts) or film/video recordings (e.g., movies, television programmes) or similar media
How is the issue of interpretation in observational designs addressed (3)
well-defined categories of behaviour well-trained observers multiple observers or coders to assess inter-rater reliability
What are behaviour categories
well/defined sets of behaviour that is to be observed which helps isolating relevant behaviours
How are observations quantified (3)
frequency: how many times does something occur in the given time-frame duration: for how long does a behaviour occur interval: does a behaviour occur in a given interval
When are the three quantification methods most appropriate
first two techniques are often well suited for specific behaviours but can lead to distorted
measurements in some situations. For example, a bird that sings continuously for the entire 30-minute
observation period would get a frequency score of only 1. Another bird that sings 25 times with
each song lasting two seconds would get a duration score of only 50 seconds. In such situations, the interval method provides a way to balance frequency and duration to obtain a more representative
measurement
How do observers overcome issues of complex situations that cannot be watched multiple times
creating a recorded sample or taking a general sample
How is a sample taken
first step in the
process of sampling observations is to divide the observation period into a series of time intervals.
What are the three forms of sampling
Time sampling: sequence of observe–record–observe–record is continued through the
series of intervals Event sampling: identifying one specific event or behaviour to be observed and recorded
during the first interval, then shifting attention to a different event or behaviour during the second
interval, and so on, for the full series of intervals. individual sampling: identifying one participant to be observed during the first interval,
then shifting attention to a different individual for the second interval, and so on
How is reliability and objectivity of observations made from content analysis archival research ensured?
behavioural categories and preparing a list of specific examples to define exactly
which events are included in each category being measured quantification methods for each behavioural category multiple observers and coders
What does reliability often refer to
the relationship between two measures as shown by its correlation
When is assessing split/half reliability common
single variable measured within a test containing multiple items so that the internal consistency can be evaluated
What is the issue of split-half reliability
scores obtained are only from half of the test items which is less reliable because it underestimates the true reliability of the full test
What is the Spearman-Brown formula, and what does it do?
adjusts the correlation between the halves of split-half reliability tests, the effect is to increase the size of the correlation to produce a better estimate for the full test
What problem of split-half reliability does the Kuder-Richardson Formula 20 solve
The idea that tests can be split in different ways which potentially skews the results
What is the Kuder-Richardson Formula 20
a formula to estimate the average of all possible split-half correlations obtainable but limited to tests with dichotomic answers
What do all of the components of the K-R20 mean
n / number of items SD p / the proportion of the participants whose response is coded 0 q / proportion of the participants whose response is coded 1
How is the K-R20 limited
It can only be used for test that have dichotonomical answer systems and Cronbach’s alpha is a modification to this
What are the components of cronbach’s alpha
the extension is that it includes the sum of the variance produces values between 0 and 1.00
What is Cohen’s Kappa formula, and what is it used for?
calculating inter-rater reliability not using a simplistic formula and relying on data prone to circumstances and chance
What do the elements of Cohen’s kappa mean
PA: observed per cent agreement PC; per cent agreement expected from chance
When is Cronbach’s alpha used?
when we have a scale that combine responses from several rating-scale items.
What’s the scientific method
Way of acquiring knowledge that includes the genesis of a hypothesis and then its systematic investigation
What are the three principles of the scientific method
Empirical: systematic/structured observation (with attempts to isolate the relationship between variables) Public: Data available for evaluation/verification to be replicated Objective: low bias
What are the steps of the research process?
Research Idea (Field Review) Hypothesis Defining Variables & Measure Participant Selection (Criteria, Ethics) Research Strategy (Design/Ethics) Research Design Selection Evaluate Data & Report Results Refine research idea
What are the four main functions of statistics?
Summarising (descriptive statistics) Providing estimates for populations taken from samples (inferential statistics) Data simplification/reduction when there is a lot of data New statistical techniques are needed for mining big data
What are the seven reasons research ethics must be considered at every stage of the process?
Dictate measuremets Participant selection Research stategies x population Research design x behavior/populus How the study is conducted Data analysis Reporting results
What is a random sample?
By drawing randomly from a population all events should have an equal chance of being included
What is meant by inference?
Estimation based on population
What are the three decisions of statistical procedures?
Type of data Differences vs. Relationships Number of groups vs. variables
What are research ethics?
Principles that concerns the responsibilty of researchers to be honest and respectful towards participants
What are the 5 research strategies?
Descriptive (examining individual variables) Correlational (two variables for each individual) → numerical Experimental (cause-effect) Quasi-experimental (less control, assignment) Non-experimental
What is a research strategy
General approach to the research shaped by the research question (what do I want)
What is meant by the research design
General framework to implement research strategy (how do i achieve what I want)
What are the three pillars of research design?
Group vs. individual Same individuals vs. different Number of included variables
How can a study’s aspects threaten external validity? (4)
General q: how can the results obtained with this procedure be replicated to other procedures? Multiple treatment interference (fatigue/practise) Novelty effect (anxiety/excitement) Experimenters’ influence
How can participants/ subjects threaten external validity?
volunteer/selection bias/ uni students WEIRD characteristics Cross-species comparison
How can the measurements threaten external validity?
Assessment sensitisation (awareness) Pre-test sensitisation Results of an operationalised concept can be moderated by the measure → generality across measures Time of measurement
What is internal validity?
Continuity that the observed results can account for the propose cause-effect relationship
How can environmental variables threaten internal validity?
The room size, the colour of the walls, time of the day, the gender of the experiementer NO SYSTEMATIC DIFFERENCES IN THE ENVIRONMENTS
How can individual differences threaten internal validity?
IQ, age, gender, health conditions WEIRD vs non WEIRD
How can time-related variables threaten internal validity?
Individual differences that accumulate over time Comparing scores and time influences
What is the biggest threat to internal validity?
The effect of extraneous variables that confound the results
What are artefacts?
External factors can become a confounding variable and distort both internal and external validity
Which artefacts concern the participants?
Demand characteristics: participants react to cues that reveal the purpose/ hypothesis Reactivity: induce behaviour (subject roles)
What do participant-related artefacts primarily threaten?
Internal validity: reactivity explains phenomena; not generalisable
What are non-participant-related artefacts?
Experimenter bias (single-blind; double-blind) Exaggerated variables
What are the four categories of formal measurement theory and who came up with them?
SS Stevens Nominal: just naming Ordinal: small, smaller, smallest Interval: 0-1 = 7-8 Ratio: zero = nothing
What is a discrete, continuous, independent and dependent variable?
Discrete: variable that is limited (age, gender) Continuous: exists on a continuum basically infinite between highest/lowest IV: variable manipulated/changed to see whether it has an effect on the DV that might change because of the manipulation DV: variable that, though measured, is not being controlled
What is the role of measurement scales?
The numbers don’t necessarily say anything concrete about the objects measured <i>ex.: if I scored high on a test, but someone else scored lower, it’s not necessary because they remembered less even though the data might suggest it → we assume that they mean I remembered more</i>
What is the purpose of a frequency distribution?
Organising data into a meaningful order of how many times
Which variable do I usually find on the X- and Y-Axis in histograms vs. line graphs?
histogram: dv-iv Line/Bar graph: iv-dv
What is the mode, median, mean (+formulas)?
Mode: the highest point in the graph Median: 50th percentileMean: Sum of N/ N
If the mean is slightly larger what does it probably say about or distribution?
Positively skewed
When will the mean and the median be equal?
Symmetric distribution
The benefit of the mode is?
- Representing categorical data * More informative *But not very reflective of the remaining data set
The benefit of the median is?
Not affected by outliers Not stable in comparison and not useful to calculation
What is the trimmed mean and when do we use it?
Trimming the interval and taking the mean when the data is very skewed
What does central tendency refer to?
The scores tendency to distribute in a certain way?
What is meant by dispersion/variability around the mean?
Determining how the scores relate towards the mean score
What are measures of variability?
Range Interquartile Range Standard deviation Sample variance if sd=0 so we square scores and then take the sum so negative scores become positive Absolute Mean Deviation SD but not squared which could= 0
What is the advantage of a bar chart?
- Comparing categories * Mirrors other visualisation techniques were the spread is along the X-axis and the frequency or percentage is along the Y-axis already hints at modality and skewness
Why is a normal distribution the most important distribution ?
We assume normal distribution, we can use most statistical techniques Most dependent variables are thought to be normally distribution We can make inferences If we draw a theoretical representation of all possible sample means the sampling distribution would also be normally distributed
What are the mean and standard deviation in the standard normal distribution?
mean=0 and sd=1 → z-score standard score specifying the amount of distance of the sd
What is an alternate name for the y-axis/x-axis?
ordinate/abscissa
What is the mean and the standard deviation of t-scores?
Mean = 50 SD= 100
What is meant by sampling error?
Chance difference = the way some statistics naturally varies from sample to sample = in that it wll always deviate from the parameter it is the random variability = standard deviation of the sampling distribution
What is sampling distrubtion?
If we calculated a statistic from different samples from one population and visualised it we would see how the statistic is distributed
What is the standard error of the mean?
The standard deviation / variability from the estimated parameter mean of the distribution
What is the purposes of the standard error?
It gives us an indication of just how much the sample statistic might differ across the samples. It’s like a z-score for all the potential differences we could observe but would not go against the finding
What are the logical steps of hypothesis testing?
Set up a research Hypothesis H1 Set up a null hypothesis H0 Get a sample and sample distribution of sample statistics (eg mean) und the H0 Calculate probability value of of sample statistic at least as large as the one obtained Reject or Fail to Reject H0
What’s the philosophical hypothesis of the null hypothesis?
M1-m2 =0 has been proposed by Fisher With the logic that we can always show that something is false
How do you calculate the IQR?
Order the scores Find the median location (N/2) Find the median of the upper and lower quartile N low /2; N high /2
What is the difference between sample and test statistics?
Stats describing samples vs. statistical results of specific proceedures with their individual sampling distributions
What is the formula for Cohen’s d effect size?
d = (M1 – M2) / Spooled
What do d= 1 and d= .5 indicate?
That the effect the difference is either twice or half as large as the standard deviation
How do I interepret the effect size using cohen’s d?
.2 is small because the mean difference is around .2 standard deviation → .5 (medium), → .8 (large)
What is the defining difference between in-between subjects and within-subjects design?
Create equivalent groups and compare them in different trearment conditions vs. use the same group of participants compared in all different trials
How can the trials be administered?
Subsequentially or intermixed
What’s instrumentation?
Changes in the measurement or measuring instruments (observations are heavily dependent on the observing researcher)
What is a research strategy?
A method of data collection
What is a non-experimental and quasi-experimental research strategy?
No manipulation and controlling for extraneous variables vs. limitation of confounding variables without controlling the environment
What aspects threaten internal validity?
Time order effects the fact that counterbalancing cannot be applied
What are the absolute deviations from the mean?
X- Mean that’s why if we average and take the root of them we get the standard sample deviation
Suppose you sell ice cream with three different flavours: chocolate, strawberry and yogurt. The ice cream flavours are measured on a ____________ level. You sell ice cream to children, adults and elderly people. These age groups are measured on a ____________ level.
nominal; ordinal
operational definition
defining a variable in terms of the set of steps or procedures that the researcher goes through in order to manipulate or measure the variable
right skewed
positively skewed
What does a negatively skewed distribution reveal?
A lot of people got close to the maximum score
What does central tendency mean?
average score
how much is removed in the 5% trimmed mean’
10% in total 5 from the top and the bottom
‘Alex has an accuracy score of 15, while Samir has an accuracy score of 5; therefore, Alex was three times as accurate as Samir.’ For this statement to be appropriate and precise, which scale(s) of measurement could accuracy have been measured on?
Question 7
a. Interval
b. Ratio
c. Ordinal
d. Nominal
This statement is only meaningful if a score of ‘0’ represents ‘no accuracy’. Therefore, the scores should be from a ratio scale if this statement is to be made. b
Age in months is an example of a variable with a ratio scale of measurement. Select one:
True
False
T
Sampling variability
differences in or across samples due to random things happening
Confidence
accuracy across 100 treatments that we’ve found the likely range limits
Type I error
<img></img>
Type II error
<img></img>
What are two ways to visually represent to measurement data variables?
- scatter plots 2. contingency tables/crosstabulation
What is a way to visually represent a mix of categorical and measurement data?
compound histogram
What is a way to visually represent categorical data pairs?
crosstabulation
How should the strength of a correlation be interpreted?
- Perfect:(-) 1/1 * Strong:(-) 09-07 * Moderate:(-).6-.4 * Weak: (-). 3-.1 * None:0
What do frequency distributions for a categorical variable not include?
cumulative percentages
What are the groupings of scores in histograms called?
bins
Do these images show the same data?
Yes
Which visual representation should you choose if you want to show that variables vary simuntaneously?
scatter-plots
What does a boxplot do?
summarises the data while showing the range, interquartile range, as well as the min, max and the median
When is the mean most useful?
best for interval/ratio measurement data (categorical data can hardly be split into 2), needs equal spacing between adjacent values
What is the mode most useful for?
all but notably for nominal/ordinal categorical data because popular choice
When you collect data from a sample, the sample variance is used to ?
make estimates or inferences about the population variance and comparing the variance of samples helps you assess group differences
How is the sample mean related to variance and standard deviation?
it is expanded on in the formulas for variance and standard deviation
Which five steps need to be taken to calculate the sample variance?
- The mean (∑ 𝑋 /N) 2. The Deviation from the mean X- (∑ 𝑋 /N) 3. Squared deviation from the mean (X- (∑ 𝑋 /N))^2 4. Find the sum for all scores and devide by N-1 5. Take the root to find the standard deviation or z-score
What are strengths naturalistic observation?
- behaviour observed in the real world 2. useful for non-manipulated behaviours 3. actual behaviours observed and recorded
What are strengths of participant observation?
- when natural observation is not possible 2. get information not easily accessible 3. participation gives unique perspective
What is a strength of structured observation?
do not have to wait for behaviours to occur
What is a weakness of structured observation?
less natural
What are weaknesses of participant observation?
- time-intensive 2. potential loss of objectivity 3. increased chance of observer influence
What are weaknesses of naturalistic observation?
- time-intensive 2. potential subjective interpretation 3. potential for observer influence
Variables are
properties of objects that vary in the values that they take on
A score is
an individual value for a variable
Measurement data describes
scores on a numerical scale
Categorical data describes
scores not on a numerical scale
A Population describes
a complete set of scores that might be of interest
A Sample is
a sub-set of scores from a population which were obtained
A parameter is
a number that summarises the entire set of scores in a population
A statistic is
a number that summarises the scores in a sample
Descriptive Statistics…
summarise samples by presenting the main points in a simplified way
Inferential statistics…
examine patterns in the data and consider the amount of data
Ethnicity or political ideology are examples
nominal variables
The lower quartile signifies
all scores one-quater into the distribution
The upper quartile signifies
all scores three-quaters into the distribution
The range signifies?
the difference between the minimum and the maximum score
The interquartile range signifies?
the difference between the upper and lower quartiles
The standard deviation is more informative about the variability than the variance.
False
The standard deviation is expressed in larger units than the variance.
False because the root is taken
What does the standard deviation tell me?
how far, on average, a value lies from the mean which is why it is derived from the variance (square root)
Which graph describes a correct null hypothesis?
right
The p-value can be defined as
the probability of obtaining a significant result when the null hypothesis is true
What are the five characteristics of normal distributions?
