C1 Intro to Probability and Data Analysis with R M4 Exploratory Data Analysis | Inference Intro Flashcards
What is the arithmetic average called?
Mean
The mean is calculated by adding all values and dividing by the number of values.
What term refers to the midpoint of a data set?
Median
The median divides the data into two equal halves.
What is the most frequent observation in a data set called?
Mode
A data set can have more than one mode or none at all.
What measure indicates variability around the mean?
Standard deviation
Standard deviation quantifies the amount of variation or dispersion in a set of values.
What is the formula for calculating the range of a data set?
Max - Min
Maximum value - Minimum value found in the data set
The range provides a measure of how spread out the values are.
What does the interquartile range represent?
The interquartile range represents the range of the middle 50% of the distribution.
It is the distance between the first quartile (25th percentile) and third quartile (75th percentile)
IQR = Q3 - Q1 where Q1 and Q3 are the 25th and 75th percentiles
The IQR is the length of the box in a box plot.
Fill in the blank: The three commonly used measures of center are mean, median, and _______.
Mode
Mode is essential in understanding the frequency of data points.
True or False: The range is defined as the maximum value minus the minimum value.
True
This measure provides a quick sense of the spread of the data.
What are the three commonly used measures of center?
- mean (the arithmetic average)
- median (the midpoint)
- mode (the most frequent observation)
What are the three commonly used measures of spread?
- standard deviation (variability around the mean)
- range (max-min)
- interquartile range (middle 50% of the distribution)
Which of the following cannot be determined from a boxplot?
Box plots do not display modality, histograms do.
modality: whether the distribution is unimodal, bimodal, uniform, etc.
What are box plots and dot plots used for?
To highlight outliers and display the median and interquartile range
What is a common tool for visualizing the relationship between two numerical variables?
Scatter plot
The primary purpose of a scatter plot is to visualize the relationship between two numerical variables
A scatterplot provides a case-by-case view of data for two numerical variables.
In a scatter plot, the explanatory variable is placed in the x-axis, with the response variable in the y-axis
What can we only talk about when using observational data?
Correlation, not causation.
What does a strong relationship in a scatter plot indicate?
Little scatter around the curve.
What is a naive approach to handling outliers in data analysis?
Immediately excluding them.
What is the purpose of histograms in data visualization?
Histogram
An histogram is a good way to visualize the distribution of a single numerical variable
In an Histogram, the height of the bers represent the number of cases that fall into each interval.
What is a dot plot used for?
Visualizing individual values.
What does a box plot display?
Abox plot summarizes a data set using five statistics while also plotting unusual observations.
In a box plot display, the median and the interquartile range of the data are strongly displayed.
What does an intensity map reveal?
Spatial distribution trends in the data.
When we encounter geographic data, we should create an intensity map, where colors are used to show higher and lower values of a variable.
The intensity maps are not generally very helpful for getting precise values in any given area, but they are very helpful for seeing geographic trends and generating interesting research questions or hypotheses.
What is the meaning of skewness of a distribution?
Skewness refers to the direction of the tail in a distribution (left or right)
- left skewed
- symmetric
- right skewed
Distibutions are skewed to the side of the long tail
What is modality?
Modality is an important aspect of shape to describe a distribution.
Modality refers to the number of peaks in a distribution.
A distribution might be unimodal with one prominent peak, bimodal with two prominent peaks, or uniform with no prominent peaks. With more than two prominent peaks a distribution is usually said to be multimodal.
Definition
Sample statistics
Sample statistics are point estimates for the unknown population parameters.
They are measurements calculated from a sample that is representative of the total population.
Compare the mean and median according to the skewness of a distributiom
left skewed: mean < median
symmetric: mean | median
right skewed: mean > median
Variance
Variance is roughly the average squared deviation from the mean
Standard deviation
Standard deviation is the square root of the variance, and standard deviation has the same units as the data.
The standard deviation represents the typical deviation of observations from the mean.
Variability vs Diversity
Distributions where more observations are clustered around the center, are less variable, versus distributions where more observations are away from the center, are more variable.
A set with more data at the ends of the distribution (away from the center) is more variable. Diversity in a set refers to the number of different discrete values, not how far or close to the center of a distribution these values are.
True or False
In a left skewed distribution the median tends to be greater than the mean
True
The statistic mean/median (mean divided by median) can be used as a measure of skewness (either right or left)
What is an outlier in a data distribution?
An outlier is an observation that appears extreme relative to the rest of the data.
Examining data for outliers serves many useful purposes, including
1. Identifying strong skew in the distribution.
2. Identifying possible data collection or data entry errors.
3. Providing insight into interesting properties of the data.
What is the purpose of whishers in a box plot?
Extending out from the box, the whiskers attempt to capture the data outside of the box.
However, their reach is never allowed to be more than 1.5 × IQR. They capture everything within this reach.
In a sense, the box is like the body of the box plot and the whiskers are like its arms trying to reach the rest of the data.
True or False
Outliers are represented as dots beyond the whiskers in a box plot
True
Any observation lying beyond the whiskers is labeled with a dot.
The purpose of labeling these points– instead of extending the whiskers to the minimum and maximum observed values– is to help identify any observations that appear to be unusually distant from the rest of the data.
robust statistics
We define robust statistics as measures on which extreme observations have little effect.
The median is a more robust statistic of center than the mean. The IQR, which is based on the median, is a more robust statistic than the standard deviation which is calculated using the mean.
Robust statistics are most useful for describing skewed distributions, or those with extreme observations. While non-robust statistics like mean and standard deviation are useful for describing symmetric distributions
What is a transformation of data?
A transformation is a rescaling of the data using a function. When data are very strongly skewed, we sometimes transform them, so that they are easier to model.
We might want to reduce skew to assist in modeling or we might want to straighten a nonlinear relationship in a scatterplot, so that we can model the relationship with simpler methods.
What is the most commonly used transformation?
The most commonly used transformation is the natural log transformation, which is often applied when much of the data cluster near zero relative to larger values in the dataset and all observations are positive.
Hypothesis testing framework
1) We start with a null hypothesis that represents that status quo.
2) We also have an alternative hypothesis that represents our research question, in other words, what we’re testing for.
3) We conduct a hypothesis test under the assumption that the null hypothesis is true, either via simulation or using theoretical methods.
A: If the test results suggest that the data do not provide convincing evidence for the alternative hypothesis, we stick with the null hypothesis.
B:If they do, then we reject the null hypothesis in favor of the alternative.
What is a contingency table?
A contingency table is a type of table used in statistics to display the frequency distribution of two categorical variables. It helps to summarize the relationship between these variables by showing how the categories of one variable relate to the categories of another.
What is the purpose of a Contingency Table?
Purpose of a Contingency Table:
Relationship Evaluation: It allows researchers to evaluate whether there is an association or relationship between the two categorical variables.
Conditional Distribution: It helps in calculating conditional distributions, which show the distribution of one variable given the levels of the other variable.
Data Visualization: It provides a clear visual representation of the data, making it easier to identify patterns or trends.
Statistical Analysis: It serves as a basis for various statistical tests, such as the Chi-square test, to determine if the observed frequencies differ significantly from expected frequencies.
What is a graphical way to represent a single categorical variable?
A graphical way of representing data with a single categorical variable is a bar plot.
Also, we usually consider the relative frequencies when evaluating the distributions of categorical variables. We can also make a bar plot of these relative frequencies, which look just like the original bar plot but just has the relative frequencies instead of the counts on the y-axis.
How are bar plots different than histograms?
First, bar plots are used for displaying distributions of categorical variables, while histograms are used for numerical variables.
Second, the axis in a histogram is a number line. Hence, the orders of the bars cannot be changed. While in a bar plot, the categories can be listed in any order, though some orderings make more sense than others, especially for original variables.
What type of plot is useful to visualize conditional frequency distributions of categorical data?
A segmented bar plot is useful for visualizing conditional frequency distributions of categorical variables.
In other words, the distribution of the levels of one variable, the response variable, conditioned on the levels of the other, the explanatory variable.
True or False
A mosaic plot can display both the marginal distribution and the conditional distribution of categorical variables.
True
Mosaic plots are useful only for categorical variables.
p-value
The probability of observing data under the assumption that the null hypothesis is true, is called the p-value.
True or False
In a right skewed distribution the median tends to be greater than the mean
False
What is suggested of a contingency table when it shows that there are considerable differences between proportions in their categories?
It suggests a relationship between categorical variables.
35/290 ≈ 12% of Republicans, 146/341 ≈ 43% of Democrats, and 69/341 ≈ 20% of Independents think that teachers belonging to unions or bargaining associations helped the quality of public school education in the United States. Since there is considerable differences between these proportions, the results of the survey suggest a relationship between opinion on teachers belonging to unions or bargaining associations and political party affiliation.
