DSE Flashcards
What is noise?
Something that may make the data unreliable. These are usually known as confounding variables.
These stop you from getting accurate data; thus, everything you measure is a sample and not the ideal or final result.
However, you can cut as much noise as possible by repeating it in different conditions and using more random data to test.
This is called Random Variations to achieve a particular accuracy.
What are the 3 main components of summary statistics?
Central Tendency
Dispersion
Skewness and Kurtosis
What is Central Tendency?
Think “what is the centre of the distribution?” and modes, median and mean.
What is Dispersion?
Think “how widely spread is my distribution about the centre?” Clustered or spread out?
What is the Skewness
Think about where the mode of the data leans into the high or low side by the x-axis.
What are the problems with means?
Heavily influenced by large values. A single great number that is an outlier can influence the mean and make it unreliable.
It is not influenced by outliers.
What are the problems with the median?
It chooses the middle only, disregarding all other data.
What are the problems with the mode?
Some datasets are multimodal, meaning they have more than one mode.
It works best in discrete datasets and not continuous since continuous have more data.
Relationship between mean, median, and mode in a symmetrical dataset.
All in the middle. mean, mode, and median.
Relation between mean, median, and mode in a dataset skewed to the left (positive).
Mode, Median Mean
Relation between mean, median, and mode in a dataset skewed to the right (negative).
Mean, Median, Mode.
What do you have to do to find the Standard Error?
std / sqrt(number of samples)
What are Quartiles?
Three values, Q1, Q2 and Q3.
What is Q1?
First, lower, first 25% percentile of value.
It splits 25% of data from the highest 75%
What is Q2?
Median. Cuts in Half
What is Q3?
third, the last 25% percentile of value.
It splits off the highest 25% of data from the lowest 75%.
What is IQR?
Inter Quantile Range.
They are between Q1 and Q3.
What is kurtosis?
How heavy the “tails” are in the distribution. (fix this)
What are the 5 summaries of a dataset?
Min
Q1
Median
Q3
Max
What is standard deviation?
a measure of spread, how spread out or clustered the data is to the mean. They measure the consistency and uncertainty of the data.
Think of them as the average distance between the data point to the mean.
What does a small or large std mean?
A small std means the number is closer from the means
A large std means the number is further from the means.
Why should you use n-1 if you are researching a sample dataset?
The sample underestimates the spread or variance of the true population as sample may miss extreme values or outliers. Funny enough, you increase the std to compensate the extreme values.
Why do you use n-1 and not n+1?
Because we need the value to have a larger std to compensate the outlier, we need to minus n by 1.
In the std formula, n is a denominator, and the bigger the denominator, the smaller the answer and we need a large std to compensate the outlier.
Remember, outliers are spread away from the means, and a large std means they are spread away from the mean.
Why not -2 or xyz?
What if the dataset has 2 data? You would get a math error.
-2 would overestimate and overcorrect the outliers and overcompensate the biases.
It may result in a too large answer.
-1 is not the true value but the best estimate and goldilock solution of of the std.
What are the general principle of a data visualisation?
Keep it simple
No unnecessary decor, variations, colours or 3D imaging.
Colour appropriate for audience.
Better not to use 3D and use 2D instead.
What is a scatter plot used for?
Use for displaying a relationship between two or multiple variables sampled.
plot() or plot3() for 3d scatter plot.
use LineSpec to show the points clearly.
What is an error bar for?
Represent uncertainty or
variance in measured values.
May be standard deviation,
SEM, or a specified
confidence interval.
Can also be used on x-values
if there isa sampling variation.
errorbar()
What is a box plot?
The 5 summary of the data, min, q1 median, q3 max.
Show various summary statics for each points.
boxplot()
What are line plots?
y = mx + b
When x affects y.
If you have more than one y value per x value, scatter plot is useful.
Otherwise if y value per x value is just 1, use line plots.
X is often time,mass, length.
plot()
What is a bar chart?
Good for categorical data.
Can be vertical or horizontal; vertical is easier, but horizontal is useful for longer names.
bar() for vertical
barh() for horizontal
bar3() for 3d visualisation if needed.
can be use for accumulation.
What is a histogram?
Distribution of numerical data, divide data into bins.
histogram()
matlab will choose a reasonable size however, you can vary it.
Don’t make it too wide or too narrow as it can skip important details or look like a mess.
to edit BinWidth, do
histogram(‘data’ , <number>)</number>
Density plot
Useful for physical and geographical data eg rainfall maps.
Choose sensible colour scale
pcolor() or contour() or surf().
How to get all unique data
Variable = unique(array) in matlab
Vector plot
plot vector quanitities over area such as wind speed and direction, magnetic fields or fluid flow.
quiver()
3D visualisation
mesh() or surf()
Useful for 3 indepdent variables (x).
why convert data to mathematical representation?
To aid future visualisation trends, summarise relationships, and inter trends and make predictions, model or confirm physical law that predict behaviour.
What are the two common correlations of data?
Linear and Rank order
What is a linear correlation?
the name says itself, the data plot makes a certain linear pattern.?
To find it
corr(x,y, ‘Type’ , ‘Pearson’) or corr(x,y)
To skip NaN add “complete”
What is a rank order (aka spearmen)?
When y always goes up when x goes up but not on the same amount. You call this monotonic and it will look like a sine wave but always increasing.
It is not monotonic when it goes up then down then up.
-y going down can make x go up.
corr(x,y, ‘Type’, ‘Spearman’)
To skip NaN add ‘complete’.
How to label figures?
plot(x,y) <this></this>
title(“<put>")4
xlabel("<put>");
ylabel("<put>");
grid(<put>);</put></put></put></put>
How to avoid overfitting?
Use your judgement, test and validate test sets.
Give it noise and retry to see how much it influences the function model.
If any parameter has negative, assume is 0.
What are some outlier ethical considerations?
DO NOT REMOVE THEM (unless it is a collection error). It is a serious ethical violation.
