Data Scientist's Toolbox

Question 1

Command Line Interface

Answer 1

A window in a computer where you can enter command lines and navigate the computer file structure by text and commands; git bash on Windows, and Terminal on Mac

Question 2

Working Directory

Answer 2

Whatever directory you are currently in when using the Command Line Interface.

Question 3

Path

Answer 3

All of the directories you need to navigate through to get back to your root directory from the directory you;re in now

Question 4

Home Directory

Answer 4

The directory the command line interface opens into on your machine

Question 5

Root Directory

Answer 5

The highest up the directory structure you can go. Nothing larger than this in your computer

Question 6

pwd

Answer 6

A command that shows the path to the working directory you’re currently in. Stands for “print working directory”

Question 7

The CLI Command recipe

Answer 7

command flags arguments

“command” is the CLI command that does a certain task

“flags” are options given to the command to trigger certain behaviours, proceeded by a “-“

“arguments” can be what the command is going to mod, or other options - not every command has these

Eg. pwd is a command with no flags or arguments

Question 8

CLI Command: Clear

Answer 8

Clears up everything on the screen back to the home directory

Question 9

CLI Command: ls

ls -a?

ls - al?

Answer 9

“ls” lists files and folders in the current directory

“ls -a” lists hidden and unhidden files and folders

“ls -l” lists details for files and folders in the current directory

“ls - al” lists details for hidden AND unhidden files and folders

-a and -l are flags

Question 10

How to Id a hidden file or folder in the CLI

Answer 10

It will start with a “.” and only be visible with certain commands

Question 11

CLI Command: cd

Answer 11

“cd” stands for “change directory”

“cd” accepts as an argument the directory you want to change to

“cd” with no argument goes back to the home directory

“cd ..” goes up one level in the directory path

Question 12

What does “/” represent in the CLI?

Answer 12

The Root Directory

Question 13

What does “ ~ “ represent in the CLI?

Answer 13

The Home directory

Question 14

CLI Command: mkdir

Answer 14

“mkdir” stands for “make directory”

same as making a new folder in the GUI

mkdir accepts as an argument the name of the directory being created

Question 15

CLI Command: touch

Answer 15

“touch” creates an empty file

“touch” accepts as an argument the name of the file being created

Question 16

CLI Command: cp

Answer 16

“cp” stands for “copy”

“cp” takes as a first argument a file, and as a second argument (separated by a space) the name of the path to where the file should be copied

the “-r” (recursive) flag can also be used to indicate the contents of a directory should be copied to the new directory

Question 17

CLI Command: rm

Answer 17

“rm” stands for “remove”

“rm” takes as an argument the name of a file you want to remove

“rm” can be modded with the flag “-r” and a directory as an argument to remove an entire directory (Note: this cannot be undone”

Question 18

CLI Command: mv

Answer 18

“mv” stands for “move” and can be used to move a file from one directory to another

it accepts as an argument the name of the file to be moved

“mv” can also be used to rename files by using the desired filename as the second argument

Question 19

CLI Command: echo

Answer 19

“echo” will print whatever arguments are provided after

Useful for printing out the contents of variables that have been stored

Question 20

CLI command: date

Answer 20

“date” will print out the date

Question 21

Adding to an index

Answer 21

“git add .” adds all new files

“git add -u” updates tracking for files that were renamed or deleted

“git add -a” does both of the previous

• should be done before committing

Question 22

Committing to a local repository

Answer 22

“git commit -m “message” “ where “message” is a useful description of the work that was done

Note: this only updates the local repo, not the remote one

Question 23

Pushing to Github

Answer 23

“git push” takes all changes made since last push and send them to github

Question 24

Branch Commands

Answer 24

”- git checkout -b branchname” create a new branch named what is put in branch name

”- git branch” see what branch you are on

”- git checkout master” switch to the master branch

Question 25

Pull Request

Answer 25

• a unique feature of github
• if you fork someone’s repo or have separate branches, you may need to merge; this is what a pull request does, and it’s done from github

Question 26

How install R packages? Multiple packages?

Answer 26

Type “install.packages(“packagename”)”
for multiple “install.packages(c(“packagename”, “packagename2”))”

Or use the “tools” menu in R Tools

then use “library(package name)” to load the package

Note: packages won’t load if dependent packages are not already loaded

then “search()” to see functions available using that package

Question 27

Types of Data Science Questions in order of difficulty

Answer 27

In order of difficulty:

Descriptive
Exploratory
Inferential
Causal
Mechanistic

Question 28

Descriptive Analysis: Goal? Example?

Answer 28

Goal: Describe a set of Data

• commonly applied to census data
• description and interpretation are different steps
• descriptions usually can’t be generalized without additional modelling

Eg. www.census.gov/2010census
books.google.com/ngrams

Question 29

Exploratory Analysis: Goal? Example(s)?

Answer 29

Goal - find relationships you didn’t know about

• good for discovering new conniptions
• useful for defining future studies
• exploratory analysis usually not the final say
• should not be used for generalizing or predicting without further analysis
• correlation does not imply causation

Eg. sdss.org - sloan digital sky survey

Question 30

Inferential Analysis: Goal? Example(s)?

Answer 30

Goal: use a relatively small sample of data to say something about a bigger population

• commonly the goal of statistical models
• involves estimating both the quantity cared about and the uncertainty of the estimate
• depends on population and sampling scheme

Eg. Effect of Air Pollution Control on Life Expectancy in the United States: An analysis of 545 US Counties for the period from 2000 to 2007

Question 31

Predictive Analysis: Goal? Example(s)?

Answer 31

Goal: To use the data on some objects to predict values for another object

• If X predicts Y, it does not mean that X causes Y
• Accurate prediction depends heavily on measuring the right variables
• more data and simple model tends to work really well
• prediction is very hard

Example: 538 blog - Nate Silver, Predicting the presidential election; target seeing purchases someone has made and determining they’re pregnant

Question 32

Causal Analysis: Goal? Example(s)?

Answer 32

Goal: Find out what happens to one variable when another variable changes

• Usually randomized studies are required to ID causation
• some approaches to inferring causation in non-randomized, but they’re finicky
• causal relationships are usually IDed as “average effects” but may not apply to every individual in the population
• causal models are the “gold standard” for data analysis

Eg. Medical studies for new processes or drugs

Question 33

Mechanistic Analysis: Goal? Example(s)?

Answer 33

Goal: Understand the exact changes in variables that lead to change in other variables for individual objects

• incredibly hard to infer, except in simple situations
• usually modelled by a deterministic set of equations (physical/engineering science) where all variables can be carefully controlled
• generally the random component of the data is measurement error
• if the equations are known but the parameters are not, they may be inferred by data analysis

Example: Pavement design - what changes lead to different outcomes in function? www.fhwa.dot.gov/resourcecenter/teams/pavement/pave_3pdg.pdf

Question 34

Definition: Data

Answer 34

“Data are values of qualitative or quantitative variables, belonging to a set of items”

Set of items: Also known as the population; the set of objects you’re interested in

Qualitative: Country of origin, sex, treatment

Quantitative: Height, weight

Question 35

Variability

Answer 35

Also called dispersion, scatter, or spread, is the distribution of measured variables in a plot. Commonly measured by variance, standard deviation, or interquartile range

Question 36

Confounding

Answer 36

When an extraneous variable correlates with the measured variables, creating the false appearance of correlation between those two variable. Eg. As shoe size goes up, so does literacy, but this is actually because as people become older (to a certain point) both literacy and shoe size increase

Question 37

Spurious Correlation

Answer 37

When tow variables are correlated but the correlation is almost entirely due to some other, far more important, variable

Question 38

Values of good experiments

Answer 38

• Can be replicated
• Measure variability
• Generalize to the problem
• Are transparent

Question 39

Data dredging

Answer 39

• data dredging is when many different hypotheses are applied to a large data set trying to find some correlation; eventually by playing with the data enough some correlation will be found but there is a good chance it can be false