Data Analysis Pipeline

Question 1

What are the 5 steps that are necessary once you collect some data?

Answer 1

1. Figure out the question.
2. Find/​acquire relevant data.
3. Clean & prepare the data.
4. Analyze the data.
5. Interpret & present results.

Question 2

It’s not unusual to spend most of your time _______ ______ when processing data

Answer 2

cleaning data

Question 3

What are some different ways to get data for the question your attempting to answer?

Answer 3

• Files (CSV, Excel, XML, etc)
• API
• DB
• A sensor

Question 4

When first retrieving data, it may have some inconsistencies that need to be cleaned such as:

• Irrelevant things
• Different ____ for similar values
• Different _______ in files
• _____ wrong

Answer 4

units
formats
shaped

Question 5

When first retrieving data, it may have straight up incorrect values. What are some causes of this?

Answer 5

• Missing values (failed sensor, incomplete collection, etc)
• Outliers (data entry errors, etc)
• Noise

Question 6

The full data analysis or pipeline is often broken into steps. What are some reasons for this?

Answer 6

• Don’t want to spam an API every time we process data
• Test runs might take too long
• Intermediate results might be meaningful

Question 7

The full data pipeline is not always obvious. In the end you may need to run ______ programs. You should always _______ your code so you know how things should be done

Answer 7

multiple, document

Question 8

What are some reasons you might have manual steps in your pipeline?

Answer 8

• Easier to do by hand than automating

- Most cases can be automatically determined, but some outliers could be left for manual intervention