Pandas Primer

Question 1

How do you read a comma-delimited file in pandas?

Answer 1

df = pd.read_csv(filepath)

Question 2

How do you create a dataframe in pandas? (2)

Answer 2

• pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns = [‘col1’, ‘col2’, ‘col3’])
• With a dictionary:
  
  • df = pd.DataFrame({“ID” : [1, 2, 3], “First Name” : [“John”, “Jim”, “Joe”], “Last Name” : [“Smith”, “Hendry”, “Wilson”]})

Question 3

How do you show a dataframe in pandas?

Answer 3

display(df)

Question 4

How do you access a cell in a dataframe?

Answer 4

• .loc[‘cobra’]
• .iloc[row_or_col_index]
• .loc[row_label,col_label]
• .iloc[row_index,col_index]

Question 5

What selection features does pandas support? (2)

Answer 5

• Slicing in .loc and .iloc with start_index:end_index
• array indexing

Question 6

What’s the difference between .loc and .iloc?

Answer 6

• loc is label-based, -> use row and column labels.
• iloc is integer position-based -> use integer position values

Question 7

how do you show the first few rows of a dataframe?

Answer 7

df.head(some_num_rows)

Question 8

What’s one interesting thing about pandas slicing?

Answer 8

• When you want an entire row or column, instead of including a “:”, can omit the row or column index entirely (along with the comma) by not using a loc or iloc function at all
• E.g.
  
  • display(df.loc[:, “Last Name”]) is equivalent to display(df[“Last Name”])

Question 9

How do you set an entry in a DataFrame?

Answer 9

df.loc[1,”Last Name”] = “some_val”

Question 10

How do you set a row in pandas?

Answer 10

df.loc[3,:] = (100, “Andrew”, “Moore”)

Question 11

What happens if you try to set a row and the input index doesn’t exist?

Answer 11

new row is appeneded to the end

Question 12

How do we select a subset of the rows that satisfy some conditions from a dataframe?

Answer 12

df[(df[“First Name”] == “Jim”) & (df[“Last Name”] == “Kilter”)]

Question 13

Given this dataframe, how do you find rows where Last Name has 6 characters?

Answer 13

df[df[“Last Name”].str.len() == 6]

Question 14

Given this dataframe, how do you find rows where First Name contains the substring “Jo”?

Answer 14

df[df[“First Name”].str.contains(“Jo”)]

Question 15

Given this dataframe, how do you find rows where First Name is either “Jim” or “Kim”?

Answer 15

df[df[“First Name”].isin([“Jim”, “Kim”])]

Question 16

What do you do if you want to find rows that do not satisfy a certain condition?

Answer 16

Use the negation symbol ~

E.g. df[~df[“First Name”].isin([“Jim”, “Kim”])]

Question 17

• What’s one trick we can use to speed up the selecting of pandas rows?
• What does it do?

Answer 17

• Use a query string to select rows, which
  
  • df.query(‘(First Name == “John”) & (Last Name == “Smith”)’)
• can avoid the creation of the intermediate boolean index and reduce runtime / memory usage:

Question 18

What’s important to remember about querying? (3)

Answer 18

1. The returned object of a query is a view of the original data frame.
2. Modifying the view will not affect the original data frame, but will yield a warning.
3. Unlike Numpy, Pandas preserves the original row index after filtering. E.g. if you make a copy of a slice that has rows 2 and 5 and try to select index 1 from that, an error will be thrown.

Question 19

How do you copy a dataframe?

Answer 19

.copy()

Question 20

If our dataframe has no row with index 0, what do we do?

Answer 20

Call .reset_index(drop = True)

E.g. df_copy_reset_index = df_copy.reset_index(drop = True)

Question 21

How can we iterate over rows of a dataframe, from slowest to fastest?

Answer 21

• Use .iloc along with row index.
• Use iterrows method.
• Use apply with axis=1.
• Fastest: Use Pandas vectorization

Question 22

How can we iterate over columns of a dataframe? (iteration syntax and how you index the column)

Answer 22

1. Call .columns to get the list of column names and iterate over it
2. Use .iloc along with the column indexes

Question 23

What’s important to note about iterrows()?

Answer 23

• the row returned is only a copy of the data
• so you cannot update the data frame during .iterrows()

Question 24

What is the default of the apply method?

Answer 24

• To loop through the columns (same as axis=0)
• To loop through rows, use axis=1

Question 25

• What can be used with pandas vectorization?

Answer 25

1. built-in pd.Series methods
2. operations that are compatible with Numpy arrays, for example basic math operations or Boolean conditons

Question 26

• What is the idea of pandas vectorization?

Answer 26

operating a procedure on the entire column array at once, instead of on individual column elements

Question 27

If we can’t vectorize an operation we want to apply to all rows, what should we do? Why?

Answer 27

1. Use the apply method
2. Why?
   
   • It’s the 2nd fastest row iteration approach of the 4
   • It can work with any input function

Question 28

If we have a vectorizable operation we want to apply over a set of rows, and the columns are numerical, what should we do?

Answer 28

apply vectorization on the underlying Numpy arrays (by calling df[column_name].to_numpy()) for an even greater speedup.

Question 29

What’s One primary advantage of Pandas over other data table packages?

Answer 29

its powerful data manipulation functions

Question 30

What’s important to remember about most pandas dataframe methods?

Answer 30

most do not modify the input dataframe and only return the output in a new dataframe

Question 31

How do you modify a pandas dataframe in-place? (2)

Answer 31

Either:

1. Look for the parameter inplace in the method API
2. Or reassign your dataframe to the method output. (e.g. df = df.fillna(0))

Question 32

How do you replace missing entries in a pandas dataframe?

Answer 32

fillna(new_val, inplace = True)

By default inplace = False I think

Question 33

• What are the main forms of input data? (2)
• Describe them.
• What are the tradeoffs?
• When should they be used?

Answer 33

• Wide
  
  • every row represents a unique observation and every column represents a feature (e.g. rows are distinct countries and the columns are relevant attributes of each country)
• Long
  
  • there is one column for the observation ID, one column for attribute name, and one for attribute value. (e.g. screenshot)
• Tradeoffs
  
  • Long is often easier to implement, as addition of a new feature does not change the table structure
  • Long is harder to understand.
• When they should be used:
  
  • The long format is useful when you are curating data and do not yet know what the final structure will be. When your data is ready for analysis, the wide format is preferred.

Question 34

How do we convert from wide to long?

Answer 34

Use .melt with four parameters:

• id_vars: names of the columns with the observation IDs
• value_vars: names of the feature columns
• var_name: name of the new column that will contain the feature names
• value_name: name of the new column that will contain the feature values.

E.g. df_wide.melt(id_vars = [“country”], value_vars = [“population_in_million”, “gdp_percapita”], var_name = “attribute”, value_name = “value”)

Question 35

How do we convert from long to wide?

Answer 35

Use .pivot_table with three parameters:

• index: name of the column with the ids.
• columns: name of the column that contains the feature names.
• values: name of the column that contains the feature values.

E.g. df_long.pivot_table(index = “country”, columns = “attribute”, values = “value”)

Question 36

How do you partition a dataframe into groups based on the values at some columns?

Answer 36

groupby()

E.g. in screenshot

Question 37

• What’s an aggregation function?
• What are some examples? (4)

Answer 37

• takes as input all the rows in a group and outputs one value
• Examples
  
  • .count()
  • .max()
  • .min()
  • .sum()

Question 38

What is .count()?

Answer 38

Count non-NA cells for each column or row.

Question 39

What can be used with .groupby?

Answer 39

.count() chained on at the end

E.g. df.groupby(“state”).count()

Question 40

How do we apply multiple aggregation functions to each group?

Answer 40

• Call .agg, which takes a mapping from column name to aggregation functions
• E.g. df.groupby(“state”).agg({“city” : “count”, “population” : [“sum”, “max”]})

Question 41

What’s an important technique in manipulating pandas dataframes? Describe it.

Answer 41

split-apply-combine pattern

• Splitting the data into groups based on some criteria
  
  • done via .groupby() itself
• Applying a function to each group independently
  
  • done by calling apply() along with the specified input functions (which can be aggregation, transformation, filtration or a combination of them).
• Combining the results into a data structure
  
  • done automatically on the returned values of apply.

Question 42

How do you define a series in pandas? Use 3 args

Answer 42

pandas. Series(data=None, index=None, dtype=None)
* data
* array-like, Iterable, dict, or scalar value

Contains data stored in Series. If data is a dict, argument order is maintained.

• index
  
  • array-like or Index (1d)

Values must be hashable and have the same length as data. Non-unique index values are allowed. Will default to RangeIndex (0, 1, 2, …, n) if not provided. If data is dict-like and index is None, then the keys in the data are used as the index. If the index is not None, the resulting Series is reindexed with the index values.

• dtype
  
  • str, numpy.dtype, or ExtensionDtype, optional

Data type for the output Series. If not specified, this will be inferred from data. See the user guide for more usages.

Question 43

What should we do if we have a row with only NAN values?

Answer 43

remove it with a call to dropna()

Question 44

What happens to a row if the operation in .appy() doesn’t return a value for that row?

Answer 44

All the values will be NaN

Question 45

What’s a key difference between .agg and .apply?

Answer 45

• .agg() can only aggregate data in each column separately
• . apply is a more general version of agg that can handle multi-column operations while also performing filtration

Question 46

What’s important to know about groupby?

Answer 46

• check that the columns which you perform groupby on have no empty values (np.nan). These will be ignored during groupby, resulting in potential loss of data.

Question 47

• How do you concatenate a df along rows?
• What about columns?

Answer 47

• pd.concat([df1, df2])
• pd.concat([df1, df3], axis = 1)

Question 48

What does it mean to concatenate along?

Answer 48

• Concat Along rows -> adds more rows
• Concat Along columns -> adds more columns

Question 49

What are the joins supported by pandas (4) and their functions?

Answer 49

• Left join keeps all rows of the left table, add entries from right table that match the corresponding columns.
• Right join is the like a left join but with the roles of the tables reversed.
• Outer join returns all rows from both left and right join.
• Inner join return the rows where the two joined columns contain the same value

Question 50

How do you join two dataframes in pandas?

Answer 50

• Usually use the merge method.
• if dfs are indexed, use the join() method because it’s faster
• df1.merge(df2, left_on = “col1”, right_on = “col1”, how = “left”)
• If the column to merge on is an index, we need to use “left_index = True” ( same for right_index vs right_on).

Question 51

How do you set the index in pandas?

Answer 51

df.set_index(‘some_col’, inplace=True)

Question 52

What do we know about the .join() method? (2)

Answer 52

• Only supports joining by indexes.
• It’s faster than the merge method because joining by indexes is faster than by column names

Question 53

Describe the groupby syntax

Answer 53

groupby(by_label_or_list_of_labels)[about_label_or_list_of_labels].aggregate_function()

Question 54

• When would you get a multi-index DF from groupby?
• How can you make that not happen?

Answer 54

• 1 situation: passing in multiple groups. (I.e. when you pass a list of column labels to groupby)
• pass as_index=False to groupby (see screenshot)

Question 55

• When is the cut method used? (2)
• What’s an example of useful usage for us?
• What’s the syntax?

Answer 55

• Useful when you need to either:
  
  • Segment and sort data values into bins.
  • Go from a continuous variable to a categorical variable.
• could convert ages to groups of age ranges, then groupby that new column
• df[‘new_col_name’] = pandas.cut(df[‘some_column’], bins=num_bins, labels=list_of_bin_labels)

Question 56

What should you do if you have too many groups in your groupby?

Answer 56

• Use the cut method to make a new column of discrete bins
• groupby the new bins