Pandas

Question 1

PD: In normal Pandas style, what is a row and what is a column?

Answer 1

A row is an item or point, and a column is a field or feature.

Question 2

PD: Why do most pandas methods return a df rather than editing the original df in place?

Answer 2

It allows for method chaining, or running several methods in a nest, which improves readability of code.

Question 3

PD: Do most pandas methods edit a df in place, or return a new df?

Answer 3

Return a new df

Question 4

PD: How do you read a table or tables from a web page?

Answer 4

pd.read_html(‘sampleurl.com’,…)

Question 5

PD: How do you read a table from a .csv?

Answer 5

pd.read_csv(\path, sep=” “,…)

Question 6

PD: How to combine the rows of 2 dfs with the same column headers?

Answer 6

pd.concat([df1,df2])

Question 7

PD: How to view the first 5 rows?

Answer 7

df.head()

Question 8

PD: How to view the last 5 rows?

Answer 8

df.tail()

Question 9

PD: How do we remove columns that aren’t useful to us from our dataframe?

Answer 9

to_drop = [“all”,”col”,”names”,”to”,”drop”]

df.drop(columns=to_drop, inplace=True)

(could remove inplace and do a df=df.drop())

Question 10

PD: What are the two ways to exclude columns that aren’t useful to us

Answer 10

Drop them using df.drop(), or don’t import then from the csv using inputs to pd.read_csv()

Question 11

PD: If I go df[“a”], what data structure comes out?

Answer 11

A pandas series

Question 12

PD: How to I tell if column “A” in a dataframe contains all unique values?

Answer 12

df[“A”].is_unique

Question 13

PD: How do I set a new column, such as column “A”, to be the index of a dataframe

Answer 13

df = df.set_index[“A”]

If you want a unique index for row names, first check that the column contains unique values using df[“A”].is_unique

Question 14

PD: What is a convenient way to standardize a column containing multiple formats or data types?

Answer 14

Regular expressions, or regex.

See the Pythonic Data Cleaning article for an example: https://realpython.com/python-data-cleaning-numpy-pandas/

Question 15

PD: How to find a series containing booleans for whether each value in column “A” is null or not in a dataframe?

Answer 15

df[“A”].isnull()

Question 16

PD: How to find the total number of nulls in column “A” in a dataframe?

Answer 16

df[“A”].isnull().sum()

Question 17

PD: How to rename cols in a dataframe?

Answer 17

df.rename(columns=listOfNewColNames, inplace=True)

Question 18

PD: What does it mean to do a split-apply-combine in pandas?

Answer 18

Say the rows of our dataframe fall into a few different “categories,” and we’d like to find a summary statistic or statistics within each of the categories. We can separate all of the categories into groups (split), then apply one or more summary functions to each group (apply), and lastly combine the summary statistics for each group into a new data frame where each row is now one of the categories (combine).

Question 19

PD: How can we, in one line of code, do all 3 parts of split-apply-combine?

Answer 19

df.groupby(“colNameToGroupBy”)[“colNameToFindSummaryStatsOn”] .agg([“list”,”of”,”summary”,”stat”,”functions”])

Video for reference: https://www.youtube.com/watch?v=qy0fDqoMJx8

Question 20

PD: How to sort the rows of a data frame based on column “A”, going from low to high

Answer 20

df.sort_values(“A”) (ascending, or low to high, is default)

Question 21

PD: How to sort the rows of a data frame based on column “A”, going from high to low

Answer 21

df.sort_values(“A”, ascending=False)

Question 22

PD: How to reset indices of DataFrame to row numbers, moving the index to a new column.

Answer 22

df.reset_index()

Question 23

PD: How do we display the data types of each column in our df?

Answer 23

df.dtypes

Question 24

PD: How to sample a subset of rows?

Answer 24

df.sample()

(See details for info on syntax and replacement
)

Question 25

PD: How to extract specific rows meeting a logical criterion?

Answer 25

df[(array of logical bools)]

For specifics and examples, see cheat sheet

Question 26

PD: How to get n rows with the largest values of a specific column?

Answer 26

df.nlargest(n,”nameOfColumn”)

Question 27

PD: How to get n rows with the smallest values of a specific column?

Answer 27

df.nsmallest(n,”nameOfColumn”)

Question 28

PD: What are the 2 ways to get a column from a df, given that the column name has no spaces or special characters

Answer 28

df[“colName”] OR df.colName

Question 29

PD: How to select multiple specific columns from a dataframe?

Answer 29

df[ [“colName1” ,”colName2” ,”colName3”] ]

Question 30

PD: How, at a high level, can we pick several columns each matching specific criteria?

Answer 30

Regular expressions. (For how, see cheat sheet and also study regexes)

Question 31

PD: What is the high-level difference between loc and iloc?

Answer 31

Loc takes a subset of rows, columns or both using labels.

iloc takes a subset of rows, columns or both using positions.

Question 32

PD: Would iloc more commonly be used for subsetting rows or columns? Why?

Answer 32

Rows. iloc subsets by position, which would be more common for rows, as we might want rows 10 through 20, for example.

(But it’s worth noting that head() and tail() can also do a lot of this functionality.)

Question 33

PD: Would loc more commonly be used for subsetting rows or columns? Why?

Answer 33

Columns. Loc subsets by label, which would be more common for columns, which typically have column names as labels.

Question 34

PD: How to find the number of unique values in columnA of a df?

Answer 34

df[“A”].nunique()

Question 35

PD: How to display the number of appearances for each unique value in column A of a df?

Answer 35

df[“A”].value_counts()

Question 36

PD: How to display the dimensions of a dataframe?

Answer 36

df.shape

Question 37

PD: How to display the number of rows in a dataframe without using df.shape?

Answer 37

len(df)

Question 38

PD: How to get basic summary statistics for each column in a df?

Answer 38

df.describe()

Question 39

PD: How to get the mean, or max, or sum, of column A of a df? Works for lots of summary stats?

Answer 39

df[“A”].mean() or df[“A”].max() or df[“A”].sum()

Question 40

PD: How to drop all rows where any column has NAs or null data?

Answer 40

df.dropna()

Question 41

PD: How to fill in all NAs with a value?

Answer 41

df.fillna(value)

Question 42

PD: How do we group our df by column A, and then find the mean value of column B within each group?

Answer 42

df.groupby(“A”)[“B”].mean()

(So you can use the [“B”] notation, or .B notation too, to extract that column for all groups, and then an aggregation function like mean() to run the aggregation for all groups. This makes the syntax very similar to if you were finding the mean of B in a single df rather than a groupby object)

Question 43

PD: How to get the size of each group in groupby object g?

Answer 43

g.size()

Question 44

PD: How to make a new column C in df that is the product of columns A and B?

Answer 44

df[“C”] = df[“A”] *df[“B”]

Question 45

PD: At a high level, how to descretize a column into several equal-sized buckets based on quantiles?

Answer 45

qcut()

Question 46

PD: Given 2 data frames adf and bdf with common column A, what is the general syntax to do a join on column A, be it a left outer, right outer, inner or full outer join?

Answer 46

pd.merge(adf, bdf, how = “type”, on=”A”)

where type is “left”, “right”, “inner” or “outer”, corresponding to the 4 join types we learned in SQL.

Question 47

PD: Given 2 data frames adf and bdf with common column A, how do we find all rows in adf with a value in column A that is also present in bdf?

Answer 47

adf[adf.A.isin(bdf.A)]

Question 48

PD: Given 2 data frames adf and bdf with common column A, how do we find all rows in adf with a value in column A that is NOT present in bdf?

Answer 48

adf[~adf.A.isin(bdf.A)]

Question 49

PD: How does pandas store strings?

Answer 49

Pandas uses the object dtype for storing strings, not the str dtype!