Pandas

Question 1

Create a Pandas dataframe with dictionaries

Answer 1

keys as column labels, values as lists of column data, then Df = pd.DataFrame(dict)

e.g.:
countries = {‘Name’: [‘UK’, ‘Germany’],
‘Capital’: [‘London’, ‘Berlin;]}

Countriesdf = pd.DataFrame(countries)

Question 2

set index labels for the rows of a pandas dataframe

Answer 2

pass a list of values to df.index, e.g.:

Countriesdf.index = [‘UK’, ‘GER’]

Question 3

import a csv file as a dataframe

Answer 3

Countriesdf = pd.read_csv(‘file/location/countries.csv’)

Question 4

make the first column of a dataframe the index of the csv (rather than a column in its own right)

Answer 4

Countriesdf = pd.read_csv(‘file/location/countries.csv’, index_col = 0)

Question 5

Get a Pandas Series from a Dataframe

Answer 5

df[‘column name’]

Question 6

Get a single column of the Pandas Dataframe

Answer 6

df[[‘column name’]]

Question 7

Get multiple columns from a Pandas Dataframe

Answer 7

df[[‘column name1’, ‘column name2’]]

Question 8

Get the 2nd - 4th (inclusive) rows of a Pandas Dataframe

Answer 8

df[1:5]

Question 9

call a row of a Pandas Dataframe with loc, as a Pandas Series

Answer 9

df.loc[‘row index’]

Question 10

Call 2 rows of a Pandas Dataframe with loc, as a dataframe

Answer 10

df.loc[[‘row index’, ‘row index 2’]]

Question 11

call the intersection of 2 rows of a pandas dataframe by name

Answer 11

df[[‘column name1’, ‘column name 2’], [‘row index 1’, ‘row index 2’]]

Question 12

get a whole column of a dataframe by name

Answer 12

df[ : [‘column name1’, ‘column name 2’]]

Question 13

Let’s say you have numpy arrays of x = [1,2,3,4,5] and y =[5,4,3,2,1]. How you use operators with them to get an array of bools corresponding to x/y>2 or y-x = 2?

Answer 13

np.logical_or(x/y>2 , y-x ==2)

(this gives the array [False, True, False, False, True], just out of interest.

Question 14

create a numpy array with a list of numbers 1,2,3,4,5

Answer 14

np.array([1,2,3,4,5])

Question 15

Let’s say you have a dataframe with a list of days and the total minutes of meditation done on those days. What are the two steps to create a list of days on which more than one hour of meditation was done?

Answer 15

1. select the meditation length column as a Pandas Series and do a comparison on that column:
   hourPlus = meditationDF[‘duration’] > 60
2. apply the results of that comparison to the dataframe:
   meditationDF[hourPlus]

Question 16

Let’s say you have a dataframe with a list of days and the total minutes of meditation done on those days. What are the two steps to create a list of days on which more than one hour of meditation was done BUT less than 2 hours was done?

Answer 16

bt1and2 = np.logical_and(meditationDF[‘duration’] > 60, meditationDF[‘duration’] < 120)

meditationDF[bt1and2]

Question 17

iterate over the rows of a dataframe and print out every row and label

Answer 17

for label, row in np.iterrows(dataframe):

print(label, ‘\n’, row)

Question 18

let’s say you’ve got a pandas dataframe with a bunch of columns, and one of them is ‘Day of week’. Iterate over the rows of the dataframe and print out only the value of that column for each row.

Answer 18

for label, row in np.iterrows(dataframe):

print(row[‘Day of week’])

Question 19

create a new column in a dataframe by applying a function to an existing column

Answer 19

dataframe[“column length”] = dataframe[‘existing column’].apply(len)

Question 20

create a new column in a dataframe by applying a method to an existing column

Answer 20

dataframe[“NEW COLUMN” = dataframe[‘existing column’].apply(str.upper)

Question 21

get some summary statistics for a dataframe

Answer 21

dataframe.describe()

Question 22

get all values a dataframe as a 2d numpy array

Answer 22

pd.DataFrame(df).to_numpy()

Question 23

get column names of a dataframe

Answer 23

dataframe.columns

Question 24

get the row numbers or names of a dataframe

Answer 24

dataframe.index

Question 25

get the dimensions of a dataframe

Answer 25

dataframe.shape

Question 26

find out if there are any missing entries in a dataframe

Answer 26

dataframe.info()

Question 27

sort a dataframe by value to get the largest values of “column1” at the top.

Answer 27

dataframe.sort_values(“column1”)

Question 28

sort a dataframe by values to get the smallest values of “column1” at the top

Answer 28

dataframe.sort_values(“column1”, ascending = False)

Question 29

sort by one value(column1), then another (column2)

Answer 29

dataframe.sort_values([“column1”, “column2”])

Question 30

get rows from a dataframe with a column1 value of greater than 5

Answer 30

overFive = dataframe[“column1”] > 5
dataframe[overFive]

This can also be done on one line:
dataframe[dataframe[‘column1’] > 5]

Question 31

Let’s say you’ve got a dataframe of fox species: ‘foxes’. Get all the rows with an ‘Ear goodness’ of above 5 and with ‘colour’ of ‘standard red’

Answer 31

goodEars = foxes[‘Ear goodness’] > 5
redOnes = foxes[‘colour’] == ‘standard red’
foxes[goodEars & redOnes]

Can also be done in one line with parentheses around each condition:

foxes[(foxes[‘Ear goodness’] >5) & (foxes[‘colour’] == ‘standard red’)]

Question 32

from your dataframe of foxes, get all the fox species that are either of the ‘colour’: ‘standard red’, or ‘black’

Answer 32

redOrBlack = foxes[‘colour’].isin([‘standard red’, ‘black’])

foxes[redOrBlack]

Question 33

Show the mean of a dataframe (meditation) column called “length in minutes”

Answer 33

meditation[“length in minutes”].mean()

Question 34

Show the standard deviation of a dataframe (meditation) column called “length in minutes”

Answer 34

meditation[“length in minutes”].std()

Question 35

Get the maximum value of a column called “length in minutes” from a dataframe called “meditation”

Answer 35

meditation[“length in minutes”].max()

Question 36

Get the minimum value of a column called “length in minutes” from a dataframe called “meditation”

Answer 36

meditation[“length in minutes”].min()

Question 37

create a function that gets the 30th percentile of a column called length in minutes in a dataset called meditation

Answer 37

this is the aggregate method - two steps:
step 1 - define a function to do what you want like this:
def perc30(column):
    return column.quantile(0.3)

step 2 -
get the column and use .agg() method to apply the function like this:
meditation[“length in minutes”].agg(perc30)

you can also pass lists of columns and lists of functions for multiple summary statistics.

Question 38

get the cumulative sum of a dataframe column

Answer 38

df[‘column’].cumsum()

Question 39

get rid of duplicates from a column in a dataframe

Answer 39

df.drop_duplicates(subset = ‘column’)

Question 40

get rid of duplicates from a dataframe - identify duplicates as matching on data from both col1 and col2

Answer 40

df.drop_duplicates(subset = [‘col1’, ‘col2’])

Question 41

get the number of each value in a dataframe column

Answer 41

df[‘column1’].value_counts()

Question 42

get the number of each value in a dataframe column, and sort them

Answer 42

df[‘column1’].value_counts(sort=True)

Question 43

get the proportion of each value in a dataframe column (e.g. 0.25 if instances of a given value constitutes a quarter of the total number of values)

Answer 43

df[column1].value_counts(normalize = True)

Question 44

Let’s say you’ve got a dataframe called meditation - get the mean ‘quality’ score for each ‘day of the week’.

Answer 44

meditation. groupby(‘day of the week’)[‘quality’].mean()
note: you can also use the agg() method in place of the mean() method here to apply custom functions, and multiple functions, like min() max() sum() etc.

Question 45

Let’s say you’ve got a dataframe called meditation - get the mean, minimum and maximum ‘quality’ scores grouped by each ‘day of the week’ and ‘retreat’ (yes/no)

Answer 45

meditation.groupby([‘day of the week’, ‘retreat’])[‘quality’].agg([min,max,np.mean])

Question 46

in a dateframe called meditation, use a pivot table to get the mean ‘quality’ for each value of the column ‘days of the week’

Answer 46

meditation.pivot_table(values= “quality”, index = “days of the week”)

Question 47

in a dateframe called meditation, use a pivot table to get the median ‘quality’ for each value of the column ‘days of the week’

Answer 47

meditation.pivot_table(values= “quality”, index = “days of the week”, aggfunc = np.median)

Question 48

with the dataframe meditation, create a pivot table to show the mean quality of meditation and standard deviation on each of the days of the week when it is and isn’t a retreat, with margins giving averages for all days, and for on and off retreat

Answer 48

meditation.pivot_table(values = ‘quality, index = [‘day of the week’, ‘retreat’], aggfunc = [np.mean, np.std], margins = True)

Question 49

Set the date column of your meditation dataframe as the index of the dataframe

Answer 49

meditation.set_index(“date”)

Question 50

reset the index of a dataframe to standard numbers

Answer 50

meditation.reset_index()

Question 51

create a hierarchical (or multi-level) index of your meditation dataframe using the columns “date” and “on_retreat”

Answer 51

meditation.set_index([“date”, “on_retreat”])

Question 52

let’s says you’ve got a meditation dataframe hierarchically indexed by 1. on_retreat and 2. amount_of_LSD. Sort the dataframe first by LSD, in ascending order, and then on_retreat, in descending order.

Answer 52

meditation.sort_index(levels=[“amount_of_LSD”, “on_retreat”], ascending = [True, False])

Question 53

slice a dataframe called Ollie that’s indexed by date (format dd/mm/yy) from 18/03/94 to 18/03/2019 (inclusive)

Answer 53

Ollie.loc[“18/03/94”: “18/03/19”]

Question 54

slice a dataframe called Ollie that’s been hierarchically indexed by both date (mm/yy) and day of week(mon, tues, etc.) from the first tuesday march 1994 to the last wednesday march 2019

Answer 54

Ollie.loc[(“03/94”, “Tues”), (“03/19”, “Weds”)]

pass a list of tuples with the outer and then inner indexes

Question 55

precondition to slicing a dataframe using indexes?

Answer 55

need to index using a column and also sort the index. eg:

df.set_index(“date of birth”).sort_index()

Question 56

slice a dataframe that’s been indexed by date, formatted yyyy-mm-dd, including all the dates from 1994 to 2019.

Answer 56

df[‘1994’: ‘2019’]

basically the point of this is that you only need to use partial strings when you’re slicing using df indexes

Question 57

get the 1st to 5th rows, and 3rd to 9th columns of a dataframe.

Answer 57

df.iloc[:5, 2:9]

of course the 5th row is number 4 and the 9th column is number 8, and slicing is end-exclusive.

Question 58

create a pivot table with dataframe meditation that has day of week as rows and retreat as columns, displaying meditation ‘quality’

Answer 58

meditation.pivot_table(‘quality, index = ‘day of week’, columns = ‘retreat’)

Question 59

working on a pivot table (meditation_pivot) with retreat y/n as columns and day of week as rows, get the mean values for the columns and then get the mean values for rows.

Answer 59

meditation_pivot.mean() - gives means for retreat (each column)

meditation_pivot.mean(axis = ‘columns’) - gives mean for each row

Question 60

get the means for the rows of a pivot table and then filter it by the max mean value of the rows

Answer 60

pivot = df.pivot_table(etc.)
pivot_means = pivot.mean(axis = 'columns')

print(pivot_means[pivot_means == pivot_means.max()])

Question 61

create a histogram of meditation length

Answer 61

meditation[‘length’].hist()

Question 62

create a histogram of meditation length with 15 bins

Answer 62

meditation[‘length’].hist(bins = 15)

Question 63

group meditation mean quality data by day of week and create a bar plot to show it

Answer 63

dayMeans = meditation.groupby(‘day of week’)[‘quality’].mean()

dayMeans.plot(kind = “bar”, title = “whatever - a good title”)

plt.show()

Question 64

Plot a line graph of quality over dates from the meditation dataframe.

Answer 64

meditation. plot(x = “date”, y = “quality”, kind = “line”)

plt. show()

Question 65

you’re creating a line plot from a dataframe - rotate the x labels by 45 degrees

Answer 65

df.plot(x = ‘col1’, y = ‘col2’, kind = ‘line’, rot = 45)

Question 66

create an arbitrary scatterplot from a dataframe

Answer 66

df.plot(x = ‘col1’, y = ‘col2’, kind= ‘scatter’)

Question 67

create a histogram of meditation quality just with the data from retreats

Answer 67

meditation[meditation[‘retreat’] == True][‘quality’].hist()

Question 68

make 2 arbitrary histograms and a legend

Answer 68

df[‘col1’].hist()
df[‘col2’.hist()
plt.legend([‘col1’, ‘col2’])
plt.show()

Question 69

make a translucent histogram from a dataframe

Answer 69

df[‘col1’].hist(alpha = 0.7)

Question 70

make a bar plot of the total amount of time spent meditating on each day of the week

Answer 70

totalDayTime = meditation.groupby(‘day of the week’)[‘length’].sum()

totalDayTime.plot(kind = “bar”)

Question 71

find whether any columns of a df have missing values

Answer 71

df.is_na().any()

Question 72

get the sum of any missing values in the columns of a dataframe

Answer 72

df.is_na().sum()

Question 73

delete any rows with missing data

Answer 73

df.dropna()

Question 74

replace any missing values with 0

Answer 74

df.fillna(0)

Question 75

create a dataframe row by row

Answer 75

do this with a list of dictionaries
e.g.:
dataframe1 = [
{‘row1’: ‘col1’, ‘col2’},
‘row2’: ‘col1’, ‘col2’}
]

row_by_row_df = pd.DataFrame(dataframe1)

Question 76

create a dataframe column by column

Answer 76

do this with a dictionary of lists
e.g.:
dataframe1 = {
‘col1’: [‘row1’, ‘row2’],
‘col2’: [‘row1’, ‘row2’]
}

col_by_col_df = pd.DataFrame(dataframe1)

Question 77

Open a csv file in Pandas

Answer 77

dateframe = pd.read_csv(“path.csv”)

Question 78

save a csv file from pandas df

Answer 78

dataframe.to_csv(“path.csv”)

Question 79

plot a simple histogram of meditation quality

Answer 79

_ = plt.hist(meditation['quality'])
_ = plt.xlabel('meditation Quality')
_ = plt.ylabel('Frequency')

Question 80

set the bin edges for a histogram, and create a histogram using them

Answer 80

bin_edges = [10,20,30,40,50,60,70,80]
_ = plt.hist(df['col1'], bins = bin_edges)

Question 81

Create a histogram with 13 bins

Answer 81

_ = plt.hist(df.’col1’, bins = 13)

Question 82

set up seaborn

Answer 82

import seaborn as sns

sns.set()

Question 83

what’s the histogram bin number rule for data sets?

Answer 83

number of bins equal to the sqrt of samples

Question 84

create a bee swarm plot of meditation quality on each day of the week from df ‘meditation’

Answer 84

_ = plt.swarmplot(x = ‘day of week’, y = ‘quality’, data = meditation)
_ = plt.xlabel(‘Day of Week’)
_ = plt.ylabel(‘Meditation Quality’)
plt.show()

Question 85

create an ecdf with dataframe data

Answer 85

x = np.sort(df[data])
n = x.size
y = np.arange(1, n+1)/n
plt.plot(x,y, marker = '.', linestyle = None)
plt.xlabel = 'xlabel'
plt.ylabel = 'ylabel'

Question 86

what the fuck does ecdf stand for??

Answer 86

empirical cumulative data function

Question 87

create multiple plots on the same graph

Answer 87

_ = plt.show()
_ = plt.show()
…etc.

Question 88

create a legend for a graph showing multiple data sets

Answer 88

plt.legend((dataSet1, dataSet2, etc.), loc = ‘upper right’)

Question 89

print the 25th, 50th, and 75th percentile for a dataframe column

Answer 89

np.percentile(df[‘col1’], [25,50,75])

Question 90

plot some percentiles of a dataset on the ECDF

Answer 90

the ‘D’ above stands for diamond

[create ecdf]

percentiles = [2.5, 25, 50, 75, 97.5]
data_ptiles = np.percentiles(df['col1'], percentiles)

_ = plt.plot(data_ptiles, percentiles/100, marker = ‘D’, color = ‘red’, linestyle = ‘none’)

Question 91

create a box plot comparing meditation quality on retreat vs off retreat (you gotta use seaborn)

Answer 91

_ = sns.boxplot(x = ‘retreat’, y = ‘quality’, data = meditation)

_ = plt.xlabel('on/off retreat')
_ = plt.ylabel('meditation quality /5')

plt.show()

Question 92

get the covariance matrix for two variables, col1 & col2

Answer 92

np.cov(df[‘col1’], df[‘col2’])

Question 93

get the correlation coefficient for two variables

Answer 93

np.corrcoef(df[‘col1’], df[‘col2’])

[0,1] or [1,0]

Question 94

get 4 random numbers

Answer 94

np.random.random(size = 4)

Question 95

create an empty array of x length

Answer 95

np.empty(x)

Question 96

create a simulation of 100 trials of 10 coin flips

Answer 96

np.random.binomial(10, 0.5, size = 100)

Question 97

compute the bin edges for a histogram such that the bars center over each integer from 0 to 10.

Answer 97

bins = np.arange(0, 11) - 0.5

Question 98

create a poisson distribution for an event with a mean of 8 occurrences, over 10,000 trials

Answer 98

np.random.poisson(8, size= 10000)

Question 99

Create a graph to demonstrate/check the normality of a data set with ecdf plots

Answer 99

so you compute the mean and std of your data set:
setMean = np.mean(df[‘col1’])
setStd = np.std(df[‘col1’])

then make a big set of perfectly normally distributed data with those numbers:

theoreticalSamples = np.random.normal(setMean, setStd, size = 10000)

x, y = ecdf(df[‘col1’])
xtheor, ytheor = ecdf(theoreticalSamples)

_ = plt.plot(x,y, marker = '.', linestyle = None)
_ = plt.plot(xtheor, ytheor, marker = '.', linestyle = None)

plt.show()