Pandas

Question 1

dimension of a Series

Answer 1

1

Question 2

dimension of a DataFrame

Answer 2

2

Question 3

create a Series object from a list, index it using another list

Answer 3

ser=pd.Series(data=list1, index=list2)

Question 4

creat a Series using a numpy array

Answer 4

import numpy as np
arr=np.array([1,2,3,4])
ser=pd.Series(arr)

Question 5

create a Series using a dictionary with keys as the index

Answer 5

ser=pd.Series(dict)

Question 6

from the Series ser access the element with index ‘k’

Answer 6

ser['k']
#just like a dict

Question 7

Describe pd.DataFrame

Answer 7

Two-dimensional, size-mutable, potentially heterogeneous tabular data.

Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.

Question 8

What arguments does pd.DataFrame take ?

Answer 8

pandas.DataFrame(data=None, index=None, columns=None, dtype=None, copy=None)

Question 9

from DataFrame df grab the columns ‘name’ and ‘age’

Answer 9

mind the two brackets

df[[‘name, ‘age’]]

Question 10

from DataFrame df grab the row with index ‘B’ as a Series

Answer 10

df.loc[‘B’]

Question 11

from DataFrame df grab columns ‘one’, ‘three’ intersections with rows ‘B’, ‘D’

Answer 11

df.loc[[‘B’, ‘D’], [‘one’, ‘three’]]

Question 12

from DataFrame df, grab row with location 3,2

Answer 12

df.iloc[3,2]

Question 13

create a new column Total which shows the sum of the columns ‘C’, ‘D’, and ‘E’

Answer 13

df[‘Total’]=df[‘C’] + df[‘D’] + df[‘E’]

Question 14

in df, delete the row with index ‘F’

Answer 14

df.drop(‘F’, axis=0, inplace=True)

Question 15

in df, delete the column with index ‘Total’

Answer 15

df.drop(‘Total’, axis=1, inplace=True)

Question 16

in df, create a new column named ‘Sex’ and assign it as the index

Answer 16

df[‘Sex’]=[‘Men’, ‘Women’]

df.set_index(‘Sex’, inplace=True)

Question 17

Where does pandas beat numpy ?

Answer 17

NumPy’s ndarray data structure provides essential features for the type of
clean, well-organized data typically seen in numerical computing tasks. While it
serves this purpose very well, its limitations become clear when we need more flexibility
(attaching labels to data, working with missing data, etc.) and when attempting
operations that do not map well to element-wise broadcasting (groupings, pivots,
etc.), each of which is an important piece of analyzing the less structured data available
in many forms in the world around us. Pandas, and in particular its Series and
DataFrame objects, builds on the NumPy array structure and provides efficient access
to these sorts of “data munging” tasks that occupy much of a data scientist’s time.

Question 18

What does this dataframe call return : df[‘x’]

Answer 18

The column Series, and not the ROW, with index ‘x’

Question 19

What is the difference between slicing using an explicit index and using an implicit index ( series[‘a’:’c’] and series[0:2]) ?

Answer 19

the final index is included in the slice in the case of explicit indexing, and excluded from the slice in the case of implicit indexing

Question 20

Suppose my Series ‘data’ has explicit integer indexing, what does each of the following expressions yield ?

data[1] ? #indexing
data[2:3] #slicing

Answer 20

indexing will use the explicit index and slicing will use the implicit index

Question 21

State different methods of creating a multi-index

Answer 21

• pass a list of two or more index arrays to the constructor
• pass a dictionary with appropriate tuples as keys
• explit constructor pd.MultiIndex:
  pd. MultiIndex.from_arrays([[‘a’, ‘a’, ‘b’, ‘b’], [1, 2, 1, 2]])
  pd. MultiIndex.from_tuples([(‘a’, 1), (‘a’, 2), (‘b’, 1), (‘b’, 2)])
  pd. MultiIndex.from_product([[‘a’, ‘b’], [1, 2]])

Question 22

how do we name the different levels of a multiIndex

Answer 22

df.index.names or by passing the parameters names directly in the MultiIndex constructor

Question 23

DataFrame health_data has two level column MultiIndexing:

columns = pd.MultiIndex.from_product([[‘Bob’, ‘Guido’, ‘Sue’], [‘HR’, ‘Temp’]], names=[‘subject’, ‘type’])

Create in two different ways a slice with only the HR data

Answer 23

OR

health_data.xs(‘HR’, level=1,axis=1, drop_level=False)

idx = pd.IndexSlice
health_data.loc[:, idx[:, ‘HR’]]

Question 24

Suppose df has a multiIndex with level 0 being [‘a’,’c’,’b’], what could be a caveat to slicing :

df[‘a’:’b’] ?

How can it be fixed ?

Answer 24

This will return an error.

Fix : data = data.sort_index()

Question 25

What does this line of code do ?

data.mean(axis=1, level=’type’)

Answer 25

Groups the columns by ‘type’ and then computes the mean along the columns

Question 26

How does pandas concatenate dataframes with similar indices ?

Answer 26

Pandas preserves indices, even if it means repeating them

Question 27

How do we raise an error if the dataframes we want to concatenate x, y have overlapping indices ? what if we want to ignore the indices instead and replace them with a default index ?

Answer 27

pd.concat([x, y], verify_integrity=True)
#OR naming the data sources (multiIndexing)
pd.concat([x, y], keys=['x', 'y'])

Question 28

concatenate df1 and df2

Answer 28

pd.concat ( [ df1, df2 ] , join=’outer’) #default
OR use df1.append(df2) #not in-place, returns a new dataframe

pd. concat ( [ df1, df2 ] , join=’inner’) #intersect cols
pd. concat ( [ df1, df2 ] , join_axes=[‘df1.columns’] ) #keep only df1 cols
pd. concat ( [ df1, df2 ] , join_axes=[‘df2.columns’] ) #keep only df2 cols

Question 29

Merge df1 and df2 on df1.name and df2.employee

Answer 29

pd.merge(df1, df3, left_on=”employee”, right_on=”name”)

Question 30

How do we specify set arithmetics for joins

Answer 30

pd.merge(df6, df7, how=’left’)

Question 31

df has columns A, B. what does df.mean() ?

Answer 31

the mean by columns

Question 32

Find mean of df a dataframe with cols A and B by row

Answer 32

df.mean(axis=’columns’)

Question 33

Get a statistical description of the data in df

Answer 33

df.dropna.describe()

Question 34

what are the three steps involved in a groupby operation in pandas ?

Answer 34

split - apply - combine

Question 35

group the rows of df by key then find min, max, and mean for each key for columns [ ‘data1’, ‘data2’ ]

Answer 35

df.groupby('key').aggregate(['min', np.aggregate(), max])
#aggregate takes a string, a function or a list thereof

Another useful pattern is to pass a dictionary mapping column names to operations to be applied on that column:

df.groupby(‘key’).aggregate({‘data1’: ‘min’, ‘data2’: ‘max’})

Question 36

from DataFrame df, keep all groups in which the standard deviation of column ‘data2’ is larger than 4

Answer 36

df.groupby(‘key’).filter(lambda x: x[‘data2’.std() > 4])

Question 37

in df, center the data by subtracting the (key’s) group-wise mean

hint:transform

Answer 37

df.groupby(‘key’).transform(lambda x: x-x.mean)

Question 38

group and sum the elements of df following the list : L = [0, 1, 0, 1, 2, 0]

Answer 38

L = [0, 1, 0, 1, 2, 0]
df.groupby(L).sum()

Another method is to provide a dictionary that maps index values to the group keys:
df2 = df.set_index(‘key’)
mapping = {‘A’: ‘vowel’, ‘B’: ‘consonant’, ‘C’: ‘consonant’}
df2.groupby(mapping).sum()

Similar to mapping, you can pass any Python function that will input the index value and output the group:
df2.groupby(str.lower).mean()

Further, any of the preceding key choices can be combined to group on a multi-index:
df2.groupby([str.lower, mapping]).mean()

Question 39

what motivated the creation of Pivot Tables ?

Answer 39

The need for a multidimensional version of GroupBy aggregation. That is, you split-apply-combine, but both the split and the combine happen across not a one-dimensional index, but across a two-dimensional grid.

Question 40

1/ Find the ratio of survival by sex and class using the titanic dataframe with both GroupBy and Pivot syntax

2/ Look at age as a third dimension using pivot_table syntax, bin the age into [0,18,80] using pd.cut

Answer 40

import seaborn as sns
titanic = sns.load_dataset(‘titanic’)

1/

#GROUPBY
titanic.groupby( ['sex', 'class'])['survived'].mean().unstack()

#PIVOT
titanic.pivot_table('survived', index='sex', columns='class', aggfunc='mean') //mean is default, can also take a column>aggfunc mapping (dict)

2/

age=pd.cut(titanic[‘age’], [0,18,80])
titanic.pivot_table(‘survived’, [‘sex’, age], ‘class’)

Question 41

At times it’s useful to compute totals along each grouping of pandas.pivot_table. This can be done via the ?

Answer 41

margins keyword :

titanic.pivot_table(‘survived’, index=’sex’, columns=’class’, margins=True)

Question 42

data = [‘peter’, ‘Paul’, None, ‘MARY’, ‘gUIDO’]

use pandas.str to capitalize the strings in a vectorized fashion. Why would numpy or list comprehension fail at efficiently achieving this ?

Answer 42

names=pd.Series(data)

names.str.capitalize()

Question 43

from dataframe names, slice the 3 first characters of each name into a new dataframe

Answer 43

names.str[0:3]

Question 44

create a date in numpy’s datetime64

Answer 44

we can now do vectorized operations on ‘date’

import numpy as np
date=np.array(‘2015-07-04’, dtype=np.datetime64)

Question 45

One detail of the datetime64 and timedelta64 objects is that they are built on a fundamental time unit. Because the datetime64 object is limited to 64-bit precision, the range of encodable times is 2pow64 times this fundamental unit. In other words, datetime64 imposes a trade-off between : ?

Answer 45

time resolution and maximum time span

Question 46

what’s the use of the pd.Period object ?

Answer 46

It encapsulates the granularity (‘D’, ‘M’) for arithmetic