Pandas Flashcards by Saad E

dimension of a Series

How well did you know this?

Not at all

Perfectly

dimension of a DataFrame

How well did you know this?

Not at all

Perfectly

create a Series object from a list, index it using another list

ser=pd.Series(data=list1, index=list2)

How well did you know this?

Not at all

Perfectly

creat a Series using a numpy array

import numpy as np
arr=np.array([1,2,3,4])
ser=pd.Series(arr)

How well did you know this?

Not at all

Perfectly

create a Series using a dictionary with keys as the index

ser=pd.Series(dict)

How well did you know this?

Not at all

Perfectly

from the Series ser access the element with index ‘k’

ser['k']
#just like a dict

How well did you know this?

Not at all

Perfectly

Describe pd.DataFrame

Two-dimensional, size-mutable, potentially heterogeneous tabular data.

Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.

How well did you know this?

Not at all

Perfectly

What arguments does pd.DataFrame take ?

pandas.DataFrame(data=None, index=None, columns=None, dtype=None, copy=None)

How well did you know this?

Not at all

Perfectly

from DataFrame df grab the columns ‘name’ and ‘age’

mind the two brackets

df[[‘name, ‘age’]]

How well did you know this?

Not at all

Perfectly

from DataFrame df grab the row with index ‘B’ as a Series

df.loc[‘B’]

How well did you know this?

Not at all

Perfectly

from DataFrame df grab columns ‘one’, ‘three’ intersections with rows ‘B’, ‘D’

df.loc[[‘B’, ‘D’], [‘one’, ‘three’]]

How well did you know this?

Not at all

Perfectly

from DataFrame df, grab row with location 3,2

df.iloc[3,2]

How well did you know this?

Not at all

Perfectly

create a new column Total which shows the sum of the columns ‘C’, ‘D’, and ‘E’

df[‘Total’]=df[‘C’] + df[‘D’] + df[‘E’]

How well did you know this?

Not at all

Perfectly

in df, delete the row with index ‘F’

df.drop(‘F’, axis=0, inplace=True)

How well did you know this?

Not at all

Perfectly

in df, delete the column with index ‘Total’

df.drop(‘Total’, axis=1, inplace=True)

How well did you know this?

Not at all

Perfectly

in df, create a new column named ‘Sex’ and assign it as the index

df[‘Sex’]=[‘Men’, ‘Women’]

df.set_index(‘Sex’, inplace=True)

How well did you know this?

Not at all

Perfectly

Where does pandas beat numpy ?

NumPy’s ndarray data structure provides essential features for the type of
clean, well-organized data typically seen in numerical computing tasks. While it
serves this purpose very well, its limitations become clear when we need more flexibility
(attaching labels to data, working with missing data, etc.) and when attempting
operations that do not map well to element-wise broadcasting (groupings, pivots,
etc.), each of which is an important piece of analyzing the less structured data available
in many forms in the world around us. Pandas, and in particular its Series and
DataFrame objects, builds on the NumPy array structure and provides efficient access
to these sorts of “data munging” tasks that occupy much of a data scientist’s time.

How well did you know this?

Not at all

Perfectly

What does this dataframe call return : df[‘x’]

The column Series, and not the ROW, with index ‘x’

How well did you know this?

Not at all

Perfectly

What is the difference between slicing using an explicit index and using an implicit index ( series[‘a’:’c’] and series[0:2]) ?

the final index is included in the slice in the case of explicit indexing, and excluded from the slice in the case of implicit indexing

Suppose my Series ‘data’ has explicit integer indexing, what does each of the following expressions yield ?

data[1] ? #indexing
data[2:3] #slicing

indexing will use the explicit index and slicing will use the implicit index

State different methods of creating a multi-index

pass a list of two or more index arrays to the constructor
pass a dictionary with appropriate tuples as keys
explit constructor pd.MultiIndex:
pd. MultiIndex.from_arrays([[‘a’, ‘a’, ‘b’, ‘b’], [1, 2, 1, 2]])
pd. MultiIndex.from_tuples([(‘a’, 1), (‘a’, 2), (‘b’, 1), (‘b’, 2)])
pd. MultiIndex.from_product([[‘a’, ‘b’], [1, 2]])

how do we name the different levels of a multiIndex

df.index.names or by passing the parameters names directly in the MultiIndex constructor

DataFrame health_data has two level column MultiIndexing:

columns = pd.MultiIndex.from_product([[‘Bob’, ‘Guido’, ‘Sue’], [‘HR’, ‘Temp’]], names=[‘subject’, ‘type’])

Create in two different ways a slice with only the HR data

health_data.xs(‘HR’, level=1,axis=1, drop_level=False)

idx = pd.IndexSlice
health_data.loc[:, idx[:, ‘HR’]]

Suppose df has a multiIndex with level 0 being [‘a’,’c’,’b’], what could be a caveat to slicing :

df[‘a’:’b’] ?

How can it be fixed ?

This will return an error.

Fix : data = data.sort_index()

What does this line of code do ? | data.mean(axis=1, level='type')

Groups the columns by 'type' and then computes the mean along the columns

How does pandas concatenate dataframes with similar indices ?

Pandas preserves indices, even if it means repeating them

How do we raise an error if the dataframes we want to concatenate x, y have overlapping indices ? what if we want to ignore the indices instead and replace them with a default index ?

``` pd.concat([x, y], verify_integrity=True) #OR naming the data sources (multiIndexing) pd.concat([x, y], keys=['x', 'y']) ```

concatenate df1 and df2

pd.concat ( [ df1, df2 ] , join='outer') #default OR use df1.append(df2) #not in-place, returns a new dataframe pd. concat ( [ df1, df2 ] , join='inner') #intersect cols pd. concat ( [ df1, df2 ] , join_axes=['df1.columns'] ) #keep only df1 cols pd. concat ( [ df1, df2 ] , join_axes=['df2.columns'] ) #keep only df2 cols

Merge df1 and df2 on df1.name and df2.employee

pd.merge(df1, df3, left_on="employee", right_on="name")

How do we specify set arithmetics for joins

pd.merge(df6, df7, how='left')

df has columns A, B. what does df.mean() ?

the mean by columns

Find mean of df a dataframe with cols A and B by row

df.mean(axis='columns')

Get a statistical description of the data in df

df.dropna.describe()

what are the three steps involved in a groupby operation in pandas ?

split - apply - combine

group the rows of df by key then find min, max, and mean for each key for columns [ 'data1', 'data2' ]

``` df.groupby('key').aggregate(['min', np.aggregate(), max]) #aggregate takes a string, a function or a list thereof ``` Another useful pattern is to pass a dictionary mapping column names to operations to be applied on that column: df.groupby('key').aggregate({'data1': 'min', 'data2': 'max'})

from DataFrame df, keep all groups in which the standard deviation of column 'data2' is larger than 4

df.groupby('key').filter(lambda x: x['data2'.std() > 4])

in df, center the data by subtracting the (key's) group-wise mean hint:transform

df.groupby('key').transform(lambda x: x-x.mean)

group and sum the elements of df following the list : L = [0, 1, 0, 1, 2, 0]

L = [0, 1, 0, 1, 2, 0] df.groupby(L).sum() Another method is to provide a dictionary that maps index values to the group keys: df2 = df.set_index('key') mapping = {'A': 'vowel', 'B': 'consonant', 'C': 'consonant'} df2.groupby(mapping).sum() Similar to mapping, you can pass any Python function that will input the index value and output the group: df2.groupby(str.lower).mean() Further, any of the preceding key choices can be combined to group on a multi-index: df2.groupby([str.lower, mapping]).mean()

what motivated the creation of Pivot Tables ?

The need for a multidimensional version of GroupBy aggregation. That is, you split-apply-combine, but both the split and the combine happen across not a one-dimensional index, but across a two-dimensional grid.

1/ Find the ratio of survival by sex and class using the titanic dataframe with both GroupBy and Pivot syntax 2/ Look at age as a third dimension using pivot_table syntax, bin the age into [0,18,80] using pd.cut

import seaborn as sns titanic = sns.load_dataset('titanic') 1/ ``` #GROUPBY titanic.groupby( ['sex', 'class'])['survived'].mean().unstack() ``` ``` #PIVOT titanic.pivot_table('survived', index='sex', columns='class', aggfunc='mean') //mean is default, can also take a column>aggfunc mapping (dict) ``` 2/ age=pd.cut(titanic['age'], [0,18,80]) titanic.pivot_table('survived', ['sex', age], 'class')

At times it's useful to compute totals along each grouping of pandas.pivot_table. This can be done via the ?

margins keyword : | titanic.pivot_table('survived', index='sex', columns='class', margins=True)

data = ['peter', 'Paul', None, 'MARY', 'gUIDO'] use pandas.str to capitalize the strings in a vectorized fashion. Why would numpy or list comprehension fail at efficiently achieving this ?

names=pd.Series(data) | names.str.capitalize()

from dataframe names, slice the 3 first characters of each name into a new dataframe

names.str[0:3]

create a date in numpy's datetime64

import numpy as np date=np.array('2015-07-04', dtype=np.datetime64) #we can now do vectorized operations on 'date'

One detail of the datetime64 and timedelta64 objects is that they are built on a fundamental time unit. Because the datetime64 object is limited to 64-bit precision, the range of encodable times is 2pow64 times this fundamental unit. In other words, datetime64 imposes a trade-off between : ?

time resolution and maximum time span

what's the use of the pd.Period object ?

It encapsulates the granularity ('D', 'M') for arithmetic