Lesson 14 ML-SciKit

Question 1

Read in the data:

df = pd.read_csv(r”C:\Users\User\Documents\CFG_DATA\Data_files\BostonHousing.csv”, sep = “,”)

Plot a histogram of the target variable “medv”

Answer 1

df[‘medv’].hist(bins=15, figsize=(20,15))

Question 2

Use seaborn to visually improve this histogram using the column df[‘medv’]

Answer 2

plt.figure(figsize = (15,10)) # matplotlib
sns.distplot(df[‘medv’], bins = 30) # seaborn

Question 3

Use seaborn to create a heatmap of a correlation matrix.

Answer 3

plt.figure(figsize = (15,10))
correlation_matrix = df.corr()

sns.heatmap(data = correlation_matrix, annot = True)
plt.show()

Question 4

Create a scatterplot to visualize the relationship between MEDV and LSTAT

Use df[‘lstat’] and df[‘medv’]

Answer 4

plt.figure(figsize = (11,11))
plt.scatter(df[‘lstat’], df[‘medv’], marker=’o’)
plt.xlabel(“LSAT”)
plt.ylabel(“MEDV”)
plt.show()

Question 5

Write the code to import the code to build a model, and define the X and Y variable based on the column names [‘lstat’,’rm’ and ‘medv’ for Y.

Answer 5

from sklearn.model_selection import train_test_split

X = df[[‘lstat’, ‘rm’]] # our feature(s)
y = df[‘medv’] # our target variable

Question 6

Split data into train and test parts

Answer 6

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=5)

Question 7

Plot a histogram of the df using bins = 15 and figsize=20,15

Answer 7

df[‘medv’].hist(bins=15, figsize=(20,15))

Question 8

Examine what you have so far for all the x_train, x_test, y_train, y_test

Answer 8

print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)

Question 9

Let’s now use scikit-learn to train our simple linear regression model.

Answer 9

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)

Question 10

What does a linear regression model do?

Answer 10

A linear regression model calculates an equation that minimizes the distance
between the observed value and the predicted value.

Question 11

Work out the proportion of salary income over and under 50K from the income column.

Answer 11

df[‘income’].value_counts()/len(df[‘income’])

Question 12

Create a pairplot for the df

Answer 12

sns.pairplot(df)

Question 13

Create a violin plot for the categorical features

categorical_features = [‘workclass’, ‘education’, ‘marital.status’, ‘occupation’, ‘relationship’, ‘race’, ‘sex’, ‘native.country’]

Answer 13

for feature in categorical_features:
plt.figure(figsize = (13,10))
ax = sns.violinplot(x=feature, y=”hours.per.week”, hue=”income”,
data=df, palette=”Set2”, split=True)
plt.xticks(rotation=90)

Question 14

Modelling categorical data - need to convert the data to a number. Convert any income who is less than 50 to 0 and greater than 50k to 1.

Answer 14

df.loc[df[‘income’] == ‘<=50K’, ‘income’] = 0
df.loc[df[‘income’] == ‘>50K’, ‘income’] = 1

Question 15

ensure that income is an integer

Answer 15

df[‘income’] = df[‘income’].astype(int)

Question 16

What is one hot code?

Answer 16

One-hot encoding converts each categorical value into a new column and assigns a 1 or 0 (True/False) value to each row.

Question 17

What is an advantage and disadvantage of one hot code?

Answer 17

Advantage: “neutral” representation of the data (does not assign an order)
Disadvantage: can significantly increase the number of columns in the dataset