60 min Blitz

Question 1

create an empty matrix with dim 5,3

Answer 1

torch.empty(5,3)

Question 2

create random matrix with dim 5,3

Answer 2

torch.random(5,3)

Question 3

create zeros matrix of a type

Answer 3

torch.zeros(5,3,dtype=torch.long)

Question 4

construct a tensor directly from data

Answer 4

torch.tensor([5.5, 3])

Question 5

create a tensor based on an existing tensor

Answer 5

x = x.new_ones(5, 3, dtype=torch.double)
x = torch.randn_like(x, dtype=torch.float)

Question 6

get size of matrix

Answer 6

x. size()

torch. Size([5,3])

Question 7

operations

Answer 7

x+y

torch. add(x,y,out=result)
y. add_(x)

Question 8

resize/reshape tensor

Answer 8

y = x.view(16)
z = x.view(-1, 8) (the size -1 is inferred from other dimensions)

Question 9

get the value of one element tensor

Answer 9

x.item()

Question 10

convert a torch tensor to a numpy array and vice versa

Answer 10

b = a.numpy()
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)

Question 11

move tensors in and out of GPU

Answer 11

if torch.cuda.is_available():
device = torch.device(“cuda”) # a CUDA device object
y = torch.ones_like(x, device=device) # directly create a tensor on GPU
x = x.to(device) # or just use strings .to("cuda")
z = x + y

Question 12

package that provides automatic differentiation for all operations on Tensors

Answer 12

autograd

Question 13

track all operations on a torch.Tensor

Answer 13

set its attribute .requires_grad as True

Question 14

compute all the gradients automatically

Answer 14

call .backward()

Question 15

attribute which has the gradient

Answer 15

.grad

Question 16

stop a tensor from tracking history

Answer 16

call .detach()

Question 17

when wrapping the code block in with torch.no_grad(): is hepful

Answer 17

when evaluating a model

Question 18

attribute the references Function

Answer 18

.grad_fn

Question 19

example conv network

Answer 19

input 32x32
C1 : feature maps 6@28x28
S2: 6@14x14
C3: 16@10x20
S4: 16@5x5
C5: 120
F6: 84
output 10

Question 20

typical traing procedure

Answer 20

• Define the neural network that has some learnable parameters (or weights)
• Iterate over a dataset of inputs
• Process input through the network
• Compute the loss (how far is the output from being correct)
• Propagate gradients back into the network’s parameters
• Update the weights of the network, typically using a simple update rule: weight = weight - learning_rate * gradient

Question 21

does torch.nn support a single sample

Answer 21

no, it only supports a mini-batch of samples

Question 22

input of nn.Conv2d

Answer 22

4D Tensor of nSamples x nChanngels x Height x Width

Question 23

fake batch dimension

Answer 23

input.unsqueeze(0)

Question 24

calculate simple mean squared error between input and target

Answer 24

target = torch.randn(10)  # a dummy target, for example
target = target.view(1, -1)  # make it the same shape as output
criterion = nn.MSELoss()

loss = criterion(output, target)

Question 25

which Tensors will have their .grad Tensor accumulated with the gradient

Answer 25

Tensors with requires_grad=True

Question 26

print(loss.grad_fn) # MSELoss print(loss.grad_fn.next_functions[0][0]) # Linear print(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU

Question 27

backpropagate the erros

Answer 27

loss.backward() conv1.bias.grad before backward tensor([0., 0., 0., 0., 0., 0.]) conv1.bias.grad after backward tensor([-0.0205, 0.0088, 0.0135, 0.0123, 0.0098, -0.0036])

Question 28

Stochastic Gradient Descent (SGD)

Answer 28

weight = weight - learning_rate * gradient

Question 29

implement SGD

Answer 29

learning_rate = 0.01 for f in net.parameters(): f.data.sub_(f.grad.data * learning_rate)

Question 30

implement other different update rules such as SGD, Nesterov-SGD, Adam, RMSProp

Answer 30

torch.optim

Question 31

create an optimizer

Answer 31

optimizer = optim.SGD(net.parameters(), lr=0.01)

Question 32

boilerplate code in training loop : - calculate loss - backpropagate loss - optimize loss

Answer 32

``` optimizer.zero_grad() # zero the gradient buffers output = net(input) loss = criterion(output, target) loss.backward() optimizer.step() # Does the update ```

Question 33

load data into np array

Answer 33

torchvision.datasets and torch.utils.data.DataLoader

Question 34

get some random training images

Answer 34

dataiter = iter(trainloader) | images, labels = dataiter.next()

Question 35

define a convolutional nn

Answer 35

``` class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(3, 6, 5) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(6, 16, 5) self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(120, 84) self.fc3 = nn.Linear(84, 10) ``` ``` def forward(self, x): x = self.pool(F.relu(self.conv1(x))) x = self.pool(F.relu(self.conv2(x))) x = x.view(-1, 16 * 5 * 5) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x ``` net = Net()

Question 36

define loss function and optimizer

Answer 36

``` criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) ```

Question 37

train the network

Answer 37

for epoch in range(2): # loop over the dataset multiple times ``` running_loss = 0.0 for i, data in enumerate(trainloader, 0): # get the inputs inputs, labels = data ``` ``` # zero the parameter gradients optimizer.zero_grad() ``` ``` # forward + backward + optimize outputs = net(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() ``` # print statistics running_loss += loss.item() if i % 2000 == 1999: # print every 2000 mini-batches print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000)) running_loss = 0.0 print('Finished Training')

Question 38

evaluate performance

Answer 38

``` correct = 0 total = 0 with torch.no_grad(): for data in testloader: images, labels = data outputs = net(images) _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() ``` print('Accuracy of the network on the 10000 test images: %d %%' % ( 100 * correct / total))

Question 39

training on GPU

Answer 39

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") print(device) net.to(device) inputs, labels = inputs.to(device), labels.to(device)

Question 40

tutorial for data parallelism

Answer 40

https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html

Question 41

rnn embedding

Answer 41

https://github.com/hunkim/PyTorchZeroToAll/blob/master/12_4_hello_rnn_emb.py

Question 42

seq2seq

Answer 42

https://github.com/hunkim/PyTorchZeroToAll/blob/master/14_1_seq2seq.py

Question 43

seq2seq_att

Answer 43

https://github.com/hunkim/PyTorchZeroToAll/blob/master/14_2_seq2seq_att.py