60 min Blitz

Question 1

create an empty matrix with dim 5,3

Answer 1

torch.empty(5,3)

Question 2

create random matrix with dim 5,3

Answer 2

torch.random(5,3)

Question 3

create zeros matrix of a type

Answer 3

torch.zeros(5,3,dtype=torch.long)

Question 4

construct a tensor directly from data

Answer 4

torch.tensor([5.5, 3])

Question 5

create a tensor based on an existing tensor

Answer 5

x = x.new_ones(5, 3, dtype=torch.double)
x = torch.randn_like(x, dtype=torch.float)

Question 6

get size of matrix

Answer 6

x. size()

torch. Size([5,3])

Question 7

operations

Answer 7

x+y

torch. add(x,y,out=result)
y. add_(x)

Question 8

resize/reshape tensor

Answer 8

y = x.view(16)
z = x.view(-1, 8) (the size -1 is inferred from other dimensions)

Question 9

get the value of one element tensor

Answer 9

x.item()

Question 10

convert a torch tensor to a numpy array and vice versa

Answer 10

b = a.numpy()
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)

Question 11

move tensors in and out of GPU

Answer 11

if torch.cuda.is_available():
device = torch.device(“cuda”) # a CUDA device object
y = torch.ones_like(x, device=device) # directly create a tensor on GPU
x = x.to(device) # or just use strings .to("cuda")
z = x + y

Question 12

package that provides automatic differentiation for all operations on Tensors

Answer 12

autograd

Question 13

track all operations on a torch.Tensor

Answer 13

set its attribute .requires_grad as True

Question 14

compute all the gradients automatically

Answer 14

call .backward()

Question 15

attribute which has the gradient

Answer 15

.grad

Question 16

stop a tensor from tracking history

Answer 16

call .detach()

Question 17

when wrapping the code block in with torch.no_grad(): is hepful

Answer 17

when evaluating a model

Question 18

attribute the references Function

Answer 18

.grad_fn

Question 19

example conv network

Answer 19

input 32x32
C1 : feature maps 6@28x28
S2: 6@14x14
C3: 16@10x20
S4: 16@5x5
C5: 120
F6: 84
output 10

Question 20

typical traing procedure

Answer 20

• Define the neural network that has some learnable parameters (or weights)
• Iterate over a dataset of inputs
• Process input through the network
• Compute the loss (how far is the output from being correct)
• Propagate gradients back into the network’s parameters
• Update the weights of the network, typically using a simple update rule: weight = weight - learning_rate * gradient

Question 21

does torch.nn support a single sample

Answer 21

no, it only supports a mini-batch of samples

Question 22

input of nn.Conv2d

Answer 22

4D Tensor of nSamples x nChanngels x Height x Width

Question 23

fake batch dimension

Answer 23

input.unsqueeze(0)

Question 24

calculate simple mean squared error between input and target

Answer 24

target = torch.randn(10)  # a dummy target, for example
target = target.view(1, -1)  # make it the same shape as output
criterion = nn.MSELoss()

loss = criterion(output, target)

Question 25

which Tensors will have their .grad Tensor accumulated with the gradient

Answer 25

Tensors with requires_grad=True

Question 26

print(loss.grad_fn) # MSELoss
print(loss.grad_fn.next_functions[0][0]) # Linear
print(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU

Question 27

backpropagate the erros

Answer 27

loss.backward()
conv1.bias.grad before backward
tensor([0., 0., 0., 0., 0., 0.])
conv1.bias.grad after backward
tensor([-0.0205, 0.0088, 0.0135, 0.0123, 0.0098, -0.0036])

Question 28

Stochastic Gradient Descent (SGD)

Answer 28

weight = weight - learning_rate * gradient

Question 29

implement SGD

Answer 29

learning_rate = 0.01
for f in net.parameters():
f.data.sub_(f.grad.data * learning_rate)

Question 30

implement other different update rules such as SGD, Nesterov-SGD, Adam, RMSProp

Answer 30

torch.optim

Question 31

create an optimizer

Answer 31

optimizer = optim.SGD(net.parameters(), lr=0.01)

Question 32

boilerplate code in training loop :

• calculate loss
• backpropagate loss
• optimize loss

Answer 32

optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update

Question 33

load data into np array

Answer 33

torchvision.datasets and torch.utils.data.DataLoader

Question 34

get some random training images

Answer 34

dataiter = iter(trainloader)

images, labels = dataiter.next()

Question 35

define a convolutional nn

Answer 35

class Net(nn.Module):
    def \_\_init\_\_(self):
        super(Net, self).\_\_init\_\_()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

Question 36

define loss function and optimizer

Answer 36

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

Question 37

train the network

Answer 37

for epoch in range(2): # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

print(‘Finished Training’)

Question 38

evaluate performance

Answer 38

correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(‘Accuracy of the network on the 10000 test images: %d %%’ % (
100 * correct / total))

Question 39

training on GPU

Answer 39

device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)
print(device)
net.to(device)
inputs, labels = inputs.to(device), labels.to(device)

Question 40

tutorial for data parallelism

Answer 40

https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html

Question 41

rnn embedding

Answer 41

https://github.com/hunkim/PyTorchZeroToAll/blob/master/12_4_hello_rnn_emb.py

Question 42

seq2seq

Answer 42

https://github.com/hunkim/PyTorchZeroToAll/blob/master/14_1_seq2seq.py

Question 43

seq2seq_att

Answer 43

https://github.com/hunkim/PyTorchZeroToAll/blob/master/14_2_seq2seq_att.py