Introduction to PyTorch

1) Initializing a Tensor:

• Directly from data: 

torch.tensor(data)

• From a NumPy array: 

torch.from_numpy(np_array)

• From another tensor: 

torch.rand_like(x_data, dtype=torch.float)

• With random or constant values: 

rand_tensor = torch.rand(shape)

ones_tensor = torch.ones(shape)

zeros_tensor = torch.zeros(shape)

2) Attributes of a Tensor:

• Describe their shape, data type, and the device:

print(f"Shape of tensor: {tensor.shape}")

print(f"Datatype of tensor: {tensor.dtype}")

print(f"Device tensor is stored on: {tensor.device}")

3) Operations on Tensors:

• Move tensors to the GPU:

if torch.cuda.is_available():

   tensor = tensor.to('cuda')

• Standard numpy-like indexing and slicing:

tensor[:,1] = 0

• Joining tensors - concatenate a sequence of tensors:

t1 = torch.cat([tensor, tensor, tensor], dim=1

• Arithmetic operations:

# This computes the matrix multiplication 

torch.matmul(tensor, tensor.T, out=y3)

# This computes the element-wise product.

torch.mul(tensor, tensor, out=z3)

• Single-element tensors - convert it to a Python numerical value using item():

agg_item = agg.item()  

• In-place operations-  Operations that store the result into the operand are called in-place. They are denoted by a _ suffix:

tensor.add_(5)

• Bridge with NumPy - Tensor to NumPy array:

t = torch.ones(5)

print(f"t: {t}")

n = t.numpy()

print(f"n: {n}")

# A change in the tensor reflects in the NumPy array.

t.add_(1)

print(f"t: {t}")

print(f"n: {n}")

• Bridge with NumPy - NumPy array to Tensor:

n = np.ones(5)

t = torch.from_numpy(n)

# Changes in the NumPy array reflects in the tensor.

np.add(n, 1, out=n)

print(f"t: {t}")

print(f"n: {n}")

1) Loading a dataset:

• Here is an example of how to load the Fashion-MNIST dataset from TorchVision:

training_data = datasets.FashionMNIST(

    root="data",

    train=True,

    download=True,

    transform=ToTensor()

)

test_data = datasets.FashionMNIST(

    root="data",

    train=False,

    download=True,

    transform=ToTensor()

)

2) Iterating and Visualizing the Dataset:

• We can index Datasets manually like a list: training_data[index]. We use matplotlib to visualize some samples in our training data.

labels_map = {

    0: "T-Shirt",

    1: "Trouser",

    2: "Pullover",

    3: "Dress",

    4: "Coat",

    5: "Sandal",

    6: "Shirt",

    7: "Sneaker",

    8: "Bag",

    9: "Ankle Boot",

}

figure = plt.figure(figsize=(8, 8))

cols, rows = 3, 3

for i in range(1, cols * rows + 1):

    sample_idx = torch.randint(len(training_data), size=(1,)).item()

    img, label = training_data[sample_idx]

    figure.add_subplot(rows, cols, i)

    plt.title(labels_map[label])

    plt.axis("off")

    plt.imshow(img.squeeze(), cmap="gray")

plt.show()

2) Creating a Custom Dataset for your files:

• A custom Dataset class must implement three functions: __init__, __len__, and __getitem__. 

• The __init__ function is run once when instantiating the Dataset object. We initialize the directory containing the images, the annotations file, and both transforms (covered in more detail in the next section).

• The __len__ function returns the number of samples in our dataset.

• The __getitem__ function loads and returns a sample from the dataset at the given index idx. Based on the index, it identifies the image's location on disk, converts that to a tensor using read_image, retrieves the corresponding label from the csv data in self.img_labels, calls the transform functions on them (if applicable), and returns the tensor image and corresponding label in a Python dict.

• Take a look at this implementation; the FashionMNIST images are stored in a directory img_dir, and their labels are stored separately in a CSV file annotations_file.

import os

import pandas as pd

import torchvision.io as tvio

class CustomImageDataset(Dataset):

    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):

        self.img_labels = pd.read_csv(annotations_file)

        self.img_dir = img_dir

        self.transform = transform

        self.target_transform = target_transform

    def __len__(self):

        return len(self.img_labels)

    def __getitem__(self, idx):

        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])

        image = tvio.read_image(img_path)

        label = self.img_labels.iloc[idx, 1]

        if self.transform:

            image = self.transform(image)

        if self.target_transform:

            label = self.target_transform(label)

        sample = {"image": image, "label": label}

        return sample

3) Preparing your data for training with DataLoaders:

• The Dataset retrieves our dataset's features and labels one sample at a time. 

• While training a model, we typically want to pass samples in "minibatches", reshuffle the data at every epoch to reduce model overfitting, and use Python's multiprocessing to speed up data retrieval.

• DataLoader is an iterable that abstracts this complexity for us in an easy API.

from torch.utils.data import DataLoader

train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)

test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)

4) Iterate through the DataLoader:

• We have loaded that dataset into the Dataloader and can iterate through the dataset as needed. 

• Each iteration below returns a batch of train_features and train_labels (containing batch_size=64 features and labels respectively). Because we specified shuffle=True, after we iterate over all batches the data is shuffled (for finer-grained control over the data loading order, take a look at Samplers.

# Display image and label.

train_features, train_labels = next(iter(train_dataloader))

print(f"Feature batch shape: {train_features.size()}")

print(f"Labels batch shape: {train_labels.size()}")

img = train_features[0].squeeze()

label = train_labels[0]

plt.imshow(img, cmap="gray")

plt.show()

print(f"Label: {label}")

1) Transforms:

• Data does not always come in its final processed form that is required for training machine learning algorithms. We use transforms to perform some manipulation of the data and make it suitable for training.

• All TorchVision datasets have two parameters (transform to modify the features and target_transform to modify the labels) that accept callables containing the transformation logic. The torchvision.transforms module offers several commonly-used transforms out of the box.

• The FashionMNIST features are in PIL Image format, and the labels are integers. For training, we need the features as normalized tensors, and the labels as one-hot encoded tensors. To make these transformations, we use ToTensor and Lambda.

ds = datasets.FashionMNIST(

    root="data",

    train=True,

    download=True,

    transform=ToTensor(),

    target_transform=Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(0, torch.tensor(y), value=1))

)

2) ToTensor()

• ToTensor converts a PIL image or NumPy ndarray into a FloatTensor and scales the image's pixel intensity values in the range [0., 1.]

3) Lambda transforms

• Lambda transforms apply any user-defined lambda function. Here, we define a function to turn the integer into a one-hot encoded tensor. It first creates a zero tensor of size 10 (the number of labels in our dataset) and calls scatter which assigns a value=1 on the index as given by the label y.

1) Build a neural network:

• Neural networks comprise of layers/modules that perform operations on data. The torch.nn namespace provides all the building blocks you need to build your own neural network. Every module in PyTorch subclasses the nn.Module. A neural network is a module itself that consists of other modules (layers). This nested structure allows for building and managing complex architectures easily.

import os

import torch

from torch import nn

from torch.utils.data import DataLoader

from torchvision import datasets, transforms

2) Get a hardware device for training:

• We want to be able to train our model on a hardware accelerator like the GPU, if it is available. Let's check to see if torch.cuda is available, else we continue to use the CPU.

device = 'cuda' if torch.cuda.is_available() else 'cpu'

print('Using {} device'.format(device))

3) Define the class:

• We define our neural network by subclassing nn.Module, and initialize the neural network layers in __init__. 

• Every nn.Module subclass implements the operations on input data in the forward method.

class NeuralNetwork(nn.Module):

    def __init__(self):

        super(NeuralNetwork, self).__init__()

        self.flatten = nn.Flatten()

        self.linear_relu_stack = nn.Sequential(

            nn.Linear(28*28, 512),

            nn.ReLU(),

            nn.Linear(512, 512),

            nn.ReLU(),

            nn.Linear(512, 10),

            nn.ReLU()

        )

    def forward(self, x):

        x = self.flatten(x)

        logits = self.linear_relu_stack(x)

        return logits

• We create an instance of NeuralNetwork, and move it to the device, and print it's structure.

model = NeuralNetwork().to(device)

print(model)

• To use the model, we pass it the input data. This executes the model's forward, along with some background operations. Do not call model.forward() directly!

• Calling the model on the input returns a 10-dimensional tensor with raw predicted values for each class. We get the prediction densities by passing it through an instance of the nn.Softmax module.

X = torch.rand(1, 28, 28, device=device)

logits = model(X) 

pred_probab = nn.Softmax(dim=1)(logits)

y_pred = pred_probab.argmax(1)

print(f"Predicted class: {y_pred}")

#OUTPUT: Predicted class: tensor([6], device='cuda:0') 

4) Model Layers:

• Let's break down the layers in the FashionMNIST model. To illustrate it, we will take a sample minibatch of 3 images of size 28x28 and see what happens to it as we pass it through the network.

input_image = torch.rand(3,28,28)

print(input_image.size())

• nn.Flatten: We initialize the nn.Flatten layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (the minibatch dimension (at dim=0) is maintained).

flatten = nn.Flatten()

flat_image = flatten(input_image)

print(flat_image.size())

• nn.Linear: The linear layer is a module that applies a linear transformation on the input using it's stored weights and biases.

layer1 = nn.Linear(in_features=28*28, out_features=20)

hidden1 = layer1(flat_image)

print(hidden1.size())

• nn.ReLU: Non-linear activations are what create the complex mappings between the model's inputs and outputs. They are applied after linear transformations to introduce nonlinearity, helping neural networks learn a wide variety of phenomena.

print(f"Before ReLU: {hidden1}\n\n")

hidden1 = nn.ReLU()(hidden1)

print(f"After ReLU: {hidden1}")

• nn.Sequential: nn.Sequential is an ordered container of modules. The data is passed through all the modules in the same order as defined. You can use sequential containers to put together a quick network like seq_modules.

seq_modules = nn.Sequential(

    flatten,

    layer1,

    nn.ReLU(),

    nn.Linear(20, 10)

)

input_image = torch.rand(3,28,28)

logits = seq_modules(input_image)

• nn.Softmax: The last linear layer of the neural network returns logits - raw values in [-infty, infty] - which are passed to the nn.Softmax module. The logits are scaled to values [0, 1] representing the model's predicted densities for each class. dim parameter indicates the dimension along which the values must sum to 1.

softmax = nn.Softmax(dim=1)

pred_probab = softmax(logits)

5) Model parameters:

• Many layers inside a neural network are parameterized, i.e. have associated weights and biases that are optimized during training. Subclassing nn.Module automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model's parameters() or named_parameters() methods.

• In this example, we iterate over each parameter, and print its size and a preview of its val

print("Model structure: ", model, "\n\n")

for name, param in model.named_parameters():

    print(f"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \n")

1) Automatic differentiation with torch.autograd:

• When training neural networks, the most frequently used algorithm is back propagation. In this algorithm, parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.

• To compute those gradients, PyTorch has a built-in differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.

• Consider the simplest one-layer neural network, with input x, parameters w and b, and some loss function. It can be defined in PyTorch in the following manner:

import torch

x = torch.ones(5)  # input tensor

y = torch.zeros(3)  # expected output

w = torch.randn(5, 3, requires_grad=True) # set requires_grad=True for learning parameters

b = torch.randn(3, requires_grad=True)

z = torch.matmul(x, w)+b

loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)

2) Tensors, Functions and Computational graph:

• In this network, w and b are parameters, which we need to optimize. Thus, we need to be able to compute the gradients of loss function with respect to those variables. In order to do that, we set the requires_grad property of those tensors.

• Note: You can set the value of requires_grad when creating a tensor, or later by using x.requires_grad_(True) method.

• A function that we apply to tensors to construct computational graph is in fact an object of class Function. This object knows how to compute the function in the forward direction, and also how to compute its derivative during the backward propagation step. A reference to the backward propagation function is stored in grad_fn property of a tensor. You can find more information of Function in the documentation.

print('Gradient function for z =',z.grad_fn)

print('Gradient function for loss =', loss.grad_fn)

3) Computing gradients:

• To optimize weights of parameters in the neural network, we need to compute the derivatives of our loss function with respect to parameters, namely, we need dloss/dw and dloss/db under some fixed values of x and y. To compute those derivatives, we call loss.backward(), and then retrieve the values from w.grad and b.grad:

loss.backward()

print(w.grad)

print(b.grad)

• Note: We can only obtain the grad properties for the leaf nodes of the computational graph, which have requires_grad property set to True. For all other nodes in our graph, gradients will not be available. In addition, we can only perform gradient calculations using backward once on a given graph, for performance reasons. If we need to do several backward calls on the same graph, we need to pass retain_graph=True to the backward call.

4) Disabling gradient tracking:

• By default, all tensors with requires_grad=True are tracking their computational history and support gradient computation. However, there are some cases when we do not need to do that, for example, when we have trained the model and just want to apply it to some input data, i.e. we only want to do forward computations through the network. 

• We can stop tracking computations by surrounding our computation code with torch.no_grad() block:

z = torch.matmul(x, w)+b

print(z.requires_grad) # True

with torch.no_grad():

    z = torch.matmul(x, w)+b

print(z.requires_grad) # False

• Another way to achieve the same result is to use the detach() method on the tensor:

z = torch.matmul(x, w)+b

z_det = z.detach()

print(z_det.requires_grad) # False

5) More on Computational Graphs:

• Conceptually, autograd keeps a record of data (tensors) and all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG) consisting of Function objects. 

• In this DAG, leaves are the input tensors, roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.

• In a forward pass, autograd does two things simultaneously:

+  run the requested operation to compute a resulting tensor

+  maintain the operation’s gradient function in the DAG.

• The backward pass kicks off when .backward() is called on the DAG root. autograd then:

+  computes the gradients from each .grad_fn,

+  accumulates them in the respective tensor’s .grad attribute

+  using the chain rule, propagates all the way to the leaf tensors.

• DAGs are dynamic in PyTorch: An important thing to note is that the graph is recreated from scratch; after each .backward() call, autograd starts populating a new graph. This is exactly what allows you to use control flow statements in your model; you can change the shape, size and operations at every iteration if needed.

6) Tensor gradients and Jacobian products:

• In many cases, we have a scalar loss function, and we need to compute the gradient with respect to some parameters. However, there are cases when the output function is an arbitrary tensor. In this case, PyTorch allows you to compute so-called Jacobian product, and not the actual gradient.

inp = torch.eye(5, requires_grad=True)

out = (inp+1).pow(2)

out.backward(torch.ones_like(inp), retain_graph=True)

print("First call\n", inp.grad)

out.backward(torch.ones_like(inp), retain_graph=True)

print("\nSecond call\n", inp.grad)

inp.grad.zero_()

out.backward(torch.ones_like(inp), retain_graph=True)

print("\nCall after zeroing gradients\n", inp.grad)

#Output:

First call

 tensor([[4., 2., 2., 2., 2.],

        [2., 4., 2., 2., 2.],

        [2., 2., 4., 2., 2.],

        [2., 2., 2., 4., 2.],

        [2., 2., 2., 2., 4.]])

Second call

 tensor([[8., 4., 4., 4., 4.],

        [4., 8., 4., 4., 4.],

        [4., 4., 8., 4., 4.],

        [4., 4., 4., 8., 4.],

        [4., 4., 4., 4., 8.]])

Call after zeroing gradients

 tensor([[4., 2., 2., 2., 2.],

        [2., 4., 2., 2., 2.],

        [2., 2., 4., 2., 2.],

        [2., 2., 2., 4., 2.],

        [2., 2., 2., 2., 4.]])

• Notice that when we call backward for the second time with the same argument, the value of the gradient is different. This happens because when doing backward propagation, PyTorch accumulates the gradients, i.e. the value of computed gradients is added to the grad property of all leaf nodes of computational graph. If you want to compute the proper gradients, you need to zero out the grad property before. In real-life training an optimizer helps us to do this.

• Note: previously we were calling backward() function without parameters. This is equivalent to calling backward(torch.tensor(1.0)), which is a useful way to compute the gradients in case of a scalar-valued function, such as loss during neural network training.

1) Optimizing the model parameters:

• Now that we have a model and data it's time to train, validate and test our model by optimizing its parameters on our data. Training a model is an iterative process; in each iteration (called an epoch) the model makes a guess about the output, calculates the error in its guess (loss), collects the derivatives of the error with respect to its parameters (as we saw in the module), and optimizes these parameters using gradient descent. For a more detailed walkthrough of this process, check out this video on backpropagation from 3Blue1Brown.

2) Setting hyperparameters:

• Hyperparameters are adjustable parameters that let you control the model optimization process. Different hyperparameter values can impact model training and convergence rates (read more about hyperparameter tuning)

• Number of Epochs - the number times to iterate over the dataset

• Batch Size - the number of data samples seen by the model in each epoch

• Learning Rate - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.

learning_rate = 1e-3

batch_size = 64

epochs = 5

3) Add an optimization loop:

• Once we set our hyperparameters, we can then train and optimize our model with an optimization loop. Each iteration of the optimization loop is called an epoch.

• Each epoch consists of two main parts:

  1. The Train Loop - iterate over the training dataset and try to converge to optimal parameters.

  2. The Validation/Test Loop - iterate over the test dataset to check if model performance is improving.

• Let's briefly familiarize ourselves with some of the concepts used in the training loop. Jump ahead to see the full-impl-label of the optimization loop.

4) Add a loss function:

• When presented with some training data, our untrained network is likely not to give the correct answer. Loss function measures the degree of dissimilarity of obtained result to the target value, and it is the loss function that we want to minimize during training. To calculate the loss we make a prediction using the inputs of our given data sample and compare it against the true data label value.

• Common loss functions include nn.MSELoss (Mean Square Error) for regression tasks, and nn.NLLLoss (Negative Log Likelihood) for classification. nn.CrossEntropyLoss combines nn.LogSoftmax and nn.NLLLoss.

• We pass our model's output logits to nn.CrossEntropyLoss, which will normalize the logits and compute the prediction error.

# Initialize the loss function

loss_fn = nn.CrossEntropyLoss()

5) Optimization pass:

• Optimization is the process of adjusting model parameters to reduce model error in each training step. Optimization algorithms define how this process is performed (in this example we use Stochastic Gradient Descent). All optimization logic is encapsulated in the optimizer object. Here, we use the SGD optimizer; additionally, there are many different optimizers available in PyTorch such as ADAM and RMSProp, that work better for different kinds of models and data.

• We initialize the optimizer by registering the model's parameters that need to be trained, and passing in the learning rate hyperparameter.

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

1. Call optimizer.zero_grad() to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.

2. Back-propagate the prediction loss with a call to loss.backwards(). PyTorch deposits the gradients of the loss w.r.t. each parameter.

3. Once we have our gradients, we call optimizer.step() to adjust the parameters by the gradients collected in the backward pass.

6) Full implementation:

• We define train_loop that loops over our optimization code, and test_loop that evaluates the model's performance against our test data.

def train_loop(dataloader, model, loss_fn, optimizer):

    size = len(dataloader.dataset)

    for batch, (X, y) in enumerate(dataloader):        

        # Compute prediction and loss

        pred = model(X)

        loss = loss_fn(pred, y)

        # Backpropagation

        optimizer.zero_grad()

        loss.backward()

        optimizer.step()

        if batch % 100 == 0:

            loss, current = loss.item(), batch * len(X)

            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def test_loop(dataloader, model, loss_fn):

    size = len(dataloader.dataset)

    test_loss, correct = 0, 0

    with torch.no_grad():

        for X, y in dataloader:

            pred = model(X)

            test_loss += loss_fn(pred, y).item()

            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= size

    correct /= size

    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

• We initialize the loss function and optimizer, and pass it to train_loop and test_loop. Feel free to increase the number of epochs to track the model's improving performance.

loss_fn = nn.CrossEntropyLoss()

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

epochs = 10

for t in range(epochs):

    print(f"Epoch {t+1}\n-------------------------------")

    train_loop(train_dataloader, model, loss_fn, optimizer)

    test_loop(test_dataloader, model, loss_fn)

print("Done!")

1) Saving and loading model weights:

• PyTorch models store the learned parameters in an internal state dictionary, called state_dict. These can be persisted via the torch.save method:

model = models.vgg16(pretrained=True)

torch.save(model.state_dict(), 'data/model_weights.pth')

• To load model weights, you need to create an instance of the same model first, and then load the parameters using the load_state_dict() method.

model = models.vgg16() # we do not specify pretrained=True, i.e. do not load default weights

model.load_state_dict(torch.load('data/model_weights.pth'))

model.eval()

• Note: Be sure to call model.eval() method before inferencing to set the dropout and batch normalization layers to evaluation mode. Failing to do this will yield inconsistent inference results.

2) Saving and loading models with shapes:

• When loading model weights, we needed to instantiate the model class first, because the class defines the structure of a network. We might want to save the structure of this class together with the model, in which case we can pass model (and not model.state_dict()) to the saving function:

torch.save(model, 'data/vgg_model.pth')

• We can then load the model like this:

model = torch.load('data/vgg_model.pth')

• Note: This approach uses Python pickle module when serializing the model, thus it relies on the actual class definition to be available when loading the model.

3) Exporting the model to ONNX:

• PyTorch also has native ONNX export support. Given the dynamic nature of the PyTorch execution graph, however, the export process must traverse the execution graph to produce a persisted ONNX model. For this reason, a test variable of the appropriate size should be passed in to the export routine (in our case, we will create a dummy zero tensor of the correct size):

input_image = torch.zeros((1,3,224,224))

onnx.export(model, input_image, 'data/model.onnx')

• There are a lot of things you can do with ONNX model, including running inference on different platforms and in different programming languages. For more details, we recommend visiting ONNX tutorial.

In this module we introduced the key concepts to building machine learning models and implemented those concepts with PyTorch. We built a Computer Vision model that could classify images of T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, and Ankle boots. Now that you learned some of the basics of machine learning concepts, keep working through the Microsoft Learn content to learn more about building different types of machine learning models with PyTorch.