02_Images and Convolutional Neural Networks

Content:

• Handling sequences with PyTorch

• Generating sequences

• Sequential Dataset

• Recurrent Neural Networks

• Sequential architectures

• Building a forecasting RNN

• LSTM and GRU cells

• RNN vs. LSTM vs. GRU

• LSTM network

• GRU network

• Training and evaluating RNNs

• RNN training loop

• Evaluating forecasting models

Sequential Data

• Order in time and space.

• Order of the data points contains dependencies between them.

• Example of sequential data:

  1. Time series.

  2. Text.

  3. Audio.

Electricity Consumption Prediction

• Task: predict future electricity consumption based on past patterns.

• Electricity consumption dataset:

Train-test Split

• No random splitting for time series!

• Look-ahead bias: model has info about the future.

• Solution: Split by time

Creating Sequences

• Sequence length = No. data points in one training example.

  • 24 x 4 = 96 => consider last 24 hour.

• Predict single next data point.

import numpy as np

def create_sequences(df, seq_length):

xs, ys = [], []

for i in range(len(df) - seq_length):

x = df.iloc[i:(i+seq_length), 1]

y = df.iloc[i+seq_length, 1]

xs.append(x)

ys.append(y)

TensorDataset

• Create training examples:

X_train, y_train = create_sequences(train_data, seq_length)

print(X_train.shape, y_train.shape)

# (34944, 96) (34944,)

• Convert to Torch dataset:

from torch.utils.data import TensorDataset

dataset_train = TensorDataset(

torch.from_numpy(X_train).float(),

torch.from_numpy(y_train).float(),

)

Recurrent Neural Network

class Net(nn.Module):

def __init__(self):

super().__init__()

self.rnn = nn.RNN(

input_size = 1,

hidden_size = 32,

num_layers = 2,

batch_first = True,

)

self.fc = nn.Linear(32,1)

def forward(self,x):

hθ = torch.zeros(2, x.size(θ), 32)

out, _ = self.rnn(x, hθ)

out = self.fc(out[:, -1, :])

return out

LSTM 

class Net(nn.Module):

def __init__(self):

super().__init__()

self.lstm = nn.LSTM(

input_size = 1,

hidden_size = 32,

num_layers = 2,

batch_first = True,

)

self.fc = nn.Linear(32,1)

def forward(self,x):

hθ = torch.zeros(2, x.size(θ), 32)

cθ = torch.zeros(2, x.size(θ), 32)

out, _ = self.lstm(x, (hθ,cθ))

out = self.fc(out[:, -1, :])

return out

GRU 

class Net(nn.Module):

def __init__(self):

super().__init__()

self.gru = nn.GRU(

input_size = 1,

hidden_size = 32,

num_layers = 2,

batch_first = True,

)

self.fc = nn.Linear(32,1)

def forward(self,x):

hθ = torch.zeros(2, x.size(θ), 32)

out, _ = self.gru(x, hθ)

out = self.fc(out[:, -1, :])

return out

Expanding and Squeezing Tensors

1. Expanding Tensor: Increasing the dimension of a tensor by adding new dimensions with size 1.

   • Compatibility with Convolutional Layers.

   • Compatibility with Recurrent Layers.

   • Broadcasting.

print(seqs.shape)

# [32, 96]

seqs = seqs.view(32,96,1)

print(seqs.shape)

# [32, 96, 1]

2. Squeezing Tensor: Removing dimensions with size 1 from a tensor

   • Removing Batch dimensions.

   • Working with Scalars.

print(seqs.shape)

# [32, 1]

seqs = seqs.squeeze()

print(seqs.shape)

# [32]

Training Loop

net = Net()

criterion = nn.MSELoss()

optimizer = optim.Adam(net.parameters(), lr = 0.001)

for epoch in range(num_epochs):

for seqs, labels in dataloader_train:

seqs = seqs.view(32, 96, 1)

outputs = net(seqs)

loss = criterion(outputs, labels)

optimizer.zero_grad()

loss.backward()

optimizer.step()

Evaluation Loop

mse = torchmetrics.MeanSquaredError()

net.eval()

with torch.no_grad():

for seqs, labels in test_loader:

seqs = seqs.view(32, 96, 1)

outputs = net(seqs).squeeeze()

mse(outputs, labels)

print(f"Test MSE: {mse.compute()}")

# Test MSE: 0.13292161