01_Training Robust Neural Network

Content:

• Pytorch and OOP.

• Dataset.

• DataLoader.

• Model.

• Optimizer, Training, and Evaluation.

• Training Loop.

• Vanishing and Exploding Gradients.

• Initialization and Activation.

• Batch Normalization

Object-Oriented Programming (OOP):

• Create objects with:

1. Abilities (methods)

2. Data (attributes)

class BankAccount:

def __init__(self, balance): #__init__ is called when BankAccount is created; balance is attribute of the BankAccount object

self.balance = balance 

def deposit(self, amount): #methods

self.balance += amount

Pytorch Dataset

1. init: load data, store as numpy array

   • super().__init__(): ensures WaterDataset behaves like torch Dataset

2. len: return the size of the dataset

3. getitem: take one argument called idx and return features and label for a single sample at index idx.

from torch.utils.data import Dataset

class WaterDataset(Dataset):

def __init__(self, csv_path):

super().__init__()

df = pd.read_csv(csv_path)

self.data = df.to_numpy()

def len(self):

return self.data.shape[0]

def __getitem__(self, idx):

features = self.data[idx, :-1]

label = self.data[idx, -1]

return features, label

Pytorch DataLoader

dataset_train = WaterDataset("water_train.csv")

from torch.utils.data import DataLoader

dataloader_train = DataLoader(

dataset_train,

batch_size = 2,

shuffle = True,

)

features, labels = next(iter(dataloader_train))

print(f"Features: {features}, \n labels: {labels}")

Sequential()

net = Sequential(

nn.Linear(9, 16),

nn.ELU(),

nn.Linear(16, 8),

nn.ELU(),

nn.Linear(8, 1),

nn.Sigmoid(),

)

Class-based Model Definition

class Net(nn.Module):

def __init__(self):

super(Net, self).__init__()

self.fc1 = nn.Linear(9, 16)

self.fc2 = nn.Linear(16, 8)

self.fc3 = nn.Linear(8, 1)

def forward(self, x):

x = nn.functional.relu(self.fc1(x))

x = nn.functional.relu(self.fc2(x))

x = nn.functional.sigmoid(self.fc3(x))

return x

net = Net()

Optimizer

from torch.optim import optim

optimizer = optim.SGD(net.parameters(), lr = 0.01)

#optimizer = optim.Adamgrad(net.parameters(), lr = 0.01) 

#optimizer = optim.RMSprop(net.parameters(), lr = 0.01)

#optimizer = optim.Adam(net.parameters(), lr = 0.01) 

Model Evaluation

from torchmetrics import Accuracy

acc = Accuracy(task = "binary")

net.eval()

with torch.no_grad():

for features, labels in dataloader_test:

outputs = net(features)

preds = (outputs > 0.5).float()

acc(preds, labels.view(-1, 1))

accuracy = acc.compute()

print(f*Accuracy: {accuracy}")

Unstable Gradients

1. Proper weights initialization.

   • Variance of layer inputs = variance of layer outputs.

   • Variance of gradients the same before and after a layer.

init.kaiming_uniform_(self.fc1.weight)

2. Good activations.

   • Relu(): Zero for negative inputs (dying neurons)

nn.functional.relu()

2. • Elu(): Non-zero gradients fro negative values - help against dying neurons. Average output around zero - helps against vanishing neurons.

nn.funtional.elu()

3. Batch normalization.

nn.BatchNorm1d(features)