Convolutional Neural Networks (CNNs)

The purpose of this notebook is to practice implementing and training CNNs. We start with a 1-dimensional convolutional layer in NumPy.

We will then use PyTorch, an optimised machine learning framework for Python based on the torch library to implement machine learning architectures.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch

# Importing losses, activation functions and layers from PyTorch 
from torch.nn import Sequential, CrossEntropyLoss, Conv1d, MaxPool1d, Flatten, Linear, ReLU, Softmax, Parameter
from torch.utils.data import TensorDataset, DataLoader

# Changing default font sizes
plt.rc('xtick', labelsize=8) # Tick labels
plt.rc('ytick', labelsize=8) # Tick labels
plt.rc('legend', fontsize=10) # Legend
plt.rc('axes', titlesize=15, labelsize=10) # Title and 'x' and 'y' labels

You will be working with the Human Activity Recognition (HAR) Using Smartphones dataset. This consists of the readings from an accelerometer carried by a human doing different activities. The six activities are: walking horizontally, walking upstairs, walking downstairs, sitting, standing and laying down. The accelerometer is inside a smartphone, and it takes six readings every 0.02 seconds: linear and gyroscopic acceleration in the x, y and z directions.

The goal is to use the accelerometer data to predict the type of activity (a multi-class classification task).

1. Loading and plotting the data

x_train = np.load('./data/HAR/x_train.npy')
y_train = np.load('./data/HAR/y_train.npy')

x_val = np.load('./data/HAR/x_val.npy')
y_val = np.load('./data/HAR/y_val.npy')

x_test = np.load('./data/HAR/x_test.npy')
y_test = np.load('./data/HAR/y_test.npy')

The input data consists of 6 features over 128 time steps, and the output is a single integer from 0 to 5, which denotes the class.

There are 7352 examples in the training set, 2447 examples in the validation set and 500 in the test set. We can check the shapes ourselves.

print(x_train.shape)
print(y_train.shape)
print(x_val.shape)
print(y_val.shape)
print(x_test.shape)
print(y_test.shape)

(7352, 6, 128)
(7352, 1)
(2447, 6, 128)
(2447, 1)
(500, 6, 128)
(500, 1)

classes = [
    'Walking',
    'Walking upstairs',
    'Walking downstairs',
    'Sitting',
    'Standing',
    'Laying'
]

# Plot a randomly selected example from each class

for l, label in enumerate(range(len(classes))):
    inx = np.where(y_train[:, 0] == label)[0]
    i = np.random.choice(inx)
    x_example = x_train[i].T
    fig, ax = plt.subplots(figsize=(10, 1))
    ax.imshow(x_example.T, cmap='Greys', vmin=-1, vmax=1)
    ax.set_ylabel('Reading')
    ax.set_xlabel('Time step')
    ax.set_title(classes[l])

​

2. 1D convolutional layer in NumPy

Now we will implement a 1D convolutional layer in numpy. The following function is designed to perform a 1D convolution on an input signal, given weight and bias parameters. The layer should have no padding and a stride of 1. The output should consist of the pre-activations of the layer, meaning that no activation function is then applied.

Notation

• c_in and c_out (equal to n_filters) represent the number of input and output channels (respectively).
• l_in and l_out denote the length of the input and output signals (respectively).
• k is the length of the convolving kernel/filter.

For a convolutional (or max pool) layer, the input and output lengths are related as follows:

$$l_{out}=\lfloor \frac{l_{in}+2\times \text{padding}-\text{kernel\_size}}{\text{stride}}+1\rfloor$$

## EDIT THIS FUNCTION
def conv1d(x, weight, bias):
    """
    Performs a 1D convolution over an input signal.

    Parameters:
    x: Input signal of shape (batch_size, c_in, l_in)
    weight: Learnable weights, shape (c_out, c_in, k)
    bias: Bias parameters of size c_out

    Returns:
    An array of shape (batch_size, c_out, l_out)

    """
    batch_size = x.shape[0] ## <-- SOLUTION
    l_in = x.shape[2] ## <-- SOLUTION
    c_out = weight.shape[0] ## <-- SOLUTION
    k = weight.shape[2] ## <-- SOLUTION

    l_out = l_in - k + 1 ## <-- SOLUTION
    outputs = np.zeros((batch_size, c_out, l_out))
    np.testing.assert_allclose(l_out, 113)

    for i in range(l_out):
        outputs[:, :, i] = (x[:, np.newaxis, :, i:i+k] * weight).sum(axis=(2, 3)) + bias ## <-- SOLUTION
    return outputs

Now we can compare our layer with the PyTorch implementation.

# Test your layer
n_filters = 8
batch_size = 16
k = 16
l_in = 128

# PyTorch
conv_layer = Conv1d(x_train.shape[1], n_filters, k)
inputs = torch.randn((batch_size, x_train.shape[1], l_in))
y_torch = conv_layer(inputs)

# Our layer
y = conv1d(inputs.numpy(), conv_layer.weight.detach().numpy(), conv_layer.bias.detach().numpy())
np.allclose(y, y_torch.detach().numpy(), atol=1e-4)  ## <-- should be 'True'

True

3. Building a CNN in PyTorch

You should now build the CNN model in PyTorch to train on the HAR dataset. This model should consist of:

• A Conv1d layer with 8 filters, kernel size of 16 and a ReLU activation function
  • The input shape should be (6, 128)
• A MaxPoo1d layer with a pooling window size of 16 and a stride of 2
• A Flatten layer
• A Linear layer with 6 neurons and a Softmax activation

The function below should build and compile this model, using an Adam optimiser and a CrossEntropyLoss criterion.

def get_model(x_train, n_filters, k, pool_size, stride_pool, classes, l2_reg=1e-3):
    """
    CNN model in PyTorch:
    - Layers are Conv1d(+ReLU), MaxPool1d, Flatten and Linear(+Softmax).
    - It features an Adam optimiser and CrossEntropyLoss criterion.
    - Conv1d and Linear layers have regularised weights according to l2_reg.

    Parameters:
    x_train: Training data
    n_filters: Number of filters to be used in the convolutional layer
    k: Kernel size in the convolutional layer
    pool_size: MaxPool1d window size
    stride_pool: Stride of the MaxPool1d sliding window
    classes: List containing the output classes
    l2_reg: Positive float corresponding to the regularisation coefficient of Conv1d and Linear

    Returns:
    Model, criterion and optimiser.

    """

    l_out_conv = x_train.shape[2] - k + 1 # Length after Conv1d (note that the stride is 1) ## <-- SOLUTION
    l_out_pool = (l_out_conv - pool_size) // stride_pool + 1 # Length after MaxPool1d ## <-- SOLUTION
    l_in_linear = n_filters * l_out_pool # Size before Linear layer ## <-- SOLUTION
    np.testing.assert_allclose(l_in_linear, 392)

    model = Sequential(
        Conv1d(x_train.shape[1], n_filters, kernel_size=k),
        ReLU(), ## <-- SOLUTION
        MaxPool1d(kernel_size=pool_size, stride=stride_pool), ## <-- SOLUTION
        Flatten(), ## <-- SOLUTION
        Linear(l_in_linear, len(classes)), ## <-- SOLUTION
        Softmax(dim=1), ## <-- SOLUTION
    )

    # L2 regularisation
    for layer in model.children():
        if isinstance(layer, Conv1d) or isinstance(layer, Linear):
            layer.weight_regularizer = Parameter(l2_reg * torch.ones_like(layer.weight))

    criterion = CrossEntropyLoss() ## <-- SOLUTION
    optimiser = torch.optim.Adam(model.parameters()) ## <-- SOLUTION

    return model, criterion, optimiser

# Run your function to get the model and print it

n_filters = 8
k = 16
pool_size = 16
stride_pool = 2
l2_reg = 1e-3

model, criterion, optimiser = get_model(x_train, n_filters, k, pool_size, stride_pool, classes, l2_reg) ## <-- SOLUTION
print(model)

Sequential(
  (0): Conv1d(6, 8, kernel_size=(16,), stride=(1,))
  (1): ReLU()
  (2): MaxPool1d(kernel_size=16, stride=2, padding=0, dilation=1, ceil_mode=False)
  (3): Flatten(start_dim=1, end_dim=-1)
  (4): Linear(in_features=392, out_features=6, bias=True)
  (5): Softmax(dim=1)
)

4. Training a CNN

Now we are ready to train the model. The training_loop function should run the training for max_num_epochs of 200 and batch_size of 128, featuring early stopping to monitor the validation accuracy. Set the max_patience parameter to 5 epochs. The function should return the training and validation history, which includes the loss and accuracy.

# Numpy arrays to PyTorch tensors
x_train_tensor = torch.tensor(x_train, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train)
x_val_tensor = torch.tensor(x_val, dtype=torch.float32)
y_val_tensor = torch.tensor(y_val)

# Creating training and validation datasets
train_dataset = TensorDataset(x_train_tensor, y_train_tensor.squeeze())
val_dataset = TensorDataset(x_val_tensor, y_val_tensor.squeeze())

# Creating corresponding DataLoaders
train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=128, shuffle=False)

class EarlyStopping:
    """
    EarlyStopping class.

    Attributes:
    max_patience: Amount of epochs with no improvement after which training is stopped.
    patience: Stores the number of epochs with no improvement.
    best_valid_loss: Stores the current value of the best (minimum) validation loss.
    early_stop: True if training needs to be stopped due to the early stopping condition being met.

    Methods:
    step(val_loss):
        Checks current state after an epoch and updates best_loss, patience and early_stop accordingly.
    """

    def __init__(self, max_patience=5):
        self.max_patience = max_patience
        self.patience = 0
        self.best_valid_loss = float('inf') ## <-- SOLUTION
        self.early_stop = False

    def step(self, val_loss):
        if val_loss < self.best_valid_loss:  ## <-- SOLUTION
            self.best_valid_loss = val_loss  ## <-- SOLUTION
            self.patience = 0  ## <-- SOLUTION
        else:
            self.patience += 1
            if self.patience >= self.max_patience:
                self.early_stop = True

def training_loop(train_loader, val_loader, max_num_epochs=200, max_patience=5):
    """
    Training loop with early stopping to monitor the validation accuracy.

    Parameters:
    train_loader: Training DataLoader
    val_loader: Validation DataLoader
    max_num_epochs: Maximum number of epochs
    max_patience: max_patience attribute of the EarlyStopping class

    Returns:
    Model, criterion and optimiser.

    """

    history = {'training_loss': [], 'validation_loss': [], 'training_accuracy': [], 'validation_accuracy': []}
    early_stopping = EarlyStopping(max_patience=5)

    for epoch in range(max_num_epochs):

        tr_loss = 0.0
        tr_accuracy = 0
        val_loss = 0.0
        val_accuracy = 0

        # Training
        model.train()
        for inputs, labels in train_loader:
            optimiser.zero_grad() # Setting gradients to zero
            outputs = model(inputs)
            loss = criterion(outputs, labels) ## <-- SOLUTION
            tr_loss += loss.item()
            tr_accuracy += (torch.max(outputs, 1)[1] == labels).sum().item() / (len(train_loader) * labels.size(0))  ## <-- SOLUTION
            loss.backward() # Computes gradients of the loss
            optimiser.step() # Optimisation step (parameters are updated)

        history['training_loss'].append(tr_loss/len(train_loader))
        history['training_accuracy'].append(100*tr_accuracy)

        # Validation
        model.eval()
        with torch.no_grad():
            for inputs, labels in val_loader:
                outputs = model(inputs)
                loss = criterion(outputs, labels)  ## <-- SOLUTION
                val_loss += loss.item()
                val_accuracy += (torch.max(outputs, 1)[1] == labels).sum().item() / (len(val_loader) * labels.size(0))  ## <-- SOLUTION

        history['validation_loss'].append(val_loss/len(val_loader))
        history['validation_accuracy'].append(100*val_accuracy)

        # Calculate accuracy and print
        print(f"Epoch {epoch + 1}/{max_num_epochs}, Training loss: {tr_loss/len(train_loader)}, Training accuracy: {100*tr_accuracy}%, Validation loss: {val_loss/len(val_loader)}, Validation accuracy: {100*val_accuracy}%")

        # Check for early stopping
        early_stopping.step(val_loss / len(val_loader))
        if early_stopping.early_stop:
            print("Early stopping acting.")  ## <-- SOLUTION
            break  ## <-- SOLUTION

    return history

# Calling the training loop
max_num_epochs = 200
max_patience = 5

history = training_loop(train_loader, val_loader, max_num_epochs, max_patience)

Epoch 1/200, Training loss: 1.7121318414293487, Training accuracy: 30.34752155172414%, Validation loss: 1.6579939246177673, Validation accuracy: 45.6640625%
Epoch 2/200, Training loss: 1.5362633055654065, Training accuracy: 56.62715517241378%, Validation loss: 1.5142081260681153, Validation accuracy: 55.39062499999999%
Epoch 3/200, Training loss: 1.457470114888816, Training accuracy: 61.172259852216705%, Validation loss: 1.4736222505569458, Validation accuracy: 59.35677083333333%
Epoch 4/200, Training loss: 1.417420794223917, Training accuracy: 63.833512931034484%, Validation loss: 1.4364268779754639, Validation accuracy: 60.68489583333334%
Epoch 5/200, Training loss: 1.388573251921555, Training accuracy: 65.13046490147785%, Validation loss: 1.4158733427524566, Validation accuracy: 62.22916666666668%
Epoch 6/200, Training loss: 1.3700056261029736, Training accuracy: 67.11437807881775%, Validation loss: 1.3984680473804474, Validation accuracy: 62.97135416666667%
Epoch 7/200, Training loss: 1.3590136170387268, Training accuracy: 66.95658866995073%, Validation loss: 1.3878288209438323, Validation accuracy: 63.127604166666686%
Epoch 8/200, Training loss: 1.3516001372501767, Training accuracy: 68.26123768472905%, Validation loss: 1.37613445520401, Validation accuracy: 64.10416666666667%
Epoch 9/200, Training loss: 1.344404882398145, Training accuracy: 69.12523091133005%, Validation loss: 1.3671935498714447, Validation accuracy: 67.97135416666667%
Epoch 10/200, Training loss: 1.3371149774255424, Training accuracy: 71.12068965517244%, Validation loss: 1.3587223291397095, Validation accuracy: 69.10416666666667%
Epoch 11/200, Training loss: 1.3309343231135402, Training accuracy: 71.72105911330051%, Validation loss: 1.3578250110149384, Validation accuracy: 70.15885416666667%
Epoch 12/200, Training loss: 1.3236742245739903, Training accuracy: 73.12384544334977%, Validation loss: 1.3490940630435944, Validation accuracy: 70.47135416666667%
Epoch 13/200, Training loss: 1.316469132900238, Training accuracy: 73.51831896551727%, Validation loss: 1.3431268513202668, Validation accuracy: 71.40885416666667%
Epoch 14/200, Training loss: 1.3090957773142848, Training accuracy: 74.33805418719213%, Validation loss: 1.3374877393245697, Validation accuracy: 75.83072916666666%
Epoch 15/200, Training loss: 1.3022815276836526, Training accuracy: 76.62022783251233%, Validation loss: 1.3333993017673493, Validation accuracy: 76.3984375%
Epoch 16/200, Training loss: 1.2962056973884846, Training accuracy: 77.96143780788175%, Validation loss: 1.3244478344917296, Validation accuracy: 77.6484375%
Epoch 17/200, Training loss: 1.2900241490068107, Training accuracy: 78.54256465517241%, Validation loss: 1.3174363672733307, Validation accuracy: 78.41145833333334%
Epoch 18/200, Training loss: 1.2849656836739902, Training accuracy: 78.79656711822659%, Validation loss: 1.3200359404087068, Validation accuracy: 77.41666666666667%
Epoch 19/200, Training loss: 1.2789889286304343, Training accuracy: 79.89532019704434%, Validation loss: 1.3159491896629334, Validation accuracy: 79.01822916666667%
Epoch 20/200, Training loss: 1.2746241031021908, Training accuracy: 80.14739839901476%, Validation loss: 1.3115354359149933, Validation accuracy: 79.05729166666667%
Epoch 21/200, Training loss: 1.268436600422037, Training accuracy: 81.13262007389159%, Validation loss: 1.3029167354106903, Validation accuracy: 78.97916666666667%
Epoch 22/200, Training loss: 1.263866765745755, Training accuracy: 81.50400246305415%, Validation loss: 1.3027888059616088, Validation accuracy: 79.29166666666666%
Epoch 23/200, Training loss: 1.2596432443322807, Training accuracy: 82.23907019704434%, Validation loss: 1.2978974342346192, Validation accuracy: 79.36979166666666%
Epoch 24/200, Training loss: 1.2545871631852512, Training accuracy: 82.97221366995075%, Validation loss: 1.292573618888855, Validation accuracy: 79.99479166666666%
Epoch 25/200, Training loss: 1.2506435061323231, Training accuracy: 82.63354371921179%, Validation loss: 1.289669579267502, Validation accuracy: 80.54166666666664%
Epoch 26/200, Training loss: 1.2458197618352955, Training accuracy: 83.38977832512319%, Validation loss: 1.2845512449741363, Validation accuracy: 80.79687499999999%
Epoch 27/200, Training loss: 1.241009025738157, Training accuracy: 84.12869458128074%, Validation loss: 1.2831344127655029, Validation accuracy: 81.08854166666667%
Epoch 28/200, Training loss: 1.2372907441237877, Training accuracy: 84.38846982758619%, Validation loss: 1.2743450820446014, Validation accuracy: 82.41666666666667%
Epoch 29/200, Training loss: 1.2339484938259782, Training accuracy: 84.5597290640394%, Validation loss: 1.2788837432861329, Validation accuracy: 80.2890625%
Epoch 30/200, Training loss: 1.230920409334117, Training accuracy: 85.13700738916256%, Validation loss: 1.2676789581775665, Validation accuracy: 81.9296875%
Epoch 31/200, Training loss: 1.2266510951107945, Training accuracy: 85.69311884236454%, Validation loss: 1.2738824486732483, Validation accuracy: 81.96614583333334%
Epoch 32/200, Training loss: 1.2224209123644336, Training accuracy: 85.86437807881775%, Validation loss: 1.263941866159439, Validation accuracy: 82.39583333333333%
Epoch 33/200, Training loss: 1.2200829592244378, Training accuracy: 86.36853448275862%, Validation loss: 1.2582094848155976, Validation accuracy: 83.68750000000001%
Epoch 34/200, Training loss: 1.2159077052412361, Training accuracy: 86.52439963054186%, Validation loss: 1.256572663784027, Validation accuracy: 82.78645833333333%
Epoch 35/200, Training loss: 1.21336764302747, Training accuracy: 86.72452278325122%, Validation loss: 1.2526042759418488, Validation accuracy: 83.45052083333331%
Epoch 36/200, Training loss: 1.2097959169026078, Training accuracy: 86.951585591133%, Validation loss: 1.2550598978996277, Validation accuracy: 83.68750000000001%
Epoch 37/200, Training loss: 1.208302919206948, Training accuracy: 87.31142241379312%, Validation loss: 1.2496072232723237, Validation accuracy: 83.84375%
Epoch 38/200, Training loss: 1.204239874050535, Training accuracy: 87.94450431034484%, Validation loss: 1.2462993741035462, Validation accuracy: 83.70572916666667%
Epoch 39/200, Training loss: 1.2009871992571601, Training accuracy: 88.48522167487684%, Validation loss: 1.2450489103794098, Validation accuracy: 84.17447916666666%
Epoch 40/200, Training loss: 1.1983779607148006, Training accuracy: 88.55834359605912%, Validation loss: 1.246843844652176, Validation accuracy: 83.39322916666664%
Epoch 41/200, Training loss: 1.1961354099470993, Training accuracy: 88.71998152709357%, Validation loss: 1.2500394701957702, Validation accuracy: 82.86458333333331%
Epoch 42/200, Training loss: 1.194460455713601, Training accuracy: 88.68149630541872%, Validation loss: 1.2375515460968018, Validation accuracy: 84.29166666666667%
Epoch 43/200, Training loss: 1.1921263933181763, Training accuracy: 89.21836514778329%, Validation loss: 1.2429938435554504, Validation accuracy: 83.90104166666667%
Epoch 44/200, Training loss: 1.190616498733389, Training accuracy: 89.18757697044337%, Validation loss: 1.235194605588913, Validation accuracy: 84.78125000000001%
Epoch 45/200, Training loss: 1.187075908841758, Training accuracy: 89.370381773399%, Validation loss: 1.240226572751999, Validation accuracy: 83.78385416666664%
Epoch 46/200, Training loss: 1.1855433028319786, Training accuracy: 89.67633928571429%, Validation loss: 1.2349590957164764, Validation accuracy: 83.90104166666666%
Epoch 47/200, Training loss: 1.184152744967362, Training accuracy: 89.7205972906404%, Validation loss: 1.2351206958293914, Validation accuracy: 84.4296875%
Epoch 48/200, Training loss: 1.1808069451101895, Training accuracy: 89.8706896551724%, Validation loss: 1.2322359025478362, Validation accuracy: 84.390625%
Epoch 49/200, Training loss: 1.179458708598696, Training accuracy: 90.18819273399015%, Validation loss: 1.2292146563529969, Validation accuracy: 85.23177083333334%
Epoch 50/200, Training loss: 1.1773293121107693, Training accuracy: 90.21705665024628%, Validation loss: 1.2295194864273071, Validation accuracy: 85.09635416666667%
Epoch 51/200, Training loss: 1.1754406012337784, Training accuracy: 90.50184729064037%, Validation loss: 1.228010755777359, Validation accuracy: 84.41145833333333%
Epoch 52/200, Training loss: 1.1741080119692047, Training accuracy: 90.3960129310345%, Validation loss: 1.2280026853084565, Validation accuracy: 84.72135416666666%
Epoch 53/200, Training loss: 1.1731348202146332, Training accuracy: 90.83474445812806%, Validation loss: 1.2285800397396087, Validation accuracy: 84.44791666666667%
Epoch 54/200, Training loss: 1.1703439169916614, Training accuracy: 91.01754926108373%, Validation loss: 1.2257872700691224, Validation accuracy: 84.44791666666667%
Epoch 55/200, Training loss: 1.1694629829505394, Training accuracy: 91.41394704433499%, Validation loss: 1.2266152143478393, Validation accuracy: 84.48697916666666%
Epoch 56/200, Training loss: 1.169368433541265, Training accuracy: 90.72891009852216%, Validation loss: 1.2266651093959808, Validation accuracy: 84.50781250000001%
Epoch 57/200, Training loss: 1.1658331751823425, Training accuracy: 91.70451046798027%, Validation loss: 1.2231320858001709, Validation accuracy: 84.625%
Epoch 58/200, Training loss: 1.1654142490748702, Training accuracy: 91.53132697044336%, Validation loss: 1.226263427734375, Validation accuracy: 84.29427083333336%
Epoch 59/200, Training loss: 1.1627582743250091, Training accuracy: 91.61791871921184%, Validation loss: 1.2233623445034028, Validation accuracy: 85.03645833333331%
Epoch 60/200, Training loss: 1.1622245147310455, Training accuracy: 91.70643472906403%, Validation loss: 1.2226067543029786, Validation accuracy: 85.23177083333333%
Epoch 61/200, Training loss: 1.1605095226189186, Training accuracy: 92.01431650246303%, Validation loss: 1.2181911230087281, Validation accuracy: 84.8984375%
Epoch 62/200, Training loss: 1.1589927940533078, Training accuracy: 91.86229987684726%, Validation loss: 1.2235017716884613, Validation accuracy: 84.31250000000001%
Epoch 63/200, Training loss: 1.158447723964165, Training accuracy: 91.73529864532017%, Validation loss: 1.2224792778491973, Validation accuracy: 84.84114583333334%
Epoch 64/200, Training loss: 1.1564319421505105, Training accuracy: 91.92387623152709%, Validation loss: 1.2213149666786194, Validation accuracy: 84.1953125%
Epoch 65/200, Training loss: 1.156325584855573, Training accuracy: 91.9508158866995%, Validation loss: 1.2250536918640136, Validation accuracy: 84.4296875%
Epoch 66/200, Training loss: 1.154990816938466, Training accuracy: 92.17210591133004%, Validation loss: 1.218558484315872, Validation accuracy: 85.30989583333331%
Early stopping acting.

# Plot the learning curves

fig = plt
上一篇：0202矩阵的运算-矩阵及其运算-线性代数-四、矩阵的转置

下一篇：Python爬虫之XPath和Beautiful Soup的使用-二、Beautiful Soup的使用