计算机视觉学习记录 - Implementing a Neural Network from Scratch - An Introduction

2022-03-05 00:57:36

0 - 学习目标

　　我们将实现一个简单的3层神经网络，我们不会仔细推到所需要的数学公式，但我们会给出我们这样做的直观解释。注意，此次代码并不能达到非常好的效果，可以自己进一步调整或者完成课后练习来进行改进。

1 - 实验步骤

1.1 - Import Packages

# Package imports

import matplotlib.pyplot as plt

import numpy as np

import sklearn

import sklearn.datasets

import sklearn.linear_model

import matplotlib

# Display plots inline and change default figure size

%matplotlib inline

matplotlib.rcParams['figure.figsize'] = (10.0, 8.0) # 指定matplotlib画布规模

1.2 - Generating a dataset

　　注意到，scikit-learn包包含了数据生成的代码，因此我们无需自己实现，直接采用其make_moons方法即可。下图中有两种类别的点，蓝点表示男患者，红点表示女患者，而xy坐标表示医学测量指标。我们的目的是去训练一个模型可以根据医学测量指标结果来划分男女患者，注意到图中的划分界限不是简单的线性的，因此采用简单的逻辑回归的效果合理不会很好。

# Generate a dataset and plot it

np.random.seed()

X, y = sklearn.datasets.make_moons(, noise=0.20)

plt.scatter(X[:,], X[:,], s=, c=y, cmap=plt.cm.Spectral)

1.3 - Logistic Regression

　　为了证明上述观点我们来训练一个逻辑回归模型看看效果。输入是xy坐标，而输出是(0,1)二分类。我们直接使用scikit-learn包中的逻辑回归算法做预测。

# Train the logistic regression classifier

clf = sklearn.linear_model.LogisticRegressionCV()

clf.fit(X, y)

Out[]:

LogisticRegressionCV(Cs=, class_weight=None, cv=None, dual=False,

           fit_intercept=True, intercept_scaling=1.0, max_iter=,

           multi_class='ovr', n_jobs=, penalty='l2', random_state=None,

           refit=True, scoring=None, solver='lbfgs', tol=0.0001, verbose=)

# Helper function to plot a decision boundary.

# If you don't fully understand this function don't worry, it just generates the contour plot below.

def plot_decision_boundary(pred_func):

    # Set min and max values and give it some padding

    x_min, x_max = X[:, ].min() - ., X[:, ].max() + .

    y_min, y_max = X[:, ].min() - ., X[:, ].max() + .

    h = 0.01

    # Generate a grid of points with distance h between them

    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    # Predict the function value for the whole gid

    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])

    Z = Z.reshape(xx.shape)

    # Plot the contour and training examples

    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)

    plt.scatter(X[:, ], X[:, ], c=y, cmap=plt.cm.Spectral)

# Plot the decision boundary

plot_decision_boundary(lambda x: clf.predict(x))

plt.title("Logistic Regression")

　　可以看到，逻辑回归使用一条直线尽可能好的分割这个二分类问题，但是由于原先数据本就不是线性可分的，因此效果并不好。

1.4 - Training a Neural Network

　　现在来构建一个有一个输入层一个隐藏层和一个输出层的简单三层神经网络来做预测。

1.4.1 - How our network makes predictions

　　神经网络通过下述公式进行预测。

$$
\begin{aligned}
z_1 & = xW_1 + b_1 \\
a_1 & = \tanh(z_1) \\
z_2 & = a_1W_2 + b_2 \\
a_2 & = \hat{y} = \mathrm{softmax}(z_2)
\end{aligned}
$$

1.4.2 - Learning the Parameters

　　学习参数是让我们的网络找到一组参数 ($W_1, b_1, W_2, b_2$)使得训练集上的损失最小化。现在我们来定义损失函数，这里我们使用常用的交叉熵损失函数，如下：

$$
\begin{aligned}
L(y,\hat{y}) = - \frac{1}{N} \sum_{n \in N} \sum_{i \in C} y_{n,i} \log\hat{y}_{n,i}
\end{aligned}
$$

　　而后我们使用梯度下降来最小化损失函数。我们将实现最简单的梯度下降算法，其实就是有着固定学习率的批量梯度下降。在实践中，梯度下降的一些变种如SGD（随机梯度下降）或者最小批次梯度下降往往有更好的表现。因此后续我们可以通过这些点来改进效果。

　　梯度下降需要计算出损失函数相对于我们要更新参数的梯度 $\frac{\partial{L}}{\partial{W_1}}$, $\frac{\partial{L}}{\partial{b_1}}$, $\frac{\partial{L}}{\partial{W_2}}$, $\frac{\partial{L}}{\partial{b_2}}$。为了计算这些梯度我们使用著名的反向传播算法，这种方法能够从输出开始有效地计算梯度。此处不细讲反向传播是如何工作的，只给出方向传播需要的公式，如下：

$$
\begin{aligned}
& \delta_3 = \hat{y} - y \\
& \delta_2 = (1 - \tanh^2z_1) \circ \delta_3W_2^T \\
& \frac{\partial{L}}{\partial{W_2}} = a_1^T \delta_3 \\
& \frac{\partial{L}}{\partial{b_2}} = \delta_3\\
& \frac{\partial{L}}{\partial{W_1}} = x^T \delta_2\\
& \frac{\partial{L}}{\partial{b_1}} = \delta_2 \\
\end{aligned}
$$

1.4.3 - Implementation

　　开始实现！

　　变量定义。

num_examples = len(X) # 训练集大小

nn_input_dim =  # 输入层维度

nn_output_dim =  # 输出层维度

# Gradient descent parameters (I picked these by hand)

epsilon = 0.01 # 梯度下降学习率

reg_lambda = 0.01 # 正规化权重

　　损失函数定义。

# Helper function to evaluate the total loss on the dataset

def calculate_loss(model):

    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']

    # 前向传播，计算出预测值

    z1 = X.dot(W1) + b1

    a1 = np.tanh(z1)

    z2 = a1.dot(W2) + b2

    exp_scores = np.exp(z2)

    probs = exp_scores / np.sum(exp_scores, axis=, keepdims=True)

    # 计算损失

    corect_logprobs = -np.log(probs[range(num_examples), y])

    data_loss = np.sum(corect_logprobs)

    # 损失值加入正规化

    data_loss += reg_lambda/ * (np.sum(np.square(W1)) + np.sum(np.square(W2)))

    return ./num_examples * data_loss

　　我们也实现了一个有用的用来计算网络输出的方法，其做了前向传播计算并且返回最高概率类别。

# Helper function to predict an output ( or )

def predict(model, x):

    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']

    # Forward propagation

    z1 = x.dot(W1) + b1

    a1 = np.tanh(z1)

    z2 = a1.dot(W2) + b2

    exp_scores = np.exp(z2)

    probs = exp_scores / np.sum(exp_scores, axis=, keepdims=True)

    return np.argmax(probs, axis=)

　　最后，使用批量梯度下降算法来训练我们的神经网络。

# This function learns parameters for the neural network and returns the model.

# - nn_hdim: Number of nodes in the hidden layer

# - num_passes: Number of passes through the training data for gradient descent

# - print_loss: If True, print the loss every  iterations

def build_model(nn_hdim, num_passes=, print_loss=False):

    # 随机初始化权重

    np.random.seed()

    W1 = np.random.randn(nn_input_dim, nn_hdim) / np.sqrt(nn_input_dim)

    b1 = np.zeros((, nn_hdim))

    W2 = np.random.randn(nn_hdim, nn_output_dim) / np.sqrt(nn_hdim)

    b2 = np.zeros((, nn_output_dim))

    # 返回字典初始化

    model = {}

    # 对于每一个批次进行梯度下降

    for i in range(, num_passes):

        # 前向传播

        z1 = X.dot(W1) + b1

        a1 = np.tanh(z1)

        z2 = a1.dot(W2) + b2

        exp_scores = np.exp(z2)

        probs = exp_scores / np.sum(exp_scores, axis=, keepdims=True)

        # 反向传播

        delta3 = probs

        delta3[range(num_examples), y] -=

        dW2 = (a1.T).dot(delta3)

        db2 = np.sum(delta3, axis=, keepdims=True)

        delta2 = delta3.dot(W2.T) * ( - np.power(a1, ))

        dW1 = np.dot(X.T, delta2)

        db1 = np.sum(delta2, axis=)

        # 加入正则化

        dW2 += reg_lambda * W2

        dW1 += reg_lambda * W1

        # 梯度下降参数更新

        W1 += -epsilon * dW1

        b1 += -epsilon * db1

        W2 += -epsilon * dW2

        b2 += -epsilon * db2

        # 分配新权重

        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

        # Optionally print the loss.

        # This is expensive because it uses the whole dataset, so we don't want to do it too often.

        if print_loss and i %  == :

          print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model

1.4.4 - A network with a hidden layer of size 3

# Build a model with a -dimensional hidden layer

model = build_model(, print_loss=True)

# Plot the decision boundary

plot_decision_boundary(lambda x: predict(model, x))

plt.title("Decision Boundary for hidden layer size 3")

Loss after iteration : 0.432387

Loss after iteration : 0.068947

Loss after iteration : 0.068926

Loss after iteration : 0.071218

Loss after iteration : 0.071253

Loss after iteration : 0.071278

Loss after iteration : 0.071293

Loss after iteration : 0.071303

Loss after iteration : 0.071308

Loss after iteration : 0.071312

Loss after iteration : 0.071314

Loss after iteration : 0.071315

Loss after iteration : 0.071315

Loss after iteration : 0.071316

Loss after iteration : 0.071316

Loss after iteration : 0.071316

Loss after iteration : 0.071316

Loss after iteration : 0.071316

Loss after iteration : 0.071316

Loss after iteration : 0.071316

　　这看起来比逻辑回归的效果好多了！

1.5 - Varying the hidden layer size

plt.figure(figsize=(, ))

hidden_layer_dimensions = [, , , , , , ]

for i, nn_hdim in enumerate(hidden_layer_dimensions):

    plt.subplot(, , i+)

    plt.title('Hidden Layer size %d' % nn_hdim)

    model = build_model(nn_hdim)

    plot_decision_boundary(lambda x: predict(model, x))

plt.show()

2 - Exercises

　　我们给出了一些练习。

Instead of batch gradient descent, use minibatch gradient descent (more info) to train the network. Minibatch gradient descent typically performs better in practice.
We used a fixed learning rate $\epsilon$ for gradient descent. Implement an annealing schedule for the gradient descent learning rate (more info).
We used a $\tanh$ activation function for our hidden layer. Experiment with other activation functions (some are mentioned above). Note that changing the activation function also means changing the backpropagation derivative.
Extend the network from two to three classes. You will need to generate an appropriate dataset for this.
Extend the network to four layers. Experiment with the layer size. Adding another hidden layer means you will need to adjust both the forward propagation as well as the backpropagation code.

3 - Exercises (1)

　　采用随机梯度下降方法，由于本题我是在第五题之后写的，因此采用的还是第五题构造的四层网络结构。

　　重写build_model，如下：

# This function learns parameters for the neural network and returns the model.

# - nn_hdim1: Number of nodes in the first hidden layer

# - nn_hidm2: Number of nodes in the second hidden layer(default 3)

# - m: Size of minibatch

# - num_passes: Number of passes through the training data for gradient descent

# - print_loss: If True, print the loss every 1000 iterations

# - d: the decay number of annealing schedule

def build_model(nn_hdim1, nn_hdim2=3, m=128, num_passes=20000, print_loss=False, d=10e-3):

    # Initialize the parameters to random values. We need to learn these.

    np.random.seed(0)

    W1 = np.random.randn(nn_input_dim, nn_hdim1) / np.sqrt(nn_input_dim)

    b1 = np.zeros((1, nn_hdim1))

    W2 = np.random.randn(nn_hdim1, nn_hdim2) / np.sqrt(nn_hdim1)

    b2 = np.zeros((1, nn_hdim2))

    W3 = np.random.randn(nn_hdim2, nn_output_dim) / np.sqrt(nn_hdim2)

    b3 = np.zeros((1, nn_output_dim))

    # This is what we return at the end

    model = {}

    # Gradient descent. For each batch...

    for i in range(0, num_passes):

        X_length = len(X)

        import random

        index = random.sample(range(0,X_length), m)

        X_ = X[index]

        y_ = y[index]

        # Forward propagation

        z1 = X_.dot(W1) + b1

        a1 = sigmoid(z1)

        z2 = a1.dot(W2) + b2

        a2 = sigmoid(z2)

        z3 = a2.dot(W3) + b3

        exp_scores = np.exp(z3)

        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        # Backpropagation

        delta4 = probs

        delta4[range(m), y_] -= 1

        dW3 = (a2.T).dot(delta4)

        db3 = np.sum(delta4, axis=0, keepdims=True)

        delta3 = delta4.dot(W3.T) * a2 * (1-a2)

        dW2 = np.dot(a1.T, delta3)

        db2 = np.sum(delta3, axis=0)

        delta2 = delta3.dot(W2.T) * a1 * (1-a1)

        dW1 = np.dot(X_.T, delta2)

        db1 = np.sum(delta2, axis=0)

        # Add regularization terms (b1 and b2 don't have regularization terms)

        dW3 += reg_lambda * W3

        dW2 += reg_lambda * W2

        dW1 += reg_lambda * W1

        epsilon_ = epsilon / (1+d*i)

        # Gradient descent parameter update

        W1 += -epsilon_ * dW1

        b1 += -epsilon_ * db1

        W2 += -epsilon_ * dW2

        b2 += -epsilon_ * db2

        W3 += -epsilon_ * dW3

        b3 += -epsilon_ * db3

        # Assign new parameters to the model

        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2, 'W3': W3, 'b3': b3}

        # Optionally print the loss.

        # This is expensive because it uses the whole dataset, so we don't want to do it too often.

        if print_loss and i % 1000 == 0:

          print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model

结果：

Loss after iteration 0: 0.690824

Loss after iteration 1000: 0.303296

Loss after iteration 2000: 0.282995

Loss after iteration 3000: 0.255683

Loss after iteration 4000: 0.229780

Loss after iteration 5000: 0.208478

Loss after iteration 6000: 0.191594

Loss after iteration 7000: 0.178304

Loss after iteration 8000: 0.167675

Loss after iteration 9000: 0.159266

Loss after iteration 10000: 0.152439

Loss after iteration 11000: 0.146777

Loss after iteration 12000: 0.142068

Loss after iteration 13000: 0.138093

Loss after iteration 14000: 0.134683

Loss after iteration 15000: 0.131718

Loss after iteration 16000: 0.129113

Loss after iteration 17000: 0.126861

Loss after iteration 18000: 0.124822

Loss after iteration 19000: 0.123014

　　可以看到，当随机梯度的$minibatch$的大小设置为128时，效果并不如批量梯度下降。但是$minibatch$的效率明显比批量梯度下降高，而且如果进行更多轮的迭代效果也很有可能变好，这也是现在训练神经网络更多的采用$minibatch$梯度下降的原因。

4 - Exercises (2)

　　使用模拟退火算法更新学习率，公式为$epsilon=\frac{epsilon_0}{1+d \times t}$。

# This function learns parameters for the neural network and returns the model.

# - nn_hdim: Number of nodes in the hidden layer

# - num_passes: Number of passes through the training data for gradient descent

# - print_loss: If True, print the loss every  iterations

# - d: the decay number of annealing schedule

def build_model(nn_hdim, num_passes=, print_loss=False, d=10e-):

    # Initialize the parameters to random values. We need to learn these.

    np.random.seed()

    W1 = np.random.randn(nn_input_dim, nn_hdim) / np.sqrt(nn_input_dim)

    b1 = np.zeros((, nn_hdim))

    W2 = np.random.randn(nn_hdim, nn_output_dim) / np.sqrt(nn_hdim)

    b2 = np.zeros((, nn_output_dim))

    # This is what we return at the end

    model = {}

    # Gradient descent. For each batch...

    for i in range(, num_passes):

        # Forward propagation

        z1 = X.dot(W1) + b1

        a1 = np.tanh(z1)

        z2 = a1.dot(W2) + b2

        exp_scores = np.exp(z2)

        probs = exp_scores / np.sum(exp_scores, axis=, keepdims=True)

        # Backpropagation

        delta3 = probs

        delta3[range(num_examples), y] -=

        dW2 = (a1.T).dot(delta3)

        db2 = np.sum(delta3, axis=, keepdims=True)

        delta2 = delta3.dot(W2.T) * ( - np.power(a1, ))

        dW1 = np.dot(X.T, delta2)

        db1 = np.sum(delta2, axis=)

        # Add regularization terms (b1 and b2 don't have regularization terms)

        dW2 += reg_lambda * W2

        dW1 += reg_lambda * W1

        epsilon_ = epsilon / (+d*i)

        # Gradient descent parameter update

        W1 += -epsilon_ * dW1

        b1 += -epsilon_ * db1

        W2 += -epsilon_ * dW2

        b2 += -epsilon_ * db2

        # Assign new parameters to the model

        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

        # Optionally print the loss.

        # This is expensive because it uses the whole dataset, so we don't want to do it too often.

        if print_loss and i %  == :

          print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model

Loss after iteration : 0.432387

Loss after iteration : 0.081007

Loss after iteration : 0.075384

Loss after iteration : 0.073729

Loss after iteration : 0.072895

Loss after iteration : 0.072376

Loss after iteration : 0.072013

Loss after iteration : 0.071742

Loss after iteration : 0.071530

Loss after iteration : 0.071357

Loss after iteration : 0.071214

Loss after iteration : 0.071092

Loss after iteration : 0.070986

Loss after iteration : 0.070894

Loss after iteration : 0.070812

Loss after iteration : 0.070739

Loss after iteration : 0.070673

Loss after iteration : 0.070613

Loss after iteration : 0.070559

Loss after iteration : 0.070509

4 - Exercises (3)

　　改用激活函数，由于本题我是在第五题之后写的，因此采用的还是第五题构造的四层网络结构，我决定采用$sigmoid$函数替代$tanh$函数做激活函数，其介绍可见于我的另一篇博客sigmoid理解。因此，反向传播公式需要重新推导，结果如下：

$$
\begin{aligned}
& \delta_4 = \hat{y} - y \\
& \delta_3 = sigmoid(x)(1-sigmoid(x)) \circ \delta_4W_3^T \\
& \delta_2 = sigmoid(x)(1-sigmoid(x)) \circ \delta_3W_2^T \\
& \frac{\partial{L}}{\partial{W_3}} = a_2^T \delta_4 \\
& \frac{\partial{L}}{\partial{b_3}} = \delta_4\\
& \frac{\partial{L}}{\partial{W_2}} = a_1^T \delta_3\\
& \frac{\partial{L}}{\partial{b_2}} = \delta_3 \\
& \frac{\partial{L}}{\partial{W_1}} = x^T \delta_2 \\
& \frac{\partial{L}}{\partial{b_1}} = \delta_2 \\
\end{aligned}
$$

　　定义$sigmoid$函数，如下：

def sigmoid(x):

    return 1 / (1 + np.exp(-x))

　　重写calculate_loss方法，如下：

# Helper function to evaluate the total loss on the dataset

def calculate_loss(model):

    W1, b1, W2, b2, W3, b3 = model['W1'], model['b1'], model['W2'], model['b2'], model['W3'], model['b3']

    # Forward propagation to calculate our predictions

    z1 = X.dot(W1) + b1

    a1 = sigmoid(z1)

    z2 = a1.dot(W2) + b2

    a2 = sigmoid(z2)

    z3 = a2.dot(W3) + b3

    exp_scores = np.exp(z3)

    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

    # Calculating the loss

    corect_logprobs = -np.log(probs[range(num_examples), y])

    data_loss = np.sum(corect_logprobs)

    # Add regulatization term to loss (optional)

    data_loss += reg_lambda/2 * (np.sum(np.square(W1)) + np.sum(np.square(W2)))

    return 1./num_examples * data_loss

　　重写predict方法，如下：

# Helper function to predict an output (0 or 1)

def predict(model, x):

    W1, b1, W2, b2, W3, b3 = model['W1'], model['b1'], model['W2'], model['b2'], model['W3'], model['b3']

    # Forward propagation

    z1 = x.dot(W1) + b1

    a1 = sigmoid(z1)

    z2 = a1.dot(W2) + b2

    a2 = sigmoid(z2)

    z3 = a2.dot(W3) + b3

    exp_scores = np.exp(z3)

    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

    return np.argmax(probs, axis=1)

　　重写build_model方法，如下：

# This function learns parameters for the neural network and returns the model.

# - nn_hdim1: Number of nodes in the first hidden layer

# - nn_hidm2: Number of nodes in the second hidden layer(default 3)

# - num_passes: Number of passes through the training data for gradient descent

# - print_loss: If True, print the loss every 1000 iterations

# - d: the decay number of annealing schedule

def build_model(nn_hdim1, nn_hdim2=3, num_passes=20000, print_loss=False, d=10e-3):

    # Initialize the parameters to random values. We need to learn these.

    np.random.seed(0)

    W1 = np.random.randn(nn_input_dim, nn_hdim1) / np.sqrt(nn_input_dim)

    b1 = np.zeros((1, nn_hdim1))

    W2 = np.random.randn(nn_hdim1, nn_hdim2) / np.sqrt(nn_hdim1)

    b2 = np.zeros((1, nn_hdim2))

    W3 = np.random.randn(nn_hdim2, nn_output_dim) / np.sqrt(nn_hdim2)

    b3 = np.zeros((1, nn_output_dim))

    # This is what we return at the end

    model = {}

    # Gradient descent. For each batch...

    for i in range(0, num_passes):

        # Forward propagation

        z1 = X.dot(W1) + b1

        a1 = sigmoid(z1)

        z2 = a1.dot(W2) + b2

        a2 = sigmoid(z2)

        z3 = a2.dot(W3) + b3

        exp_scores = np.exp(z3)

        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        # Backpropagation

        delta4 = probs

        delta4[range(num_examples), y] -= 1

        dW3 = (a2.T).dot(delta4)

        db3 = np.sum(delta4, axis=0, keepdims=True)

        delta3 = delta4.dot(W3.T) * a2 * (1-a2)

        dW2 = np.dot(a1.T, delta3)

        db2 = np.sum(delta3, axis=0)

        delta2 = delta3.dot(W2.T) * a1 * (1-a1)

        dW1 = np.dot(X.T, delta2)

        db1 = np.sum(delta2, axis=0)

        # Add regularization terms (b1 and b2 don't have regularization terms)

        dW3 += reg_lambda * W3

        dW2 += reg_lambda * W2

        dW1 += reg_lambda * W1

        epsilon_ = epsilon / (1+d*i)

        # Gradient descent parameter update

        W1 += -epsilon_ * dW1

        b1 += -epsilon_ * db1

        W2 += -epsilon_ * dW2

        b2 += -epsilon_ * db2

        W3 += -epsilon_ * dW3

        b3 += -epsilon_ * db3

        # Assign new parameters to the model

        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2, 'W3': W3, 'b3': b3}

        # Optionally print the loss.

        # This is expensive because it uses the whole dataset, so we don't want to do it too often.

        if print_loss and i % 1000 == 0:

          print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model

结果：

Loss after iteration 0: 0.685280

Loss after iteration 1000: 0.221651

Loss after iteration 2000: 0.140964

Loss after iteration 3000: 0.117482

Loss after iteration 4000: 0.107298

Loss after iteration 5000: 0.101436

Loss after iteration 6000: 0.097515

Loss after iteration 7000: 0.094664

Loss after iteration 8000: 0.092479

Loss after iteration 9000: 0.090744

Loss after iteration 10000: 0.089330

Loss after iteration 11000: 0.088153

Loss after iteration 12000: 0.087156

Loss after iteration 13000: 0.086301

Loss after iteration 14000: 0.085559

Loss after iteration 15000: 0.084907

Loss after iteration 16000: 0.084331

Loss after iteration 17000: 0.083816

Loss after iteration 18000: 0.083354

Loss after iteration 19000: 0.082936

　　由上可以看到，换成$sigmoid$激活函数的效果其实反而不如$tanh$激活函数。但现在神经网络常用的还是$sigmoid$激活函数，具体原因后续我将再写博客进行分析。总之不能以偏概全，在这个小实验中还有很多其他的参数也会影响到结果，因此不能绝对的说$sigmoid$就必定好于$tanh$！

5 - Exercises (4)

　　将网络的识别类别从两类扩充到三类，只需要修改变量nn_output_dim为3即可。由于没有找到合适的三分类数据集，因此不做测试。

6 - Exercises (5)

　　将网络扩展到四层，则反向传播公式如下（其中关于tanh激活函数见于我的另一篇博客tanh理解，DNN的反向传播算法理解见于我的另一篇博客DNN反向传播）：

$$
\begin{aligned}
& \delta_4 = \hat{y} - y \\
& \delta_3 = (1 - \tanh^2z_2) \circ \delta_4W_3^T \\
& \delta_2 = (1 - \tanh^2z_1) \circ \delta_3W_2^T \\
& \frac{\partial{L}}{\partial{W_3}} = a_2^T \delta_4 \\
& \frac{\partial{L}}{\partial{b_3}} = \delta_4\\
& \frac{\partial{L}}{\partial{W_2}} = a_1^T \delta_3\\
& \frac{\partial{L}}{\partial{b_2}} = \delta_3 \\
& \frac{\partial{L}}{\partial{W_1}} = x^T \delta_2 \\
& \frac{\partial{L}}{\partial{b_1}} = \delta_2 \\
\end{aligned}
$$

　　重写calculate_loss方法如下：

# Helper function to evaluate the total loss on the dataset

def calculate_loss(model):

    W1, b1, W2, b2, W3, b3 = model['W1'], model['b1'], model['W2'], model['b2'], model['W3'], model['b3']

    # Forward propagation to calculate our predictions

    z1 = X.dot(W1) + b1

    a1 = np.tanh(z1)

    z2 = a1.dot(W2) + b2

    a2 = np.tanh(z2)

    z3 = a2.dot(W3) + b3

    exp_scores = np.exp(z3)

    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

    # Calculating the loss

    corect_logprobs = -np.log(probs[range(num_examples), y])

    data_loss = np.sum(corect_logprobs)

    # Add regulatization term to loss (optional)

    data_loss += reg_lambda/2 * (np.sum(np.square(W1)) + np.sum(np.square(W2)))

    return 1./num_examples * data_loss

　　重写predict方法如下：

# Helper function to predict an output (0 or 1)

def predict(model, x):

    W1, b1, W2, b2, W3, b3 = model['W1'], model['b1'], model['W2'], model['b2'], model['W3'], model['b3']

    # Forward propagation

    z1 = x.dot(W1) + b1

    a1 = np.tanh(z1)

    z2 = a1.dot(W2) + b2

    a2 = np.tanh(z2)

    z3 = a2.dot(W3) + b3

    exp_scores = np.exp(z3)

    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

    return np.argmax(probs, axis=1)

　　重写build_model方法如下：

# This function learns parameters for the neural network and returns the model.

# - nn_hdim1: Number of nodes in the first hidden layer

# - nn_hidm2: Number of nodes in the second hidden layer(default 3)

# - num_passes: Number of passes through the training data for gradient descent

# - print_loss: If True, print the loss every 1000 iterations

# - d: the decay number of annealing schedule

def build_model(nn_hdim1, nn_hdim2=3, num_passes=20000, print_loss=False, d=10e-3):

    # Initialize the parameters to random values. We need to learn these.

    np.random.seed(0)

    W1 = np.random.randn(nn_input_dim, nn_hdim1) / np.sqrt(nn_input_dim)

    b1 = np.zeros((1, nn_hdim1))

    W2 = np.random.randn(nn_hdim1, nn_hdim2) / np.sqrt(nn_hdim1)

    b2 = np.zeros((1, nn_hdim2))

    W3 = np.random.randn(nn_hdim2, nn_output_dim) / np.sqrt(nn_hdim2)

    b3 = np.zeros((1, nn_output_dim))

    # This is what we return at the end

    model = {}

    # Gradient descent. For each batch...

    for i in range(0, num_passes):

        # Forward propagation

        z1 = X.dot(W1) + b1

        a1 = np.tanh(z1)

        z2 = a1.dot(W2) + b2

        a2 = np.tanh(z2)

        z3 = a2.dot(W3) + b3

        exp_scores = np.exp(z3)

        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        # Backpropagation

        delta4 = probs

        delta4[range(num_examples), y] -= 1

        dW3 = (a2.T).dot(delta4)

        db3 = np.sum(delta4, axis=0, keepdims=True)

        delta3 = delta4.dot(W3.T) * (1 - np.power(a2, 2))

        dW2 = np.dot(a1.T, delta3)

        db2 = np.sum(delta3, axis=0)

        delta2 = delta3.dot(W2.T) * (1 - np.power(a1, 2))

        dW1 = np.dot(X.T, delta2)

        db1 = np.sum(delta2, axis=0)

        # Add regularization terms (b1 and b2 don't have regularization terms)

        dW3 += reg_lambda * W3

        dW2 += reg_lambda * W2

        dW1 += reg_lambda * W1

        epsilon_ = epsilon / (1+d*i)

        # Gradient descent parameter update

        W1 += -epsilon_ * dW1

        b1 += -epsilon_ * db1

        W2 += -epsilon_ * dW2

        b2 += -epsilon_ * db2

        W3 += -epsilon_ * dW3

        b3 += -epsilon_ * db3

        # Assign new parameters to the model

        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2, 'W3': W3, 'b3': b3}

        # Optionally print the loss.

        # This is expensive because it uses the whole dataset, so we don't want to do it too often.

        if print_loss and i % 1000 == 0:

          print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model

效果：

Loss after iteration 0: 0.471144

Loss after iteration 1000: 0.065453

Loss after iteration 2000: 0.063970

Loss after iteration 3000: 0.063321

Loss after iteration 4000: 0.062937

Loss after iteration 5000: 0.062673

Loss after iteration 6000: 0.062476

Loss after iteration 7000: 0.062319

Loss after iteration 8000: 0.062190

Loss after iteration 9000: 0.062081

Loss after iteration 10000: 0.061986

Loss after iteration 11000: 0.061902

Loss after iteration 12000: 0.061827

Loss after iteration 13000: 0.061760

Loss after iteration 14000: 0.061698

Loss after iteration 15000: 0.061641

Loss after iteration 16000: 0.061589

Loss after iteration 17000: 0.061540

Loss after iteration 18000: 0.061494

Loss after iteration 19000: 0.061451