摘要

LSTM是RNN的一个优秀的变种模型，继承了大部分RNN模型的特性；在本次学习中，展示了LSTM的手动推导过程，用代码逐行模拟实现LSTM的运算过程，并与Pytorch API输出的结验证是否保持一致。

Abstract

LSTM is an excellent variant model of RNN, inheriting most of the characteristics of RNN model. In this study, the manual derivation process of LSTM is demonstrated, and the operation process of LSTM is simulated line by line with the code, and whether it is consistent with the junction output of Pytorch API is verified.

1 LSTM

1.1 使用Pytorch LSTM

实例化LSTM类，需要传递的参数：

input_size：输入数据的特征维度
hidden_size：隐含状态$h_t$的大小
num_layers：默认值为1，大于1，表示多个RNN堆叠起来
batch_first：默认是False；若为True,输入输出格式为：(batch, seq, feature) ；若为False：输入输出格式为： (seq, batch, feature)；
bidirectional：默认为False；若为True，则是双向RNN，同时输出长度为2*hidden_size
proj_size：若不为None，将使用具有相应大小投影的 LSTM

函数输入值：（input，（h_0,c_0））

1. input：当batch_first=True 输入格式为（N,L,$H_{in}$）;当batch_first=False 输入格式为（L,N,$H_{in}$）;
2. h_0：默认输入值为0；输入格式为：（D*num_layers,N,$H_{out}$）
3. c_0：默认输入值为0，输入格式为：（D*num_layers,N,$H_{cell}$）

其中：

N = batch size
L = sequence length
D = 2 if bidirectional=True otherwise 1
$H_{in}$ = input_size
$H_{out}$ = hidden_size
$H_{cell}$ = proj_size if proj_size>0 否则 hidden_size

函数输出值：（output，（h_n, c_n））

1. output：当batch_first=True 输出格式为（N，L，D*$H_{out}$）;当batch_first=False 输出格式为（L，N，D*$H_{out}$）;
2. h_n：输出格式为：（D*num_layers，N，$H_{out}$）
3. c_n：输出格式为：（D*num_layers，N，$H_{cell}$）

1.1.1 LSTM API代码实现

首先实例化一些参数：

# 定义常量
bs, T, input_size, hidden_size = 2, 3, 4, 5

#输入序列
input = torch.randn(bs, T, input_size)
c0 = torch.randn(bs, hidden_size)
h0 = torch.randn(bs, hidden_size)

调用Pytorch中的LSTM API:
并查看返回的结果及其形状

# 调用官方的LSTM API
lstm_layer = nn.LSTM(input_size, hidden_size, batch_first=True)
output, (hn, cn) = lstm_layer(input, (h0.unsqueeze(0), c0.unsqueeze(0)))

print("----官方LSTM的输出----")
print(output)
print(output.shape) # torch.Size([2, 3, 5])---[bs, T, hidden_size]
print(hn)
print(hn.shape) # torch.Size([1, 2, 5])---[num_layers,bs,hidden_size]
print(cn)
print(cn.shape) # torch.Size([1, 2, 5])---[num_layers,bs,hidden_size]

输出为：

----官方LSTM的输出----
tensor([[[ 0.3045, -0.0770,  0.0110, -0.1592, -0.2298],
         [ 0.1142, -0.1522, -0.0975, -0.1512,  0.2834],
         [ 0.1298,  0.0090,  0.0183, -0.2742,  0.0428]],

        [[ 0.1908, -0.1505, -0.0659,  0.0520,  0.0340],
         [ 0.2583, -0.0599, -0.0442, -0.1428, -0.0102],
         [ 0.1604,  0.0004, -0.1105, -0.2062,  0.0760]]],
       grad_fn=<TransposeBackward0>)
torch.Size([2, 3, 5])
tensor([[[ 0.1298,  0.0090,  0.0183, -0.2742,  0.0428],
         [ 0.1604,  0.0004, -0.1105, -0.2062,  0.0760]]],
       grad_fn=<StackBackward0>)
torch.Size([1, 2, 5])
tensor([[[ 0.3637,  0.0263,  0.0326, -0.5326,  0.0537],
         [ 0.4828,  0.0008, -0.2197, -0.3999,  0.1016]]],
       grad_fn=<StackBackward0>)
torch.Size([1, 2, 5])

获取 pytorch LSTM内置参数的形状

for k, v in lstm_layer.named_parameters():
    print(k, v)

其中：

weight_ih_l0维度为：[4*hidden_size,input_size]
weight_hh_l0维度为：[4*hidden_size,hidden_size]
bias_ih_l0维度为：[4*hidden_size]
bias_hh_l0维度为：[4*hidden_size]

1.1.2 LSTMP代码实现

如果指定proj_size >0 ，将会使用带投影的LSTM，以下列方式更改LSTM单元。
首先，$h_t$的维度将从hidden_size更改为：proj_size
其次，每一层的输出隐藏状态将乘以一个可学习的投影矩阵，$h_t=w_{hr}{h}_t$

因此，LSTMP 网络的输出也将具有不同的形状。

首先实例化一些参数：

bs, T, input_size, hidden_size = 2, 3, 4, 5
proj_size = 3

input = torch.randn(bs, T, input_size)
h0 = torch.randn(bs, proj_size)
c0 = torch.randn(bs, hidden_size)

调用PyTorch中的 LSTMP API：
并查看返回的结果及其形状

# 使用官方的LSTMP
layer_LSTMP = nn.LSTM(input_size, hidden_size, batch_first=True, proj_size=proj_size)
output, (h_n, c_n) = layer_LSTMP(input, (h0.unsqueeze(0), c0.unsqueeze(0)))
print(output)
print(output.shape) # torch.Size([2, 3, 3])---[bs,T,proj_size]
print(h_n)
print(h_n.shape)    # torch.Size([1, 2, 3])---[num_layers,bs,proj_size]
print(c_n)
print(c_n.shape)    # torch.Size([1, 2, 5])---[num_layers,bs,hidden_size]

输出为：

tensor([[[-0.1677, -0.0150, -0.0289],
         [-0.1369, -0.0738, -0.0311],
         [ 0.0168, -0.0184,  0.0303]],

        [[-0.0935, -0.0204, -0.1073],
         [-0.0137, -0.0056, -0.0085],
         [-0.0654, -0.0376,  0.0562]]], grad_fn=<TransposeBackward0>)
torch.Size([2, 3, 3])
tensor([[[ 0.0168, -0.0184,  0.0303],
         [-0.0654, -0.0376,  0.0562]]], grad_fn=<StackBackward0>)
torch.Size([1, 2, 3])
tensor([[[ 0.1691, -0.0772,  0.0209,  0.1153,  0.2290],
         [ 0.4153, -0.0653, -0.0928,  0.3042,  0.2146]]],
       grad_fn=<StackBackward0>)
torch.Size([1, 2, 5])

获取 pytorch LSTMP内置参数

for k, v in layer_LSTMP.named_parameters():
    print(k, v.shape)

其中：

weight_ih_l0维度为：[4*hidden_size,input_size]
weight_hh_l0维度为：[4*hidden_size,Proj_size]
bias_ih_l0维度为：[4*hidden_size]
bias_hh_l0维度为：[4*hidden_size]
weight_hr_l0维度为：[Proj_size,hidden_size]

1.2 手写一个lstm_forward函数 实现单向LSTM的计算原理

这里先将lstm_forward函数中的每个参数的维度写出来：

def lstm_forward(input, w_ih, w_hh, b_ih, b_hh, initial_states):
    h0, c0 = initial_states # 初始状态
    bs, T, input_size = input.shape
    hidden_size = w_ih.shape[0]//4

    prev_h = h0
    prev_c = c0
    # input 的维度为：[batch_size, T, input_size]
    # 其中 w_ih和w_hh 的维度为：[4*hidden_size, input_size]，[4*hidden_size, hidden_size]
    batch_w_ih = w_ih.unsqueeze(0).tile(bs, 1, 1) # 这样w_ih的变为：[batch_size, 4*hidden_size, input_size]
    batch_w_hh = w_hh.unsqueeze(0).tile(bs, 1, 1) # 这样w_hh的变为：[batch_size, 4*hidden_size, hidden_size]

    output_size = hidden_size
    print(output_size)    # 5
    # 输出序列
    output = torch.zeros(bs, T, output_size)

    for t in range(T):
        # 获取当前时刻的x
        x = input[:, t, :]  #这时候x的维度为：[bs, input_size]
        # 进行矩阵运算，先将x的维度扩为：[bs, input_size, 1]
        # 矩阵运算之后的维度为：[bs, 4*hidden_size, 1]
        w_times_x = torch.bmm(batch_w_ih, x.unsqueeze(-1))
        w_times_x = w_times_x.squeeze(-1)    # 此时维度为：[bs, 4*hidden_size]

        w_times_h_prev = torch.bmm(batch_w_hh, prev_h.unsqueeze(-1))    # [bs, 4*hidden_size，1]
        w_times_h_prev = w_times_h_prev.squeeze(-1) # 维度为：[bs, 4*hidden_size]

        # 计算输入门（i），遗忘门（f）,cell门（g）,输出门（o）
        i_t = torch.sigmoid(w_times_x[:, :hidden_size] + w_times_h_prev[:, :hidden_size] + b_ih[:hidden_size] + b_hh[:hidden_size])
        f_t = torch.sigmoid(w_times_x[:, hidden_size:2*hidden_size] + w_times_h_prev[:, hidden_size:2*hidden_size] +
                            b_ih[hidden_size:2*hidden_size] + b_hh[hidden_size:2*hidden_size])
        g_t = torch.tanh(w_times_x[:, 2*hidden_size:3*hidden_size] + w_times_h_prev[:, 2*hidden_size:3*hidden_size] +
                            b_ih[2*hidden_size:3*hidden_size] + b_hh[2*hidden_size:3*hidden_size])
        o_t = torch.sigmoid(w_times_x[:, 3*hidden_size:4*hidden_size] + w_times_h_prev[:, 3*hidden_size:4*hidden_size] +
                            b_ih[3*hidden_size:4*hidden_size] + b_hh[3*hidden_size:4*hidden_size])

        prev_c = f_t * prev_c + i_t * g_t
        prev_h = o_t * torch.tanh(prev_c)   # [bs, hidden_size]

        output[:, t, :] = prev_h

    return output, (prev_h, prev_c)

• 使用pytorch中LSTM的内置参数测试该函数

c_output, (c_hn, c_cn) = lstm_forward(input, lstm_layer.weight_ih_l0, lstm_layer.weight_hh_l0,
                                      lstm_layer.bias_ih_l0, lstm_layer.bias_hh_l0, (h0, c0))
print("----手写LSTM的输出----")
print(c_output)
print(c_hn)
print(c_cn)

查看两者的输出结果完全相同

1.3 手写一个lstmp_forward函数 实现单向LSTMP的计算原理

def lstmp_forward(input, initial_states, w_ih, w_hh, b_ih, b_hh, w_hr=None):
    h0, c0 = initial_states
    bs, T, input_size = input.shape
    hidden_size = w_ih.shape[0]//4

    prev_h = h0 # [bs, proj_size]
    prev_c = c0 # [bs, hidden_size]

    # w_ih维度：[4*h_size, input_size] w_hh维度：[4*h_size, proj_size]    w_hr维度：[proj_size, h_size]
    batch_w_ih = w_ih