LSTM
Understanding LSTM Networks 和 人人都能看懂的LSTM 这两篇文章介绍了 LSTM 的原理。
2D-LSTM
2D-LSTM 是作用于三维输入(
W
×
H
×
D
W \times H \times D
W×H×D )的 LSTM ，分别取横向和纵向上一时刻的隐藏状态和输出作为该时刻的输入，如下图所示
数据传播的顺序依靠对角线原则，如下图所示
图中的数字表示计算的顺序。
下图展示了 2D-LSTM 单元的结构，蓝线表示与标准单元不同的地方。
上图中
x
j
,
i
x_{j, i}
xj,i 为当前的输入，
s
j
,
i
−
1
s_{j, i-1}
sj,i−1 为上一时刻横向的输出，
s
j
−
1
,
i
s_{j-1, i}
sj−1,i 为上一时刻纵向的输出。
input gate
i
j
,
i
=
σ
(
W
1
x
j
,
i
+
U
1
s
j
−
1
,
i
+
V
1
s
j
,
i
−
1
)
i_{j, i} = \sigma(W_1x_{j, i} + U_1s_{j-1, i} + V_1s_{j, i-1})
ij,i=σ(W1xj,i+U1sj−1,i+V1sj,i−1)
output gate
o
j
,
i
=
σ
(
W
2
x
j
,
i
+
U
2
s
j
−
1
,
i
+
V
2
s
j
,
i
−
1
)
o_{j, i} = \sigma(W_2x_{j, i} + U_2s_{j-1, i} + V_2s_{j, i-1})
oj,i=σ(W2xj,i+U2sj−1,i+V2sj,i−1)
candidate value
c
^
j
,
i
=
g
(
W
3
x
j
,
i
+
U
3
s
j
−
1
,
i
+
V
3
s
j
,
i
−
1
)
\hat{c}_{j, i} = g(W_3x_{j, i} + U_3s_{j-1, i} + V_3s_{j, i-1})
c^j,i=g(W3xj,i+U3sj−1,i+V3sj,i−1)
forget gate
f
j
,
i
=
σ
(
W
4
x
j
,
i
+
U
4
s
j
−
1
,
i
+
V
4
s
j
,
i
−
1
)
f_{j, i} = \sigma(W_4x_{j, i} + U_4s_{j-1, i} + V_4s_{j, i-1})
fj,i=σ(W4xj,i+U4sj−1,i+V4sj,i−1)
2D-LSTM 新加入了一个系数，用于比较
s
j
−
1
,
i
s_{j-1, i}
sj−1,i 和
s
j
,
i
−
1
s_{j, i-1}
sj,i−1 的重要程度。
λ
j
,
i
=
σ
(
W
5
x
j
,
i
+
U
5
s
j
−
1
,
i
+
V
5
s
j
,
i
−
1
)
\lambda_{j, i} = \sigma(W_5x_{j, i} + U_5s_{j-1, i} + V_5s_{j, i-1})
λj,i=σ(W5xj,i+U5sj−1,i+V5sj,i−1)
新状态
c
j
,
i
=
f
j
,
i
∘
[
λ
j
,
i
∘
c
j
−
1
,
i
+
(
1
−
λ
j
,
i
)
∘
c
j
,
i
−
1
]
+
c
^
j
,
i
∘
i
j
,
i
c_{j, i} = f_{j, i} \circ [\lambda_{j, i} \circ c_{j-1, i} + (1 - \lambda_{j, i}) \circ c_{j, i-1}] + \hat{c}_{j, i} \circ i_{j, i}
cj,i=fj,i∘[λj,i∘cj−1,i+(1−λj,i)∘cj,i−1]+c^j,i∘ij,i
输出
s
j
,
i
=
g
(
c
j
,
i
∘
o
j
,
i
)
s_{j, i} = g(c_{j, i} \circ o_{j, i})
sj,i=g(cj,i∘oj,i)
Reference
[1] Bahar, P. , C. Brix , and H. Ney . “Towards Two-Dimensional Sequence to Sequence Model in Neural Machine Translation.” (2018).
[2] Voigtlaender, P. , P. Doetsch , and H. Ney . “Handwriting Recognition with Large Multidimensional Long Short-Term Memory Recurrent Neural Networks.” 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR) IEEE, 2017.