Vanishing Gradients and Fancy RNNs
Vanishing gradient problem
Why is vanishing gradient a problem?
Another explanation: Gradient can be viewed as a measure of
the effect of the past on the future. If the gradient becomes vanishingly small over longer distances (step t to step t+n), then we can’t tell whether:
- There’s no dependency between step t and t+n in the data
- We have wrong parameters to capture the true dependency between t and t+n
Effect of vanishing gradient on RNN-LM
Why is exploding gradient a problem?
If the gradient becomes too big, then the SGD update step
becomes too big:
\[
\theta^{\text {new}}=\theta^{\text { old }}-\alpha \nabla_{\theta} J(\theta)
\]
This can cause bad updates: we take too large a step and reach
a bad parameter configuration (with large loss). In the worst case, this will result in Inf or NaN in your network (then you have to restart training from an earlier checkpoint).
Gradient clipping: solution for exploding gradient
Gradient clipping: if the norm of the gradient is greater than
some threshold, scale it down before applying SGD update
Intuition: take a step in the same direction, but a smaller step
How to fix vanishing gradient problem?
The main problem is that it’s too difficult for the RNN to learn to preserve information over many timesteps. In a vanilla RNN, the hidden state is constantly being rewritten:
\[
\boldsymbol{h}^{(t)}=\sigma\left(\boldsymbol{W}_{h} \boldsymbol{h}^{(t-1)}+\boldsymbol{W}_{x} \boldsymbol{x}^{(t)}+\boldsymbol{b}\right)
\]
Long Short-Term Memory (LSTM)
How does LSTM solve vanishing gradients?
- The LSTM architecture makes it easier for the RNN to preserve information over many timesteps
- LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide an easier way for the model to learn long-distance dependencies
Gated Recurrent Units (GRU)
LSTM vs GRU
- The biggest difference is that GRU is quicker to compute and has fewer parameters
- There is no conclusive evidence that one consistently performs better than the other
- LSTM is a good default choice (especially if your data has particularly long dependencies, or you have lots of training data)
- Rule of thumb: start with LSTM, but switch to GRU if you want something more efficient
Conclusion: Though vanishing/exploding gradients are a general
problem, RNNs are particularly unstable due to the repeated
multiplication by the same weight matrix