Recurrent Neural Networks and Gated Recurrent Units

Recurrent Neural Networks (RNNs)

Up until now, we have assumed that our inputs are vectors or scalars of a fixed size. But in the real world, data often comes in sequences of different lengths, such as sentences in a movie review, audio waves, or time-series data. To handle inputs of varying lengths without changing our network structure for each new sample, we need to introduce two key concepts: a memory state and a recursive function.

To count the number of 1s in a binary sequence, start with a counter set to 0. Then go through the sequence one element at a time, adding 1 to the counter each time you encounter a 1.

Sequence: [1, 0, 1]
Step 1: s = 0 + 1 -> 1
Step 2: s = 1 + 0 -> 1
Step 3: s = 1 + 1 -> 2

This simple loop relies on two rules:

1. A memory (s): It retains a summary of everything processed so far.
2. A shared function: The exact same rule is applied to every single element in the sequence, regardless of whether the sequence has 3 elements or 3,000.

An RNN works on the same principle. It uses a hidden memory vector $\mathbf{h}_t \in \mathbb{R}^{d_h}$ instead of a simple counter. At each time step $t$, the hidden state is updated by a function that considers both the current input token $\mathbf{x}_t$ and the previous memory state $\mathbf{h}_{t-1}$. $$\mathbf{h}_t = \tanh(W\mathbf{x}_t + U\mathbf{h}_{t-1})$$ Where $W$ and $U$ are shared weight matrices across all time steps, and $\tanh$ introduces non-linearity.

Mapping Inputs to Outputs

Depending on the task at hand, we can configure this recurrent engine to handle data in different shapes and sizes.

Architecture A: Fixed-Size Output (Many-to-One)

• The Goal: Predict a single label for an entire sequence.
• Common Use Case: Sentiment Analysis (e.g., classifying a product review as positive, negative, or neutral).

We go through the entire sequence until we reach the final token at index $l$. The final hidden state $\mathbf{h}_l$ is a compressed, mathematical summary of the entire sentence. We then pass this final state through a Softmax layer to calculate the category probabilities. $$\mathbf{\mu} = \text{softmax}(V\mathbf{h}_l)$$

Architecture B: Variable-Length Output (Many-to-Many)

• The Goal: Predict a label for every single token in the sequence.
• Common Use Case: Part-of-Speech (POS) Tagging (e.g., labeling every word in a sentence as a noun, verb, or adjective).

Instead of waiting until the end of the sentence, we branch off at every intermediate time step $t$. The current hidden state $\mathbf{h}_t$ is passed directly into a classifier to generate an immediate, real-time prediction: $$\mathbf{\mu}_t = \text{softmax}(V\mathbf{h}_t)$$

Backpropagation Through Time (BPTT)

Our weight parameters ($W$ and $U$) are used at every time step, which creates a problem when calculating their gradients. To solve this, we need to use a special version of the chain rule called Backpropagation Through Time (BPTT). When we calculate the derivative of the total cost with respect to our shared weight matrix $W$, we can't consider each time step separately. This is because $W$ affects $\mathbf{h}_1$, which then affects $\mathbf{h}_2$, and $\mathbf{h}_2$ affects $\mathbf{h}_3$, and so on. $$\frac{\partial C}{\partial W} = \sum_{t=1}^l \frac{\partial C}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial W}$$

The Structural Catch: When multiple derivative pathways flow backward into a shared parameter node, the computation graph must sum the incoming gradients before applying the local derivative. BPTT works in reverse chronological order, from the last time step to the first, accumulating the gradients for $W$ and $U$ at each node.

Modeling Output Structures

In our basic many-to-many setup, the network predicts the label $\mathbf{\mu}_t$ using only the inputs it has seen up to that point ($\mathbf{x}_{\le t}$). This relies on a massive assumption: the network treats each prediction as independent of the previous one. In natural language, this does not hold. The choice of word $t-1$ affects the choice of word $t$. For example, if word $t-1$ is an adjective, word $t$ is more likely to be a noun. To make our network learn this grammar, we can change our recurrence function to include the previous actual output $\mathbf{y}_{t-1}$ in the next hidden state. $$\mathbf{h}_t = f(\mathbf{x}_t, \mathbf{y}_{t-1}, \mathbf{h}_t-1)$$

Testing Dilemma: Greedy vs. Beam Search

During training, feeding $\mathbf{y}_{t-1}$ into the model is straightforward because we have the correct answers. This technique is called Teacher Forcing. At test time, we no longer have these correct answers to rely on. We have to choose an inference strategy:

• Greedy Search: We take the single highest-probability prediction at step 1, lock it in as $\hat{\mathbf{y}}_1$, and feed it into step 2. The risk? If the model makes even a slight mistake early on, that error cascades, throwing off every subsequent prediction down a "garden path" of nonsense.
• Beam Search: Instead of placing all our chips on the single best word, we maintain a running list of the top $N$ best alternative sequences (called the beam width). At each step, we expand those candidates and keep the top $N$ overall highest-probability paths. This safety net prevents an early mistake from instantly ruining the entire sequence prediction.

Gated Recurrent Units (GRUs) and LSTMs

A simple Recurrent Neural Network (RNN) has a major flaw: it overwrites all of its memory at each time step. This is highly inefficient. For comparison, a modern CPU doesn't clear its entire set of registers when it executes an instruction. Instead, it changes a few registers and leaves the rest alone. To make deep learning more efficient and stable, allowing networks to remember things over many steps, we need to use gated architectures.

To make an RNN realistic, we need two basic control mechanisms. First, we need an Update Gate ($u$) to decide which parts of our memory state should be modified and which parts should stay the same. Second, we need a Reset Gate ($r$) to determine how much of our past memory the current instruction needs to use.

If we were to copy computer hardware exactly, these gates would be binary vectors with absolute $0$s and $1$s. But discrete step functions are not differentiable - their derivatives are zero almost everywhere, which stops backpropagation from working. To make our computational graph trainable, we change this rule by turning the gates into continuous, real-valued coefficient vectors between $0$ and $1$ using the Sigmoid activation function ($\sigma$). This allows our gates to be smoothly variable, deciding what percentage of information to pass through, and letting gradients flow backward during training.

Gated Recurrent Unit (GRU)

At each discrete time step $t$, a GRU performs a sequence of tightly coupled vector calculations to update its single hidden state $\mathbf{h}_t$. The process begins with the Reset Gate $\mathbf{r}_t$, which decides how much of the past hidden state $\mathbf{h}_{t-1}$ is relevant to our current calculation:

$$\mathbf{r}_t = \sigma(W_r \mathbf{x}_t + U_r \mathbf{h}_{t-1})$$

Once we have this gate vector, we calculate a fresh Candidate Hidden State $\tilde{\mathbf{h}}_t$. Notice how the reset gate element-wise modulates ($\odot$) the past memory, allowing the network to actively forget past context if the reset gate is closed:

$$\tilde{\mathbf{h}}_t = \tanh(W \mathbf{x}_t + U(\mathbf{r}_t \odot \mathbf{h}_{t-1}))$$

Simultaneously, the Update Gate $\mathbf{u}_t$ calculates the exact balance between keeping the old memory and adopting the new candidate:

$$\mathbf{u}_t = \sigma(W_u \mathbf{x}_t + U_u \mathbf{h}_{t-1})$$

Finally, we compute the Final Hidden State $\mathbf{h}_t$ through a linear interpolation that blends the past and the present.

$$\mathbf{h}_t = (1 - \mathbf{u}_t) \odot \mathbf{h}_{t-1} + \mathbf{u}_t \odot \tilde{\mathbf{h}}_t$$

The Catch: This formulation stops the Vanishing Gradient Problem because when the update gate $\mathbf{u}_t$ is close to $0$, the current candidate is ignored, and $\mathbf{h}_t$ is approximately equal to $\mathbf{h}_{t-1}$. This design creates a direct pathway in the computational graph, allowing gradients to flow backward across many time steps without being multiplied by small weight matrices. As a result, the vanishing gradient issue is resolved.

Long Short-Term Memory (LSTM)

The GRU is a modern version of an earlier architecture proposed in 1997: the Long Short-Term Memory (LSTM) network. Unlike a GRU, which has a single hidden state, an LSTM has two distinct vectors. The Cell State ($\mathbf{c}_t$) is an internal, long-term component that changes slowly through linear updates and does not interact directly with the outside layers. The Hidden State ($\mathbf{h}_t$) is the short-term output that is visible to the rest of the network at the current step.

An LSTM uses three gates to control the interactions between these two states. The Forget Gate ($\mathbf{f}_t$) decides what information to discard from the long-term cell state. The Input Gate ($\mathbf{i}_t$) decides what new information to add to the cell state. The Output Gate ($\mathbf{o}_t$) determines what information from the cell state to pass on to the short-term hidden state. $$\mathbf{f}_t = \sigma(W_f \mathbf{x}_t + U_f \mathbf{h}_{t-1})$$ $$\mathbf{i}_t = \sigma(W_i \mathbf{x}_t + U_i \mathbf{h}_{t-1})$$ $$\mathbf{o}_t = \sigma(W_o \mathbf{x}_t + U_o \mathbf{h}_{t-1})$$

To compute the updates, we first update our long-term cell state by forgetting the past and injecting a new candidate state scaled by the input gate: $$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tanh(W_c \mathbf{x}_t + U_c \mathbf{h}_{t-1})$$

Once the long-term cell state is updated, we filter it through our output gate and squeeze the values using a hyperbolic tangent function to produce our final visible short-term hidden state: $$\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$$

GRU vs. LSTM

The Expert Consensus

There is no absolute mathematical winner between the two architectures. LSTMs are more expressive because they separate internal cell memory from external visibility, making them powerful for complex, long-sequence tasks like machine translation. GRUs, on the other hand, offer a faster alternative that often performs as well as LSTMs while using less hardware memory, making them a better choice when resources are limited or when working with smaller datasets.

Why Not Rectifiers in RNNs?

Rectifiers are activation functions used in neural networks, most notably the Rectified Linear Unit (ReLU). They work by passing the input directly to the output if it's positive, and outputting zero if it's not. This simple design helps speed up training and prevents the vanishing gradient problem.

In modern feedforward neural networks, such as MLPs and CNNs, piecewise linear activation functions like the Rectified Linear Unit (ReLU) are widely used. These functions help solve the vanishing gradient problem in deep networks, making training very fast. But if we use unbounded functions like ReLU in a simple Recurrent Neural Network (RNN), the model will likely fail. In RNNs, bounded logistic functions like $\tanh$ are used instead.

This difference in approach is due to a fundamental difference in architecture. Deep feedforward networks use different weights at each layer, whereas an RNN uses the same weight matrix at every time step. This creates a feedback loop that can cause problems when using unbounded activations, as gradients and hidden states can increase exponentially.

The Exploding Memory Problem

The main difference between a traditional logistic function (like $\tanh$ or sigmoid) and a rectifier (like ReLU) is that logistic functions are strictly bounded between fixed limits, while rectifiers are unbounded as the input grows.

This boundary is important for recurrent networks. Consider a simple RNN that uses a standard linear activation, where $g(a) = a$, instead of a bounding function. If we unroll this recurrence over $l$ time steps, the final hidden state $\mathbf{h}_l$ becomes a large accumulation of our shared hidden-to-hidden transition matrix $U$: $$\mathbf{h}_l = \sum_{t=1}^l \left( \prod_{l'=1}^{l-t} U \right) W\phi(\mathbf{x}_t)$$

If we break down the shared matrix $U$ using eigendecomposition ($U = QSQ^{-1}$), we can see the mathematical trap. Multiplying $U$ repeatedly over $l$ steps means we are raising its internal eigenvalues ($S$) to the power of the sequence length: $$U^{l-t} = QS^{l-t}Q^{-1}$$

What happens if the largest eigenvalue ($\lambda_{\max}$) of our hidden weight matrix $U$ is even slightly greater than 1? Because the sequence length $l$ acts as an exponent, the norm of our hidden memory state will explode exponentially: $$\Vert{}\mathbf{h}_l\Vert{} \to \infty$$

If the network runs for dozens or hundreds of time steps, these hidden states will quickly trigger a numerical overflow (NaN), completely breaking the model.

By swapping in a bounded function like $\tanh$, we enforce a strict physical boundary. No matter how massive the inputs or the weights become, every single element in our hidden memory state vector is strictly locked between $[-1, 1]$. This simple boundary acts as an emergency brake, keeping the network's long-term memory stable and preventing it from spiraling out of control.

Is $\tanh$ a Perfect Blessing? (The Gradient Dilemma)

The $\tanh$ function keeps our memory states stable during the forward pass, but it causes a large penalty during the backward pass. This is one of the biggest challenges in sequence modeling. When we use Backpropagation Through Time (BPTT) to compute our gradients, we consider how a change in an early input at step $t'$ affects a hidden state at a later step $t_1$. The derivative of this effect over time can be broken down using the chain rule into a product of matrix multiplications: $$\frac{\partial \mathbf{h}_{t_1}}{\partial \mathbf{h}_{t_0+1}} = \prod_{t=t_0+2}^{t_1} \left( \text{diag}(\mathbf{g}'(\mathbf{h}_t)) U \right)$$

Here, $\mathbf{g}'(\mathbf{h}_t)$ represents the local derivative of our activation function. For $\tanh$, the maximum possible value of its derivative is $1$ (and it quickly drops toward $0$ as the input moves away from zero). This means the maximum eigenvalue of our effective backward transition matrix is scaled at every single time step by the activation's derivative: $\text{e}_{\text{max, effective}} \le \lambda \cdot \text{e}_{\text{max}}$, where $\lambda \le 1$.

This leaves us caught between two structural nightmares:

• Exploding Gradients: If our weights $U$ are large enough to counteract the dampening effect of the derivative ($\text{e}_{\text{max}} > \frac{1}{\lambda}$), the backpropagated error signal will grow exponentially, causing the optimization algorithm to wildly overshoot.
• Vanishing Gradients: If our weights are even slightly smaller ($\text{e}_{\text{max}} < \frac{1}{\lambda}$), the error signal will vanish exponentially toward $0$. If a sequence is long, the gradient from a distant time step becomes effectively zero, meaning our network is physically incapable of learning long-term dependencies.

How We Fix the Gradients

Exploding gradients are easy to diagnose and fix. The explosion shows up as a very large gradient norm ($\|\nabla\|$), which can be controlled with a safety mechanism called Gradient Clipping. If the total norm of our gradients is greater than a set threshold $\tau$, we reduce the gradient vector to a smaller size while keeping its direction: $$\tilde{\nabla} = \begin{cases} \tau \frac{\nabla}{\|\nabla\|} & \text{if } \|\nabla\| > \tau \\ \nabla & \text{otherwise} \end{cases}$$ Vanishing gradients cannot be fixed with a simple numerical wrapper like clipping. If a gradient vanishes, the error signal vanishes, and we cannot tell if the model is done learning or simply broken.

This is exactly why we use Gated Recurrent Units (GRUs) and LSTMs. If we expand the hidden state equation of a GRU out over time, we see that the update gate $\mathbf{u}_t$ acts as a linear conveyor belt: $$\mathbf{h}_t = \left( \prod_{k=t'+1}^{t} (1 - \mathbf{u}_k) \right) \odot \mathbf{h}_{t'} + \dots$$

Instead of forcing the error signal to go through the matrix multiplications and non-linearities of a simple RNN layer repeatedly, the GRU creates a direct, linear connection across time.

If the network learns that a dependency from step $t'$ is important for a prediction at step $t$, it closes the update gate ($\mathbf{u} \approx 0$). This makes the coefficient $(1 - \mathbf{u})$ close to $1$, allowing information in the forward pass and the gradient in the backward pass to bypass intermediate steps without disappearing.