Neural Machine Translation

Machine Translation (MT) is essentially about translating a text sequence from a source language to an equivalent sequence in a target language. Since a phrase can be translated correctly in several ways, our system needs to produce a probability distribution that covers all possible translations, rather than a single fixed sentence.

Let the input source sentence be a sequence of discrete words: $$X = (x_1, x_2, \dots, x_{T_x})$$

And let the generated target sentence be: $$Y = (y_1, y_2, \dots, y_{T_y})$$

Conditional Language Modelling

The translation function $f$ reads the source sequence $X$ and maps it to a standard $k$-simplex, which represents a valid categorical probability distribution over every possible word in the target vocabulary $V_y$. The lengths of the sequences $X$ and $Y$ are denoted by $T_x$ and $T_y$, respectively.

We can factorize the conditional probability of the target translation $P(Y \mid X)$ using the chain rule of probability. $$P(Y \mid X) = \prod_{t=1}^{T_y} P(y_t \mid y_1, y_2, \dots, y_{t-1}, X)$$

This formulation shows that machine translation is essentially conditional language modeling. Instead of predicting the next word based only on past target words, the generation is explicitly conditioned on the entire source sentence $X$.

To train this network, we feed it a parallel dataset $D = \{(X_1, Y_1), \dots, (X_N, Y_N)\}$ containing $N$ accurate translation pairs. We then optimize our model parameters $\theta$ by minimizing the global Negative Log-Likelihood (NLL): $$\mathcal{L}(\theta) = -\frac{1}{N} \sum_{n=1}^N \sum_{t=1}^{T_y} \log p(y_t^n \mid y_{

Parallel Corpus

To teach a machine learning model to translate effectively, we need a parallel corpus, a dataset that matches sentences phrase by phrase across languages. This type of corpus is different from typical text collections. It helps neural networks map the structure and meaning of sentences from one language to another. Getting high-quality parallel data on a large scale is a major challenge in natural language processing.

This scarcity of data is mainly caused by two obstacles: copyright laws and a lack of resources for certain languages. Most high-quality human translations, such as books, news feeds, and product listings, are protected by strict international copyright laws. Using these materials to train models can lead to legal problems, as the models may be seen as unauthorized copies. The distribution of data is also uneven. While some Western languages have large public datasets, many major languages spoken by hundreds of millions of people, like Hindi, Arabic, and Korean, have very few parallel resources available. This creates a big difference in translation accuracy between languages.

To avoid copyright issues and obtain authentic, high-quality data, researchers have traditionally used public parliamentary proceedings from regions with many languages. These governing bodies are required by law to publish their records in all official languages, which has resulted in the creation of high-quality, professionally translated datasets that are not copyrighted. Two notable examples are the Canadian Hansards, which digitally records every speech given in the Canadian parliament in both French and English, and the Europarl Corpus, which tracks European Parliament proceedings in up to 23 languages at the same time.

These political records provide a good foundation, but their formal, bureaucratic language is a limitation. To improve translation systems and help them understand how people normally communicate, modern NLP systems do not rely on manual government collections. They use automated web-crawlers, like the Common Crawl Parallel Corpus, to search the internet and find matching multilingual websites across many different domains..

How a Parallel Corpus Works

Automatic Evaluation

Evaluating machine translation performance is much harder than standard classification. If a classifier makes an incorrect prediction, its accuracy drops. With translation, the problem is that there's no one "right" answer.

For example, consider three different human translations of the same Chinese sentence:

• It is a guide to action that ensures that the military will forever heed Party commands.
• It is the guiding principle which guarantees the military forces always being under the command of the Party.
• It is the practical guide for the army always to heed the directions of the party.

These sentences look very different when you compare them word for word, but they are all valid translations. Since grading thousands of sentences by hand is time-consuming and costly, we have to use a computerized measure: BLEU.

BLEU Score

The Bilingual Evaluation Understudy (BLEU) score evaluates translation quality by measuring how many $n$-grams in the machine's output ($Y$) appear inside a human-written reference translation ($Y_{\text{ref}}$).

A. Modified $n$-Gram Precision

Standard precision metrics can be tricked by repetitive loops. For example, if a model generates "the the the", a simple check against a reference containing "the" would mark it as $100\%$ accurate. BLEU addresses this issue by limiting the count of an $n$-gram to its maximum frequency in the reference text ($\hat{c}$): $$p_n = \frac{\sum_{S \in C} \sum_{\text{ngram} \in S} \min\left(c(\text{ngram}), c_{\text{ref}}(\text{ngram})\right)}{\sum_{S \in C} \sum_{\text{ngram} \in S} c(\text{ngram})}$$

To capture both individual word choice and structural fluency, BLEU calculates the geometric mean of modified precisions across 1-, 2-, 3-, and 4-grams: $$P = \exp\left(\frac{1}{4} \sum_{n=1}^4 \log p_n\right)$$

B. Short-Text Loophole and Brevity Penalty

Relying strictly on precision introduces a massive loophole: a model can maximize its score by printing just a few highly accurate words.

If the human reference is: "I like a llama, a cute animal that lives in South America." and a machine outputs only four words: "cute animal that lives", its 1-unit through 4-unit precisions evaluate to an absolute maximum score: $$p_1 = \frac{4}{4} = 1, \quad p_2 = \frac{3}{3} = 1, \quad p_3 = \frac{2}{2} = 1, \quad p_4 = \frac{1}{1} = 1 \implies P = 1.0$$

To penalize truncated translations, BLEU scales the geometric precision by a Brevity Penalty (BP) based on the generation length ($l$) and reference length ($r$): $$\text{BP} = \begin{cases} 1 & \text{if } l \ge r \\ \exp\left(1 - \frac{r}{l}\right) & \text{if } l < r \end{cases}$$

Applying this to our short four-word example drops the final score to an accurate $0.064$. $$\text{BLEU} = \text{BP} \cdot P$$

Limitations of BLEU

Since it only considers exact-string token matches, it can be overly harsh on models that use correct synonyms or different grammatical forms, such as those used in rule-based or deep neural systems. To overcome this limitation, modern evaluations often use BLEU in combination with other metrics like METEOR or TER.

The Encoder-Decoder Framework

The translation pipeline can be thought of as a simple encoder-decoder model. This model takes a sentence of any length and turns it into a dense vector. It then uses a separate model to generate a sequence in the target language from this vector.

Source Sequence (X) ---> [ ENCODER (RNN) ] ---> Context Vector (c) ---> [ DECODER (RNN) ] ---> Target Sequence (Y)

Encoder

The encoder reads a source sequence $X = (x_1, x_2, \dots, x_{T_x})$ token-by-token. Its sole objective is to condense the semantic properties of the source text into a single, dense state. The mathematical step updates as: $$\mathbf{h}_t = \phi_{\text{enc}}\left(\mathbf{h}_{t-1}, \mathbf{E}_x^\top \mathbf{x}_t\right)$$

• $\mathbf{E}_x \in \mathbb{R}^{|V_x| \times d}$ represents the source word embedding matrix.
• $\mathbf{x}_t$ is the one-hot representation of the current source word.
• $\phi_{\text{enc}}$ is a non-linear gated transition unit, typically an LSTM or a GRU.

The initial hidden state $\mathbf{h}_0$ is set to an all-zero vector. Once the network consumes the final token at time step $T_x$, the final hidden state $\mathbf{h}_{T_x}$ serves as our fixed-size context vector ($\mathbf{c}$):

$$\mathbf{c} = \mathbf{h}_{T_x}$$

Decoder

The decoder is conceptually a recurrent language model, with one major modification: its hidden transitions and output projections are directly conditioned on the encoder's context vector $\mathbf{c}$. First, we initialize the decoder's starting trajectory ($\mathbf{z}_0$) using the context vector: $$\mathbf{z}_0 = \phi_{\text{init}}(\mathbf{c})$$

At each generation step $t$, the decoder updates its hidden state $\mathbf{z}_t$ by consuming a concatenated input vector consisting of the embedding of the previously emitted target word $\mathbf{y}_{t-1}$ and the static context vector $\mathbf{c}$: $$\mathbf{z}_t = \phi_{\text{dec}}\left(\mathbf{z}_{t-1}, \begin{bmatrix} \mathbf{E}_y^\top \mathbf{y}_{t-1} \\ \mathbf{c} \end{bmatrix}\right)$$

Once updated, the hidden state is projected into the target vocabulary space to output a categorical probability distribution: $$p(y_t = w' \mid \mathbf{y}_{

Because the entire pipeline is connected via a single continuous vector $\mathbf{c}$, the model can be trained end-to-end. We can backpropagate error gradients from the final target word prediction all the way through the decoder, across the context junction, and back to the initial encoder weights using standard stochastic gradient descent.

Token Inference

Once a model is trained, generating an optimal translation sequence from the conditional distribution $P(Y \mid X)$ introduces a brand-new problem. We must choose an inference strategy to map out token selections.

A. Stochastic Sampling

One approach is to sample tokens step-by-step from the calculated distribution $p(y_t \mid \mathbf{y}_{

B. The Search Problem

The goal is to find the translation sequence $\hat{Y}$ that maximizes the total conditional log-probability:

$$\hat{Y} = \arg\max_{Y} \sum_{t=1}^{T_y} \log p(y_t \mid \mathbf{y}_{

To find the exact maximum, you would need to search every possible path in a tree where each node branches into $|V_y|$ potential words. The size of this tree grows exponentially ($|V_y|^{T_y}$), making the search space too large to handle.

Approximate Search Strategies

To make inference computationally feasible, we must use approximate search algorithms.

A. Greedy Search

Greedy search is a local optimizer. At each step $t$, it selects the single word with the highest isolated probability:

$$\hat{y}_t = \arg\max_{w' \in V_y} \log p(y_t = w' \mid \hat{\mathbf{y}}_{

Greedy search is very cheap, requiring only $\mathcal{O}(1)$ vocabulary evaluations per step. However, it has a major drawback: a single mistake or sub-optimal choice can permanently affect the model's generation trajectory, with no way to correct it.

B. Beam Search

Beam search is a method that balances exact global search and localized greedy search. It maintains a fixed number ($K$) of high-probability sentence hypotheses at the same time.

Beam Search Protocol:

1. At Step $t=1$: Evaluate all $|V|$ options and keep the top $K$ highest-probability words as initial hypotheses: $(\mathbf{w}_1^1), (\mathbf{w}_1^2), \dots, (\mathbf{w}_1^K)$.
2. At Step $t+1$: For each of the $K$ active tracks, compute the conditional distribution over all vocabulary items. This gives $K \times |V|$ new path candidates.
3. Pruning: Calculate the cumulative joint log-probability for all $K \times |V|$ paths. Sort them and keep only the top $K$ overall winners, discarding the rest.
4. Termination: Repeat steps 2 and 3 until a path ends with an $\langle \text{eos} \rangle$ token or reaches a maximum length limit.

Beam Search Non-Monotonicity Paradox

As the beam width parameter $K$ approaches infinity, it's natural to think that translation quality will keep improving because the model searches a larger part of the probability space. However, this isn't the case. The complex way vocabulary probabilities are allocated can lead to a larger beam width producing a mathematically inferior total path compared to a narrower beam.

Mathematical Proof of the Paradox

Let's trace a toy example with a three-token vocabulary $V = \{a, b, c\}$ and a target sentence length of 3.

Step 1 ($t=1$): The model predicts the following initial probabilities: $$p(a) = 0.50, \quad p(b) = 0.15, \quad p(c) = 0.45$$

• If $K=1$ (Greedy): We retain only path (a).
• If $K=2$ (Beam): We retain paths (a) and (c).

Step 2 ($t=2$): The model evaluates the next token distribution based on the active hypotheses:

• From path (a): $p(a \mid a) = 0.40, \; p(b \mid a) = 0.30, \; p(c \mid a) = 0.30$
• From path (c): $p(a \mid c) = 0.45, \; p(b \mid c) = 0.45, \; p(c \mid c) = 0.10$

Now let's compute the cumulative joint probabilities ($p(y_1) \times p(y_2 \mid y_1)$):

• For $K=1$: The greedy track branches to (a, a) with a joint score of $0.50 \times 0.40 = \mathbf{0.20}$.
• For $K=2$: We evaluate all 6 combinations across both branches: $$p(a, a) = 0.50 \times 0.40 = 0.200$$ $$p(c, a) = 0.45 \times 0.45 = \mathbf{0.2025}$$ $$p(c, b) = 0.45 \times 0.45 = \mathbf{0.2025}$$ (All other combinations yield lower values)

The $K=2$ beam search selects the two best overall paths: (c, a) and (c, b). Crucially, the greedy path (a, a) is completely pruned away and dropped from the beam.

Step 3 ($t=3$): We evaluate the final token probabilities to close out our sentences:

• Greedy Path (a, a) Continuation: $p(a \mid a, a) = 0.90 \implies \text{Final } p(a, a, a) = 0.20 \times 0.90 = \mathbf{0.1800}$
• Beam Path (c, a) Continuation: $p(a \mid c, a) = 0.70 \implies \text{Final } p(c, a, a) = 0.2025 \times 0.70 = \mathbf{0.14175}$
• Beam Path (c, b) Continuation: $p(c \mid c, b) = 0.60 \implies \text{Final } p(c, b, c) = 0.2025 \times 0.60 = \mathbf{0.1215}$
  The Verdict:
• The Greedy Search ($K=1$) successfully returned the sequence (a, a, a) with a final probability of $0.1800$.
• The Beam Search ($K=2$) returned the sequence (c, a, a) with a final probability of $0.14175$.

By widening our beam to $K=2$, our search algorithm fell into a local trap at step 2, causing it to discard an optimal pathway and ultimately return a lower-probability translation.

Solution

To reduce this non-monotonic behavior, developers often run multiple parallel beam searches using different narrow widths (e.g., $K$ can be 1, 2, 4, 8, or 16). The system gathers the top candidates from each independent run and chooses the sequence with the highest score. This approach limits the worst-case memory consumption to $\mathcal{O}(K_{\max}|V|)$, allowing the decoder to find a reliable and high-quality translation path.

Attention-Based Neural Machine Translation

The classic encoder-decoder framework had a major architectural limitation: it condensed a source sentence of any length into a single, fixed-size context vector $\mathbf{c}$. This forced the encoder to pack the entire semantic meaning of a long, complex sentence into a static vector, which required learning a highly non-linear mapping. As sentences grew longer, this approach quickly broke down. We overcome this limitation with an Attention-Based Encoder-Decoder Model, which allows the network to create a dynamic context representation that can change size according to the input sequence.

To achieve this flexible representation, the encoder uses a Bidirectional Recurrent Neural Network (BiRNN) to process the source text in two parallel paths. The Forward RNN reads the sequence $X = (x_1, \dots, x_{T_x})$ from left to right, updating its internal state based on past words to generate forward hidden states:

$$\overrightarrow{\mathbf{h}}_t = \phi_{\text{fenc}}\left(\overrightarrow{\mathbf{h}}_{t-1}, \mathbf{E}_x^\top \mathbf{x}_t\right)$$

The Reverse RNN also processes the same sequence, but in the opposite direction - from the last token $x_{T_x}$ to the first token $x_1$. This generates a sequence of reverse hidden states that capture the context flowing from the end of the sentence:

$$\overleftarrow{\mathbf{h}}_t = \phi_{\text{renc}}\left(\overleftarrow{\mathbf{h}}_{t-1}, \mathbf{E}_x^\top \mathbf{x}_t\right)$$

After both sweeps are complete, the network combines the summaries from each direction at every token position $t$. It does this by joining the forward state $\overrightarrow{\mathbf{h}}_t$, which represents the past context $x_{\le t}$, with the reverse state $\overleftarrow{\mathbf{h}}_t$, which represents the future context $x_{\ge t}$. This creates a unified vector that depends on the context at each position.

$$\mathbf{h}_t = \begin{bmatrix} \overrightarrow{\mathbf{h}}_t \\ \overleftarrow{\mathbf{h}}_t \end{bmatrix}$$

Gathering these vectors across the entire sequence gives the complete structural representation of the source sentence: a dynamic, variable-length context set $C = \{\mathbf{h}_1, \mathbf{h}_2, \dots, \mathbf{h}_{T_x}\}$. Each vector in this set holds information from both ends of the sentence, so no token is left isolated. Instead, each one is mapped in the context of the entire sequence surrounding it.

When generating text, the decoder has a difficult task: it must select a single, highly relevant vector from this variable-length context set $C$ at each time step, without increasing its parameters. The attention mechanism solves this problem by calculating a unique weight configuration over the source tokens at every step. This happens in a four-step loop that lets the network change its focus as it translates, depending on what it is working on at the time.

The loop begins with relevance alignment scoring. At generation step $t$, the decoder determines how relevant the $j$-th source context vector $\mathbf{h}_j$ is for generating the next target word. The decoder calculates this raw relevance score ($e_{j,t}$) using an alignment function ($f_{\text{score}}$) that considers the decoder's previous hidden state $\mathbf{z}_{t-1}$, the previously emitted token vector $\mathbf{E}_y^\top \mathbf{y}_{t-1}$, and the target source vector $\mathbf{h}_j$:

$$e_{j,t} = f_{\text{score}}\left(\mathbf{z}_{t-1}, \mathbf{E}_y^\top \mathbf{y}_{t-1}, \mathbf{h}_j\right)$$

The score depends on $\mathbf{z}_{t-1}$ because this hidden state summarizes everything that has been translated up to step $t-2$. By combining this history with the immediate past word $\mathbf{y}_{t-1}$, the scoring mechanism understands what has just been said and what still needs to be translated next.

After calculating these raw scores for every source position, the network applies softmax normalization to turn them into a valid probability distribution. These normalized values are the attention weights ($\alpha_{j,t}$), which represent the percentage of focus the model should give to each input word.

$$\alpha_{j,t} = \frac{\exp(e_{j,t})}{\sum_{j'=1}^{T_x} \exp(e_{j',t})}$$

With these weights in hand, the model computes a dynamic context vector. It calculates a time-dependent context vector $\mathbf{c}_t$ by taking a weighted average across the entire context set $C$. This step compresses the entire source sentence into a single vector that is tailored for the current time step, giving more importance to the words with the highest attention weights:

$$\mathbf{c}_t = \sum_{j=1}^{T_x} \alpha_{j,t} \mathbf{h}_j$$

The loop ends with the decoder state update. The model updates its current hidden state $\mathbf{z}_t$ by using the recurrence function ($\phi_{\text{dec}}$) with its previous state, the embedding of the last emitted word, and the new context vector $\mathbf{c}_t$:

$$\mathbf{z}_t = \phi_{\text{dec}}\left(\mathbf{z}_{t-1}, \begin{bmatrix} \mathbf{E}_y^\top \mathbf{y}_{t-1} \\ \mathbf{c}_t \end{bmatrix}\right)$$ By injecting $\mathbf{c}_t$ directly into the hidden state update, the decoder ensures that its immediate reasoning is guided by a clear and focused view of the input sentence. This prevents long-range details from being lost as the sequence grows.

The design has a key structural benefit: it is behaviorally flexible. The attention mechanism acts as a soft bias rather than a rigid constraint, giving the model the freedom to use complex structural alignments or ignore them if the data requires it. We can see this flexibility by looking at how the model behaves in extreme cases. For example, if $f_{\text{score}}$ always outputs the same value regardless of its input vectors, the attention weights become uniform ($\alpha_{j,t} = \frac{1}{T_x}$), and the context vector simplifies to a basic average: $\mathbf{c}_t = \frac{1}{T_x} \sum \mathbf{h}_j$. The encoder can also learn to mark the boundary states ($\overrightarrow{\mathbf{h}}_{T_x}$ and $\overleftarrow{\mathbf{h}}_1$) with a unique value, causing the attention function to focus on these points at all time steps. In both cases, the system falls back to a simpler encoder-decoder configuration. Since the model can adjust its attention path when needed, we do not need to penalize these simpler paths during training.

Because these attention weights are explicit scalar probabilities between $0$ and $1$, we can collect them for any given source-target pair and plot them as a grayscale 2D alignment matrix $A \in [0, 1]^{\vert{}X\vert{} \times \vert{}Y\vert{}}$:

$$A = \begin{bmatrix} \alpha_{1,1} & \alpha_{1,2} & \dots & \alpha_{1,\vert{}Y\vert{}} \\ \alpha_{2,1} & \alpha_{2,2} & \dots & \alpha_{2,\vert{}Y\vert{}} \\ \vdots & \vdots & \ddots & \vdots \\ \alpha_{\vert{}X\vert{},1} & \alpha_{\vert{}X\vert{},2} & \dots & \alpha_{\vert{}X\vert{},\vert{}Y\vert{}} \end{bmatrix}$$

For English-to-French translation, the network creates clear syntactic and semantic alignments without any explicit phrase-matching supervision. The matrix shows linear word orders as a crisp diagonal line of high-weight cells. Complex grammatical inversions, such as translating the English noun phrase "European Economic Area" to the French "Zone Économique Européenne", are handled through non-monotonic reordering. This shows that the decoder can shift its focus backward and forward over the source words to match the target language's grammar rules.

The practical impact of transitioning to this attention-based design is massive. Benchmarked on the standard WMT'14 English-to-French translation task, the baseline attention model delivers an immediate, dramatic jump in translation quality over the old architecture.

Neural Machine Translation matches and surpasses the translation quality of statistical machine translation systems that were carefully engineered by hand. This is achieved by using dynamic attention tracking and modern scale optimizations, including larger vocabularies and a technique called Byte-Pair Encoding that breaks words into subword units.

Historical Synthesis: Warren Weaver’s Memorandum (1949)

In 1949, mathematician and scientist Warren Weaver wrote a memorandum titled Translation. This was decades before the development of modern computer clusters and deep learning architectures. Despite the early stage of computing at the time, Weaver outlined ideas that closely resemble modern Statistical Machine Translation and Neural Machine Translation. His insights show that modern architectures, such as Bidirectional RNNs and continuous vector embeddings, are based on core linguistic principles established at the beginning of computer science.

The Cryptographic Analogy and Linguistic Independence

Warren Weaver told a story about a mathematician who decrypted a Turkish text without knowing the language. The mathematician relied on the statistical and structural patterns of the symbols. Weaver noted that the key point was that the decoding was done by someone who did not know Turkish. This idea is still the basis for modern data-driven translation engines.

A modern Neural Machine Translation system can work without any prior knowledge of the languages involved. It only needs a large parallel corpus of text. The system does not require hand-crafted rules or dictionaries for the source or target language. By treating text as sequences of characters or subword tokens, the network translates text as a sequence-to-sequence mapping task. The math used for translating English to French is the same as for translating Korean to Arabic. The system works by recognizing patterns in the data.

Context-Dependency and the Eradication of Polysemy

Weaver strongly criticized simple translation tables that rely on direct word-for-word translations. He argued that this approach cannot handle polysemy, where words have multiple meanings. According to Weaver, a translation method that only accounts for a one-to-one correspondence of words will not be useful, especially when dealing with words that have multiple meanings. To address this issue, Weaver suggested looking at a word in context, considering a central word and a certain number of words on either side. He believed that by examining a large enough context, the meaning of the word would become clear.

Modern NMT directly implements Weaver’s window hypothesis, resolving the historical data sparsity problem he accurately predicted. Weaver knew that building a count-based lookup dictionary for a string $2N+1$ words long would be impossible because the number of combinations is "horrifying." Statistical count-based $n$-grams hit a hard generalization wall due to this exact combinatorial explosion. Neural networks bypass this by mapping words into a continuous vector space, smoothing out the probability surface.

By using a Bidirectional Recurrent Neural Network (BiRNN) in the encoder, modern architectures push Weaver's window to its absolute mathematical limit within a sentence ($N \to \infty$). The forward and backward recurrences pass hidden states across the entire sequence, ensuring that every context vector $\mathbf{h}_t$ explicitly represents the central word in relation to all preceding and succeeding tokens.

The Search for an Interlingua: A Universal Code Space

In the final section of his memorandum, Weaver proposed a conceptual bridge for multilingual translation: "Perhaps the way is to descend, from each language, down to the common base of human communication – the real but as yet undiscovered universal language – and then re-emerge by whatever particular route is convenient." This idea is reflected in modern deep learning, where the hidden layer trajectories and context vectors inside encoder-decoder pipelines serve as a universal bridge, often referred to as an Interlingua.

This mathematical space is different from traditional spoken language. It doesn't rely on discrete alphabet symbols or ideograms, but instead exists as a dense, high-dimensional continuous vector space. In this space, semantic meaning and surface-level grammar can be parsed and modified using vector arithmetic.

A basic encoder-decoder can only translate one language pair, but this design can be scaled up. The continuous bottleneck allows a single network to handle multiple languages and tasks, projecting dozens of source languages into a shared vector space before generating text in any target language. Weaver's vision of a universal communication base has been realized, not through hand-crafted grammar rules, but through the weights of a neural network.