3:2
The transformer is a neural network architecture introduced in 2017 that underlies nearly all modern large language models, as well as many image and audio models. Its key innovation is the attention mechanism, which allows the model to weigh the relevance of every other token in a sequence when processing each token, rather than processing text strictly one step at a time as older recurrent architectures did.
Before transformers, sequence models like recurrent neural networks (RNNs) processed text one token at a time, carrying forward a hidden state from each step to the next. This made them slow to train (since each step depends on the previous one, limiting parallelization) and prone to "forgetting" information from earlier in long sequences. Transformers process an entire sequence at once and use attention to directly relate any two positions in the sequence regardless of distance, which is both far more parallelizable on modern hardware and better at capturing long-range dependencies in text.
Before a transformer can process text, it must be tokenized (split into sub-word units) and each token converted into a vector called an embedding. These embeddings are learned during training and capture semantic relationships - tokens used in similar contexts end up with similar embedding vectors, giving the model a numeric representation it can perform mathematical operations on.
Self-attention is the mechanism that lets each token "look at" every other token in the sequence and decide how much attention to pay to it when building its own representation. For each token, the model computes three vectors - a query, a key, and a value - and uses the similarity between queries and keys to determine attention weights, which are then used to combine the values into a new, context-aware representation of that token.
Attention(Q, K, V) = softmax(Q · K^T / sqrt(d_k)) · VIn practice, this means a word like "bank" can end up with a different internal representation depending on whether nearby tokens suggest a riverbank or a financial institution, since attention lets the surrounding context directly influence how that token is encoded.
Rather than computing attention once, transformers compute it multiple times in parallel using different learned projections, called attention heads. Each head can learn to focus on a different kind of relationship - one might track grammatical structure, another might track long-range topical relevance - and their outputs are combined afterward. This gives the model a richer, multi-faceted view of how tokens relate to each other than a single attention computation could provide.
Because self-attention treats a sequence as a set of tokens rather than an ordered list, it has no inherent sense of word order on its own. To fix this, transformers add positional encodings - vectors that represent each token's position in the sequence - to the token embeddings before processing, giving the model the information it needs to distinguish "the dog bit the man" from "the man bit the dog."
The original transformer paper described an encoder-decoder architecture, where an encoder processes an input sequence and a decoder generates an output sequence based on it (useful for tasks like translation). Many modern LLMs use only the decoder half, trained to predict the next token in a sequence, which is well suited to open-ended text generation. Encoder-only models, by contrast, are commonly used for tasks like classification or search, where the goal is to produce a rich representation of input text rather than generate new text.
By: Tomas Silny
Edited: 2026-08-13 06:49:58