What is Transformer?
The neural-network architecture behind most modern language and multimodal models.
The transformer is a neural-network architecture, introduced in 2017, that uses an "attention" mechanism to weigh how words relate to each other.
It's the foundation of almost every modern LLM and many image and audio models.
The transformer is the neural-network architecture, introduced in 2017, that underpins almost every modern language model and many image and audio models. Its key innovation is the "attention" mechanism, which lets the model weigh how strongly each part of the input relates to every other part — so when processing a word, it can focus on the other words that give it meaning, no matter how far apart they are. This solved a long-standing problem of earlier architectures that struggled to connect distant pieces of context.
Attention also made models far more parallelisable to train, which is what allowed them to scale to the enormous sizes that produced today's capabilities. Practically, you don't need to know the maths, but it helps to know that the "GPT" in many model names stands for Generative Pre-trained Transformer — the transformer is the engine, and the breakthroughs in language, image and multimodal AI of the last several years largely trace back to it.
Why it matters
The transformer is the foundational architecture behind the modern AI era, so the term shows up constantly. Knowing it's the design that lets models weigh relationships across a whole input — and that it's what "GPT" refers to — demystifies a lot of AI terminology and explains why models improved so dramatically once this architecture arrived.
A concrete example
In the sentence "the trophy didn't fit in the suitcase because it was too big," understanding what "it" refers to requires connecting distant words. The transformer's attention mechanism lets the model weigh that "it" relates to "trophy," not "suitcase" — the kind of long-range understanding that earlier architectures handled poorly.
Where you’ll meet it in AI tools
You won't see "transformer" in a product's interface, but it's the architecture inside nearly every AI tool you use — language assistants, coding copilots, image generators and more. The model names tools advertise (GPT-family and others) are transformer-based, so the term is most useful for understanding the technology underneath rather than for choosing a tool.
Transformer: FAQ
What does the transformer architecture do?
It lets a model weigh how every part of an input relates to every other part, using a mechanism called attention, so it can understand context and long-range relationships in text (and other data) far better than earlier designs. It also trains efficiently at massive scale. Together these properties are why transformers power almost all modern language and many multimodal models — they made it possible to build models that genuinely understand context and to scale them to today's capabilities.
Is GPT a transformer?
Yes — "GPT" stands for Generative Pre-trained Transformer, so the transformer architecture is literally part of the name. The GPT family, and the vast majority of modern large language models from every provider, are built on transformers. The transformer is the underlying engine; specific models differ in size, training and tuning, but share that foundational architecture introduced in 2017.
Related terms
Find language models tools
Browse hand-reviewed AI tools — compare on pricing, features and real ratings.