What Is Positional Encoding? How Transformers Know Word Order
Attention alone treats input as an unordered set — positional encoding is what tells a transformer which word came first, second, and so on.
The gap attention leaves open
The core attention mechanism computes relevance between tokens without any inherent notion of order — as far as raw attention is concerned, "the cat sat" and "sat the cat" could look identical. Positional encoding fixes this by adding information about each token's position in the sequence before attention runs.
How position gets encoded
A positional encoding adds a pattern of values to each token's representation that varies systematically with its position, giving the model a way to tell tokens apart by where they occur, not just what they are. Different transformer designs implement this differently, including newer approaches like rotary position embedding.
Why this matters for meaning
Word order changes meaning constantly in language ("dog bites man" versus "man bites dog"), so without positional information, a transformer would be unable to distinguish sentences that use identical words in a different order — positional encoding is what makes the architecture order-aware at all.
Frequently Asked Questions
What is positional encoding in a transformer?
A technique that adds position information to each token's representation before attention runs, since attention alone has no inherent sense of token order.
Why does word order matter to a language model?
The same words in a different order can mean something completely different ("dog bites man" vs. "man bites dog"), so a model needs positional information to distinguish these cases correctly.