What Is Cross-Attention? Letting One Sequence Attend to Another
Self-attention looks within one sequence. Cross-attention looks between two — the mechanism that lets a decoder stay grounded in a separate input, like source text or an image.
Attending across, not just within
Regular self-attention lets each token in a sequence weigh the relevance of other tokens in that same sequence. Cross-attention instead lets tokens in one sequence (say, a decoder generating output) attend to tokens in a completely different sequence (say, the encoder's representation of a source input) — connecting two separate sequences rather than relating one sequence to itself.
Where it shows up
Cross-attention is the connective tissue in an encoder-decoder model: as the decoder generates each output token, cross-attention lets it look back at the full encoded input to decide what's relevant right now, rather than relying purely on what it has generated so far.
Cross-attention with images
The same mechanism lets a vision-language model generate text that attends to specific parts of an image, or lets a diffusion model generation process attend to a text prompt at every denoising step — any time a model needs to stay grounded in a separate input while producing output.
Frequently Asked Questions
What is cross-attention?
An attention mechanism that lets tokens in one sequence attend to tokens in a different sequence, rather than attending within the same sequence — connecting two separate inputs, like a decoder to an encoder's output.
How is cross-attention different from self-attention?
Self-attention relates a sequence to itself; cross-attention relates one sequence to a different one entirely, such as a generated response attending back to a source document or image.