What Is a Feed-Forward Network? The Simplest Building Block Inside a Transformer
Every transformer layer contains a feed-forward network alongside its attention mechanism — a simpler, older type of network doing a specific, essential job.
Information flowing in one direction, no loops
A feed-forward network is the simplest type of neural network — information flows in a single direction, from input through one or more layers to output, with no loops, cycles, or feedback connections anywhere in between, unlike a recurrent neural network.
Its specific role inside a transformer
Every transformer layer contains both an attention sub-layer and a feed-forward sub-layer working together — attention handles mixing information between different token positions, while the feed-forward network processes each token's representation individually afterward, adding further transformation and capacity.
Why both pieces are needed together
Attention alone, without the feed-forward component, would have less capacity to transform individual token representations in complex ways. The two components serve complementary roles — attention for mixing information across positions, feed-forward for processing each position further — which is why both are essential parts of every transformer layer.
Frequently Asked Questions
What is a feed-forward network?
The simplest type of neural network, where information flows in a single direction from input to output with no loops or feedback connections — one of the two core components inside every transformer layer.
What role does the feed-forward network play inside a transformer?
Attention mixes information across token positions; the feed-forward network then processes each token's representation individually afterward, adding further transformation and capacity — the two components serve complementary roles.