What Is an Adapter Layer? Inserting Small Trainable Modules Into a Frozen Model
A parameter-efficient fine-tuning approach that inserts small new trainable layers directly into a frozen model, rather than modifying its existing weights.
Small new modules, inserted between existing layers
An adapter layer is a small, new trainable module inserted between the existing layers of a frozen pretrained model — during fine-tuning, only these newly inserted adapter modules get trained, while the original model's parameters remain entirely untouched.
How this differs from LoRA
LoRA adds new parameters that get mathematically combined with existing weights without changing the network's structure. Adapter layers instead physically insert new layers into the network's processing path, meaning every request has to pass through these additional layers during inference.
Why this approach remains useful
Because the base model stays completely frozen and unchanged, the same original model can support many different adapter modules for different tasks, each swapped in independently — a flexible way to maintain multiple specialized capabilities without duplicating the entire underlying model for each one.
Frequently Asked Questions
What is an adapter layer?
A small, new trainable module inserted between the existing layers of a frozen pretrained model — during fine-tuning, only these inserted adapter modules get trained, leaving the original model's parameters untouched.
How is an adapter layer different from LoRA?
LoRA adds parameters that mathematically combine with existing weights without changing the network structure. Adapter layers physically insert new layers into the processing path, meaning requests pass through them during inference.