What Is a Hyperparameter? The Settings That Shape How Training Happens
Not every number in a model is learned from data — hyperparameters are the settings a human chooses beforehand that shape how training itself unfolds.
Settings chosen beforehand, not learned from data
A hyperparameter is a setting chosen before training begins that shapes how the training process itself works — learning rate, batch size, number of layers — as distinct from a parameter, which is a value the model actually learns and adjusts automatically during training itself.
Why the distinction between hyperparameters and parameters matters
Parameters get updated automatically by gradient descent based on training data; hyperparameters have to be set by a human (or a separate automated search process) beforehand, and they directly shape how well and how efficiently that automatic parameter-learning process actually goes.
Why getting hyperparameters right genuinely matters
The exact same model architecture and dataset can produce a noticeably better or worse final model purely based on hyperparameter choices — poor hyperparameter settings can cause training to fail to converge at all, or converge to a meaningfully worse solution than better-chosen settings would have reached.
Frequently Asked Questions
What is a hyperparameter?
A setting chosen before training begins that shapes how the training process works, like learning rate or batch size, as distinct from a parameter, which the model learns and adjusts automatically during training.
Why does getting hyperparameters right matter so much?
The same architecture and dataset can produce a noticeably better or worse model purely based on hyperparameter choices — poor settings can cause training to fail to converge or reach a meaningfully worse solution.