Hyperparameter
lastmod 2026-04-22created 2015-12-1810 edits
In Bayesian statistics, a hyperparameter is a parameter of a prior distribution; the term is used to distinguish them from parameters of the model for the underlying system under analysis.
For example, if one is using a beta distribution to model the distribution of the parameter p of a Bernoulli distribution, then:
- p is a parameter of the underlying system (Bernoulli distribution), and
- α and β are parameters of the prior distribution (beta distribution), hence hyperparameters.
One may take a single value for a given hyperparameter, or one can iterate and take a probability distribution on the hyperparameter itself, called a hyperprior.
In the Machine learning
In neural networks (or machine learning), a hyperparameter refers to a variable that is <span style="color:red;">NOT! one of the main variables to be tuned or optimized through neural network training, but rather a variable such as the learning rate (\(\eta\); Learning rate) or the regularization parameter (\(\lambda\)) that people set with a priori knowledge (Priori) or that is set automatically through an external model mechanism.
For that reason, they are also called Meta-parameters or Free-parameters. In Bayesian statistics, the term is also used to represent the prior distribution (Priori distribution).
List of hyperparameter
- Learning rate
- The learning rate used in Gradient descent.
- Regularization parameter
- The regularization parameter (\(\lambda\)), used to avoid the problem of Overfitting, can be used to control the rate of Weight decay
- Mini-batch size
- A large Mini-batch size can be effective when using a parallel computing architecture, while a small size allows more updates.
- Number of Training iterations
- A variable that determines early termination of training (Early stopping) to prevent Overfitting.
- Number of Hidden units
- The more Hidden layers there are, the more the network can be optimized for specific training data. Also, keeping the number of neurons the same in every hidden layer is more effective than varying the number of neurons across the same number of hidden layers. In addition, it is often effective for the number of neurons in the first hidden layer to be greater than the number of neurons in the input layer.
- Weight initialization
- Important because the initial Bias value has a large effect on the training result.
Optimization
Lists four representative methods for Hyperparameter optimization.
- Manual search
- Adjust values heuristically (Heuristic). Time and quality can vary depending on which "search theory" is used.
- Grid search
- Analyze the problem with a priori knowledge, substitute values at regular intervals, and find the optimal value.
- Random search
- Random substitution.
- Bayesian optimization
- A well-known paper is “Practical Bayesian optimization of machine learning algorithms” by Adams et al. The Bayesian optimization method tends to find the optimal value in less time than Random search or Grid search.
Hyperparameter Optimization (HPO)
- Optuna - Currently the industry standard. Supports dynamic search spaces using a "define-by-run" API. Supports advanced algorithms such as the Tree-structured Parzen Estimator (TPE) and can efficiently prune unpromising trials.
- Ray Tune - A powerful library optimized for distributed computing. Ideal for scaling searches from a single laptop to large clusters, and provides state-of-the-art algorithms such as Population Based Training (PBT) and HyperBand/ASHA.
- Hyperopt - One of the oldest libraries using Bayesian optimization. It boasts strong performance, but as of 2026 the open-source version is no longer actively maintained, and many users have migrated to Optuna or Ray Tune.