Hyperopt

lastmod 2026-04-24created 2025-12-294 edits

Python library that makes it easy to find hyperparameters

Hyperparameter optimization with Hyperopt

Hyperopt is a Python library that searches a model's hyperparameters using a Bayesian Optimization algorithm. It finds the optimal combination more efficiently than Grid Search or Random Search.

Main steps

Optimization with Hyperopt generally proceeds in the following 4 steps.

1. Define the Objective Function: Define the target function to optimize. Hyperopt works toward minimizing the Loss value this function returns.
2. Set up the Search Space: Set the ranges and distributions of the hyperparameters (using hp.choice, hp.uniform, hp.loguniform, etc.).
3. Choose the optimization algorithm: Usually the TPE (Tree-structured Parzen Estimator) algorithm is used to predict the next candidate based on the results of past trials.
4. Analyze and apply the results: Apply the optimal values (best) obtained through the fmin function to the actual model.

PyTorch example

Code structure that performs optimization by embedding the PyTorch training loop inside the objective function.

import torch
import torch.nn as nn
import torch.optim as optim
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

# 1. 모델 정의
class SimpleNet(nn.Module):
    def __init__(self, input_size):
        super(SimpleNet, self).__init__()
        self.fc = nn.Linear(input_size, 1)

    def forward(self, x):
        return self.fc(x)

# 2. Hyperopt 목적 함수
def objective(params):
    lr = params['lr']
    optimizer_name = params['optimizer']

    model = SimpleNet(input_size=10)
    criterion = nn.MSELoss()

    # Optimizer 선택
    if optimizer_name == 'adam':
        optimizer = optim.Adam(model.parameters(), lr=lr)
    else:
        optimizer = optim.SGD(model.parameters(), lr=lr)

    # 학습 시뮬레이션
    inputs = torch.randn(32, 10)
    targets = torch.randn(32, 1)

    model.train()
    for epoch in range(10):
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()

    # 최소화할 Loss 반환
    return {'loss': loss.item(), 'status': STATUS_OK}

# 3. 탐색 공간 설정
space = {
    'lr': hp.loguniform('lr', -5, -1), # 1e-5 ~ 1e-1
    'optimizer': hp.choice('optimizer', ['adam', 'sgd'])
}

# 4. 실행
trials = Trials()
best = fmin(fn=objective,
            space=space,
            algo=tpe.suggest,
            max_evals=20,
            trials=trials)

print(f"Best parameters: {best}")

Key tips

• Setting the learning rate (LR): Searching on a log scale with hp.loguniform is efficient.
• Checking the history: The Trials object lets you track the detailed results and Loss changes of every trial (iteration).
• Scalability: For deep learning models that take a long time to train, combining with Ray Tune to run parallel searches is recommended.

See also

• MLOps