Skip to content
Search

Katib

lastmod 2026-10-27created 2023-03-148 edits

Runs as a Kubeflow component and provides Hyperparameter tunning

Components

Consists of Experiment, Suggestion, Trial, and Job

Experiment

A single unit of execution, same as the Job concept. A custom resource in k8s that runs Trials

  • Katib defines and manages one set of HPO as a custom resource called Experiment. An Experimen is the task of searching Hyperparameters for a target value; to find Hyperparameters, you must create an Experiment resource
  • Trial Count : number of runs (parallel)
  • Trial Template : Trial pod template
  • Objective: Defines the goal of Hyperparameters tuning; specifies whether to maximize or minimize the metric during Hyperparameters tuning.
  • Search Parameter: Defines all the Hyperparameters values to use in the Hyperparameters tuning job and the constraints on the Hyperparameters. For example, defines the Learning rate values and that sgd and adam are used as optimizers.
  • Search Algorithm: Defines the algorithm to use for the Hyperparameters tuning job; a variety of algorithms such as Random Search, Grid Search, and Bayesian Optimization are used. The algorithm used when searching for the optimal Hyperparameters values.

Trial

The iteration unit of the optimization process. Creates as many Trials as the Experiment's Trial Count value and runs them sequentially. One Trial runs one worker job

  • A Trial is a custom resource representing one job (unit of execution) for evaluating the suggested Hyperparameters.
  • Assigns the suggested parameter values to a Woker Job instance and runs it.
  • The Experiment keeps running Trials until it reaches the objective or the configured maximum number of trials.

Suggestion

A collection of hyperparameter values generated by the Search Algorithm

  • One Experiment creates one Suggestion
  • Katib creates one Suggestion custom resource for each Experiment custom resource
  • A Suggestion holds the sets of Hyperparameters values suggested by the Search Algorithm
  • Katib creates Trials to evaluate each set of suggested Hyperparameters values

Worker job

The process that uses the parameters and Suggestion values to evaluate and compute each value of a Trial. It actually performs training, using K8s Job, TFJob, PyTorch

Troubleshooting

Invalid value 

[500] admission webhook "validator.experiment.katib.kubeflow.org" denied the request: [spec.trialTemplate.trialParameters[0]: Invalid value: "": name and reference must be specified and name must not contain '{' or '}', spec.trialTemplate.trialParameters[1]: Invalid value: "": name and reference must be specified and name must not contain '{' or '}', spec.trialTemplate: Invalid value: "": parameters: [${trialParameters.learningRate} ${trialParameters.momentum}] in spec.trialTemplate not found in spec.trialParameters: [{learningRate } {momentum }]]

You must finish configuring parameters such as learningRate and momentum. 

[500] admission webhook "validator.experiment.katib.kubeflow.org" denied the request: [spec.trialTemplate.trialParameters[0].reference: Invalid value: "0.01": parameter reference 0.01 does not exist in spec.parameters: [{lr double {0.03 0.01 [] 0.01 uniform}} {num-layers int {64 1 [] 1 uniform}} {optimizer categorical { [sgd adam ftrl] uniform}}], spec.trialTemplate.trialParameters[1].reference: Invalid value: "0.5": parameter reference 0.5 does not exist in spec.parameters: [{lr double {0.03 0.01 [] 0.01 uniform}} {num-layers int {64 1 [] 1 uniform}} {optimizer categorical { [sgd adam ftrl] uniform}}]]

Http failure response for https://kubeflow.yourdomain.com/katib/create_experiment/: 0 Unknown Error 

Http failure response for https://kubeflow.yourdomain.com/katib/create_experiment/: 0 Unknown Error

See also

Favorite site

author Zer0 sitemap