Kubeflow:Install

lastmod 2026-01-02created 2025-12-21153 edits

Notes on how to install kubeflow.

Prerequisites

v1.10.2

This branch is the master branch targeting Kubernetes version 1.32. (Check with the kubectl version command)
For details on the specific Kubernetes version for each release, see the release notes.
Use a local Kind (installed below) or your own Kubernetes cluster with a default StorageClass .
Kustomize version 5.4.3 or later. (Check with the kubectl version command)
Make sure your Kubectl version is compatible with your Kubernetes cluster (version skew policy).
16GB of RAM is recommended.
8 CPU cores ~~are recommended.~~ <- Recommended, my foot... if you hit an #Insufficient cpu error midway, you're in trouble. Just use 8 cores or more, no exceptions.
kind version 0.27 or later.
You can also use docker, podman, or more recent tools such as those that run OCI images for Kind clusters.
You can exclude components in example/kustomization.yaml to fit Kubeflow into 4–8GB of memory and 2–4 CPU cores.
Linux kernel subsystem changes to support many pods:
- sudo sysctl fs.inotify.max_user_instances=2280
- sudo sysctl fs.inotify.max_user_watches=1255360

Kubeflow Components Versions

v1.11.0

Component	Local Manifests Path	Upstream Revision	CPU (millicores)	Memory (Mi)	PVC Storage (GB)	Added to my k3s cluster
Training Operator	applications/training-operator/upstream	v1.9.2	3m	25Mi	0GB	X
Trainer	applications/trainer/upstream	v2.1.0	8m	143Mi	0GB	X
Notebook Controller	applications/jupyter/notebook-controller/upstream	v1.10.0	5m	93Mi	0GB	O
PVC Viewer Controller	applications/pvcviewer-controller/upstream	v1.10.0	15m	128Mi	0GB	O
Tensorboard Controller	applications/tensorboard/tensorboard-controller/upstream	v1.10.0	15m	128Mi	0GB	O
Central Dashboard	applications/centraldashboard/upstream	v1.10.0	2m	159Mi	0GB	O
Profiles + KFAM	applications/profiles/upstream	v1.10.0	7m	129Mi	0GB	O
PodDefaults Webhook	applications/admission-webhook/upstream	v1.10.0	1m	14Mi	0GB	O
Jupyter Web Application	applications/jupyter/jupyter-web-app/upstream	v1.10.0	4m	231Mi	0GB	O
Tensorboards Web Application	applications/tensorboard/tensorboards-web-app/upstream	v1.10.0				O
Volumes Web Application	applications/volumes-web-app/upstream	v1.10.0	4m	226Mi	0GB	O
Katib	applications/katib/upstream	v0.19.0	13m	476Mi	10GB	O
KServe	applications/kserve/kserve	v0.15.2	600m	1200Mi	0GB	X
KServe Models Web Application	applications/kserve/models-web-app	v0.15.0	6m	259Mi	0GB	X
Kubeflow Pipelines	applications/pipeline/upstream	2.15.0	970m	3552Mi	35GB	X
Kubeflow Model Registry	applications/model-registry/upstream	v0.3.4	510m	2112Mi	20GB	X
Spark Operator	applications/spark/spark-operator	2.4.0	9m	41Mi	0GB	X
Istio	common/istio	1.28.0	750m	2364Mi	0GB	O
Knative	common/knative/knative-serving common/knative/knative-eventing	v1.16.2 v1.16.4	1450m	1038Mi	0GB	X
Cert Manager	common/cert-manager	1.16.1	3m	128Mi	0GB	O
Dex	common/dex	2.43.1	3m	27Mi	0GB	O
OAuth2-Proxy	common/oauth2-proxy	7.10.0	3m	27Mi	0GB	O
Total			4380m	12341Mi	65GB

Install

Download the manifests repository:

git clone https://github.com/kubeflow/manifests.git kubeflow-manifests
cd kubeflow-manifests

## git checkout v1.10.2  ## MinIO 이미지 Pull 이 안되는 이슈가 있다. (ImagePullBackOff)
git checkout v1.11.0

There are two ways:

Go into the #Example installation directory and install everything at once.
Install step by step, starting from the #Step-by-Step installation section.

Uninstall

Remove by namespace:

# Kubeflow 관련 네임스페이스 확인
kubectl get namespaces | grep -E 'kubeflow|istio|knative|cert-manager'

# 특정 네임스페이스 삭제 (설치 환경에 맞게 조정)
kubectl delete namespace kubeflow
kubectl delete namespace kubeflow-user-example-com
kubectl delete namespace istio-system
kubectl delete namespace knative-serving
kubectl delete namespace knative-eventing
kubectl delete namespace cert-manager
kubectl delete namespace auth

Check for remaining CRDs and remove them:

# Kubeflow 관련 CRD 확인
kubectl get crd | grep -E 'kubeflow|istio|knative|cert-manager|seldon'

# 목록 확인 후 문제가 없다면 주의해서 CRD 삭제하자:
kubectl get crd | grep -E 'kubeflow|istio|knative|cert-manager|seldon' | awk '{print $1}' | xargs kubectl delete crd

Check for remaining resources:

kubectl api-resources --verbs=list --namespaced -o name  | xargs -n 1 kubectl get --show-kind --ignore-not-found -A  | grep kubeflow

If the output looks like this:

default       11m         Normal    Stopping    decoratorcontroller/kubeflow-pipelines-profile-controller        Stopping controller: kubeflow-pipelines-profile-controller
default       11m         Normal    Stopped     decoratorcontroller/kubeflow-pipelines-profile-controller        Stopped controller: kubeflow-pipelines-profile-controller
default       11m         Normal    Stopping    decoratorcontroller/kubeflow-pipelines-profile-controller        Stopping controller: kubeflow-pipelines-profile-controller
default       11m         Normal    Stopped     decoratorcontroller/kubeflow-pipelines-profile-controller        Stopped controller: kubeflow-pipelines-profile-controller

These are Kubernetes Event resources. They are transient, log-like resources and are deleted automatically over time.

If you want to clean them up right away, you can delete them like this:

sudo kubectl get events --all-namespaces --field-selector involvedObject.name=kubeflow-pipelines-profile-controller -o name | xargs sudo kubectl delete

Checking resources that are actually running

To check more precisely whether actual resources (not Events) remain:

# 실제 Kubeflow 파드 확인
kubectl get pods --all-namespaces | grep kubeflow

# Kubeflow 서비스 확인
kubectl get svc --all-namespaces | grep kubeflow

# Kubeflow 디플로이먼트 확인
kubectl get deployments --all-namespaces | grep kubeflow

# Kubeflow 스테이트풀셋 확인
kubectl get statefulsets --all-namespaces | grep kubeflow

# Kubeflow CRD 확인
kubectl get crd | grep kubeflow

Example installation

INFORMATION
The `kubectl apply` command may fail on the first try. This is inherent to how Kubernetes and kubectl work (e.g., CRs must be created after CRDs are ready). The fix is to rerun the command until it succeeds. For the one-liner, a bash one-liner that retries the command is provided.

You can install all official Kubeflow components (in the applications subdirectory) and all common services (in the common subdirectory) with the following command.

Check the contents of kustomization.yaml, edit some of the commented parts, then install:

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

For reference, the kustomization.yaml file in v1.10.2 looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

sortOptions:
  order: legacy
  legacySortOptions:
    orderFirst:
    - Namespace
    - ResourceQuota
    - StorageClass
    - CustomResourceDefinition
    - MutatingWebhookConfiguration
    - ServiceAccount
    - PodSecurityPolicy
    - NetworkPolicy
    - Role
    - ClusterRole
    - RoleBinding
    - ClusterRoleBinding
    - ConfigMap
    - Secret
    - Endpoints
    - Service
    - LimitRange
    - PriorityClass
    - PersistentVolume
    - PersistentVolumeClaim
    - Deployment
    - StatefulSet
    - CronJob
    - PodDisruptionBudget
    orderLast:
    - ValidatingWebhookConfiguration

resources:
# Cert-Manager
- ../common/cert-manager/base
- ../common/cert-manager/kubeflow-issuer/base
# Istio
- ../common/istio/istio-crds/base
- ../common/istio/istio-namespace/base
- ../common/istio/istio-install/overlays/oauth2-proxy
# NOTE: For Google Kubernetes Engine (GKE), use:
# - ../common/istio/istio-install/overlays/gke
#GKE mounts `/opt/cni/bin` as read-only for security reasons, preventing the Istio CNI installer from writing the CNI binary.
#Use the GKE-specific overlay: `kubectl apply -k common/istio/istio-install/overlays/gke`.
#This overlay uses GKE's writable CNI directory at `/home/kubernetes/bin`.
#For more details, see [Istio CNI Prerequisites](https://istio.io/latest/docs/setup/additional-setup/cni/#prerequisites) and [Platform Prerequisites](https://istio.io/latest/docs/ambient/install/platform-prerequisites/)
# oauth2-proxy
# NOTE: only uncomment ONE of the following overlays, depending on your cluster type
- ../common/oauth2-proxy/overlays/m2m-dex-only     # for all clusters
#- ../common/oauth2-proxy/overlays/m2m-dex-and-kind # for KIND clusters (allows K8S JWTs for gateway auth)
#- ../common/oauth2-proxy/overlays/m2m-dex-and-eks  # for EKS clusters (NOTE: requires you to configure issuer, see overlay)
# Dex
- ../common/dex/overlays/oauth2-proxy
# KNative
- ../common/knative/knative-serving/overlays/gateways
# Uncomment the following line if `knative-eventing` is required
# - ../common/knative/knative-eventing/base
- ../common/istio/cluster-local-gateway/base
# Kubeflow namespace
- ../common/kubeflow-namespace/base
# NetworkPolicies
- ../common/networkpolicies/base
# Kubeflow Roles
- ../common/kubeflow-roles/base
# Kubeflow Istio Resources
- ../common/istio/kubeflow-istio-resources/base
# Kubeflow Pipelines
- ../applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user
# Katib
- ../applications/katib/upstream/installs/katib-with-kubeflow
# Central Dashboard
- ../applications/centraldashboard/overlays/oauth2-proxy
# Admission Webhook
- ../applications/admission-webhook/upstream/overlays/cert-manager
# Jupyter Web App
- ../applications/jupyter/jupyter-web-app/upstream/overlays/istio
# Notebook Controller
- ../applications/jupyter/notebook-controller/upstream/overlays/kubeflow
# Profiles + KFAM with PSS (Pod Security Standards)
- ../applications/profiles/pss
# PVC Viewer
- ../applications/pvcviewer-controller/upstream/base
# Volumes Web App
- ../applications/volumes-web-app/upstream/overlays/istio
# Tensorboards Controller
- ../applications/tensorboard/tensorboard-controller/upstream/overlays/kubeflow
# Tensorboard Web App
- ../applications/tensorboard/tensorboards-web-app/upstream/overlays/istio
# Training Operator
- ../applications/training-operator/upstream/overlays/kubeflow
# User namespace
- ../common/user-namespace/base
# KServe
- ../applications/kserve/kserve
- ../applications/kserve/models-web-app/overlays/kubeflow
# Spark Operator
- ../applications/spark/spark-operator/overlays/kubeflow

# Ray is an experimental integration
# Here is the documentation for Ray: https://docs.ray.io/en/latest/
# Here is the internal documentation for Ray: - ../experimental/ray/README.md
# - ../experimental/ray/kuberay-operator/overlays/kubeflow

components:
# Pod Security Standards
# https://kubernetes.io/docs/concepts/security/pod-security-standards/
# Uncomment to enable baseline level standards
# - ../experimental/security/PSS/static/baseline
# Uncomment to enable restricted level standards
# - ../experimental/security/PSS/static/restricted
# Uncomment to enable baseline level standards for dynamic namespaces
# - ../experimental/security/PSS/dynamic/baseline
# Uncomment to enable restricted level standards for dynamic namespaces
# - ../experimental/security/PSS/dynamic/restricted

Another good approach is to use the Example above as a reference and install only what you need, Step-by-Step.

How to remove

kubectl delete -k example

Step-by-Step installation

Download the manifests repository and run the installation from the project root directory.

git clone https://github.com/kubeflow/manifests.git kubeflow-manifests
cd kubeflow-manifests
git checkout v1.11.0

INFORMATION
The installation order changed in the move from v1.10.2 -> v1.11.0. The steps below reflect the revised installation order.

Kubeflow namespace

Creates the namespace where the Kubeflow components will live. This namespace is named kubeflow.

kustomize build common/kubeflow-namespace/base | kubectl apply -f -

Output:

namespace/kubeflow created
namespace/kubeflow-system created  ## v1.11.0 에서 추가됨.

Cert-Manager

Cert-manager is used by many Kubeflow components to provide certificates for admission webhooks.

INFORMATION
If you already have cert-manager installed, you can skip base. However, you must manually patch cert-manager with the #pod-security.kubernetes.io/enforce=restricted label.

kustomize build common/cert-manager/base | kubectl apply -f -

## 이거부터 해야 아래 kubeflow-issuer 가 정상 동작됨:
kubectl wait --for=condition=Ready pod -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager

kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -
kubectl wait --for=jsonpath='{.subsets[0].addresses[0].targetRef.kind}'=Pod endpoints -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager

Then check that the Pods are in the Running state:

kubectl get -n cert-manager pods

Output:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5f864bbfd-4cdwm               1/1     Running   0          5m43s
cert-manager-cainjector-589dc747b5-nxlds   1/1     Running   0          5m43s
cert-manager-webhook-5987c7ff58-7qmsd      1/1     Running   0          5m43s

If you get an error like this:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.202.64:443: connect: connection refused

This is because the webhook is not yet ready to receive requests. Wait a few seconds and try applying the manifests again.

Output of each command:

## kustomize build common/cert-manager/base | kubectl apply -f -
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager-webhook created
role.rbac.authorization.k8s.io/cert-manager-tokenrequest created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
rolebinding.rbac.authorization.k8s.io/cert-manager-cert-manager-tokenrequest created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
service/cert-manager created
service/cert-manager-cainjector created
service/cert-manager-webhook created
deployment.apps/cert-manager created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

## kubectl wait --for=condition=Ready pod -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager
pod/cert-manager-5f864bbfd-bh855 condition met
pod/cert-manager-webhook-5987c7ff58-ddhkw condition met

## kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -
clusterissuer.cert-manager.io/kubeflow-self-signing-issuer created

## kubectl wait --for=jsonpath='{.subsets[0].addresses[0].targetRef.kind}'=Pod endpoints -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
endpoints/cert-manager condition met
endpoints/cert-manager-webhook condition met

==== pod-security.kubernetes.io/enforce=restricted ====

Enforcing Pod Security Standards | Kubernetes

What is kubeflow-issuer

An Issuer resource of cert-manager that automatically issues and manages TLS/SSL certificates in a Kubeflow environment.

Defines the certificate issuer
- Specifies the certificate authority (CA), such as Let's Encrypt, Vault, or self-signed
- Kubeflow components automatically request and renew the certificates they need for HTTPS communication
Kubeflow component security
- Applies TLS to services such as Kubeflow Dashboard, Jupyter Notebook, and Katib
- Integrates with Istio Gateway to provide encrypted communication for external access

kubectl apply -k common/cert-manager/kubeflow-issuer/base

Output:

clusterissuer.cert-manager.io/kubeflow-self-signing-issuer created

Most Kubeflow components use Istio to secure their traffic, enforce network permissions, and implement routing policies. This installation uses Istio CNI, which removes the need for privileged init containers and improves compatibility with Pod Security Standards. If your cluster uses Cilium CNI, you must configure it correctly for Istio as described here . Otherwise, you will get RBAC access denied errors in the central dashboard.

First, see the Istio#Platform Requirements section and load the required modules.

echo "Installing Istio CNI configured with external authorization..."
kustomize build common/istio/istio-crds/base | kubectl apply -f -
kustomize build common/istio/istio-namespace/base | kubectl apply -f -

# For most platforms (Kind, Minikube, AKS, EKS, etc.)
kustomize build common/istio/istio-install/overlays/oauth2-proxy | kubectl apply -f -

# For Google Kubernetes Engine (GKE), use:
# kustomize build common/istio/istio-install/overlays/gke | kubectl apply -f -

echo "Waiting for all Istio Pods to become ready..."
kubectl wait --for=condition=Ready pods --all -n istio-system --timeout 300s

By the way, the istio-ingressgateway Pod spent quite a while in ContainerCreating... Checking the events, it seems pulling the gcr.io/istio-release/proxyv2:1.28.0 image takes a while...? Anyway, it took about 6 minutes.

Output of each command:

## kustomize build common/istio/istio-crds/base | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io created
customresourcedefinition.apiextensions.k8s.io/proxyconfigs.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/telemetries.telemetry.istio.io created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/wasmplugins.extensions.istio.io created
customresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io created

## kustomize build common/istio/istio-namespace/base | kubectl apply -f -
namespace/istio-system created

## kustomize build common/istio/istio-install/overlays/oauth2-proxy | kubectl apply -f -
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/istio-reader-service-account created
serviceaccount/istiod created
serviceaccount/istio-cni created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
role.rbac.authorization.k8s.io/istiod created
clusterrole.rbac.authorization.k8s.io/istio-cni created
clusterrole.rbac.authorization.k8s.io/istio-cni-repair-role created
clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system created
clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-istio-system created
clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
rolebinding.rbac.authorization.k8s.io/istiod created
clusterrolebinding.rbac.authorization.k8s.io/istio-cni created
clusterrolebinding.rbac.authorization.k8s.io/istio-cni-repair-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system created
configmap/istio created
configmap/istio-sidecar-injector created
configmap/values created
configmap/istio-cni-config created
service/istio-ingressgateway created
service/istiod created
deployment.apps/istio-ingressgateway created
deployment.apps/istiod created
Warning: spec.template.metadata.annotations[container.apparmor.security.beta.kubernetes.io/install-cni]: deprecated since v1.30; use the "appArmorProfile" field instead
daemonset.apps/istio-cni-node created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istiod created
gateway.networking.istio.io/istio-ingressgateway created
authorizationpolicy.security.istio.io/global-deny-all created
authorizationpolicy.security.istio.io/istio-ingressgateway created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-istio-system created

## kubectl wait --for=condition=Ready pods --all -n istio-system --timeout 300s
pod/istio-ingressgateway-547db5c8dd-kmftd condition met
pod/istiod-5fbdbdb45-nsw68 condition met

Oauth2-proxy

oauth2-proxy extends the functionality of the Istio Ingress-Gateway so that it acts as an OIDC client. It supports user sessions as well as token-based machine-to-machine authentication.

The following overlays are mutually exclusive, so pick and install only one of the options.
For more options, see common/oauth2-proxy/overlays/

Options 1

Works on most clusters, but does not allow using K8s service account tokens from outside the cluster through the Istio Ingress-Gateway.

kustomize build common/oauth2-proxy/overlays/m2m-dex-only/ | kubectl apply -f -
kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy

Output of each command:

## kustomize build common/oauth2-proxy/overlays/m2m-dex-only/ | kubectl apply -f -
namespace/oauth2-proxy created
serviceaccount/oauth2-proxy created
configmap/oauth2-proxy-hk55gm96k4 created
configmap/oauth2-proxy-parameters-74659b6648 created
configmap/oauth2-proxy-theme-5t624ft8b8 created
secret/oauth2-proxy-h675gf55ht created
service/oauth2-proxy created
deployment.apps/oauth2-proxy created
virtualservice.networking.istio.io/oauth2-proxy created
authorizationpolicy.security.istio.io/istio-ingressgateway-oauth2-proxy created
authorizationpolicy.security.istio.io/istio-ingressgateway-require-jwt created
requestauthentication.security.istio.io/dex-jwt created

## kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy
pod/oauth2-proxy-649b9846d8-ddkh6 condition met
pod/oauth2-proxy-649b9846d8-vgpp7 condition met

Options 2

Works with the right configuration on Kind, K3D, Rancher, GKE, and many other clusters, and lets you use Kubernetes service account tokens from outside the cluster through the Istio ingress gateway.

For example, you can use it for automation with GitHub Actions. Ultimately, you need to patch the issuer and jwksUri fields in the request authentication resource in the istio-system namespace, as in the /common/oauth2-proxy/overlays/m2m-dex-and-kind/kustomization.yaml file.

For how to patch, see the #Upgrading and Extending section below.

You can find your cluster's issuer by running curl --insecure -H "Authorization: Bearer `cat /var/run/secrets/kubernetes.io/serviceaccount/token`"https://kubernetes.default/.well-known/openid-configuration from a pod inside the cluster.

kustomize build common/oauth2-proxy/overlays/m2m-dex-and-kind/ | kubectl apply -f -
kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy
kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=cluster-jwks-proxy' --timeout=180s -n istio-system

Options 3

This works on most EKS clusters using K8s service account tokens that can be used from outside the cluster through the Istio Ingress-Gateway.

First, you need to adjust AWS_REGION and CLUSTER_ID in the common/oauth2-proxy/overlays/m2m-dex-and-eks/ file.

kustomize build common/oauth2-proxy/overlays/m2m-dex-and-eks/ | kubectl apply -f -
kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy

How to create and use tokens after installation

Once you finish an installation that includes Kubernetes service account token support, you can create and use tokens.

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
TOKEN="$(kubectl -n $KF_PROFILE_NAMESPACE create token default-editor)"
client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token)
curl -v "localhost:8080/jupyter/api/namespaces/${$KF_PROFILE_NAMESPACE}/notebooks" -H "Authorization: Bearer ${TOKEN}"

To use OAuth2 Proxy without Dex and connect directly to your own IDP, see this document. However, you can also keep Dex and extend it by adding a connector to your own IDP, as described in the #Dex section below.

Dex

Dex is an OpenID Connect (OIDC) identity provider that supports multiple authentication backends. The default installation includes a static user with the email address [email protected]. The default user password is 12341234. For production Kubeflow deployments, you must change the default password by following the relevant section.

kustomize build common/dex/overlays/oauth2-proxy | kubectl apply -f -
kubectl wait --for=condition=Ready pods --all --timeout=180s -n auth

Output of each command:

## kustomize build common/dex/overlays/oauth2-proxy | kubectl apply -f -
namespace/auth created
customresourcedefinition.apiextensions.k8s.io/authcodes.dex.coreos.com created
serviceaccount/dex created
clusterrole.rbac.authorization.k8s.io/dex created
clusterrolebinding.rbac.authorization.k8s.io/dex created
configmap/dex created
secret/dex-oidc-client created
secret/dex-passwords created
service/dex created
deployment.apps/dex created
virtualservice.networking.istio.io/dex created

## kubectl wait --for=condition=Ready pods --all --timeout=180s -n auth
pod/dex-84ccb4b7c8-f22kr condition met

Connecting other identity providers

To connect to the identity provider of your choice (LDAP, GitHub, Google, Microsoft, OIDC, SAML, GitLab), see https://dexidp.io/docs/connectors/oidc/ . Using OIDC is generally recommended because it is compatible with most providers.

For example, the following example uses Azure. You need to edit the https://github.com/kubeflow/manifests/blob/master/common/dex/overlays/oauth2-proxy/config-map.yaml file and add a patches section to the main Kustomization file to add environment variables to the https://github.com/kubeflow/manifests/blob/master/common/dex/base/deployment.yaml file. See the #Upgrading and Extending section for details.

apiVersion: v1
kind: ConfigMap
metadata:
  name: dex
data:
  config.yaml: |
    issuer: https://$KUBEFLOW_INGRESS_URL/dex
    storage:
      type: kubernetes
      config:
        inCluster: true
    web:
      http: 0.0.0.0:5556
    logger:
      level: "debug"
      format: text
    oauth2:
      skipApprovalScreen: true
    enablePasswordDB: true
    #### WARNING: YOU SHOULD NOT USE THE DEFAULT STATIC PASSWORDS
    #### and patch /common/dex/base/dex-passwords.yaml in a Kustomize overlay or remove it
    staticPasswords:
    - email: [email protected]
      hashFromEnv: DEX_USER_PASSWORD
      username: user
      userID: "15841185641784"
    staticClients:
    # https://github.com/dexidp/dex/pull/1664
    - idEnv: OIDC_CLIENT_ID
      redirectURIs: ["/oauth2/callback"]
      name: 'Dex Login Application'
      secretEnv: OIDC_CLIENT_SECRET
    #### Here come the connectors to OIDC providers such as Azure, GCP, GitHub, GitLab, etc.
    #### Connector config values starting with a "$" will read from the environment.
    connectors:
    - type: oidc
      id: azure
      name: azure
      config:
        issuer: https://login.microsoftonline.com/$TENANT_ID/v2.0
        redirectURI: https://$KUBEFLOW_INGRESS_URL/dex/callback
        clientID: $AZURE_CLIENT_ID
        clientSecret: $AZURE_CLIENT_SECRET
        insecureSkipEmailVerified: true
        scopes:
        - openid
        - profile
        - email
        #- groups # groups might be used in the future

For Keycloak, you can find rough guidelines at https://github.com/kubeflow/manifests/blob/master/common/dex/README.md .

KNative

Knative is an open-source tool for building, deploying, and managing serverless applications on top of Kubernetes.

Knative is used by KServe, an official Kubeflow component.

INFORMATION
If you are not going to use KServe, you don't need to install it.

kustomize build common/knative/knative-serving/overlays/gateways | kubectl apply -f -
kustomize build common/istio/cluster-local-gateway/base | kubectl apply -f -

Optionally, you can install Knative Eventing and use it for inference request logging.

kustomize build common/knative/knative-eventing/base | kubectl apply -f -

While running it, I got the following error:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.knative-serving.svc:443/defaulting?timeout=10s": no endpoints available for service "webhook"

It's because the webhook isn't ready yet. Wait a bit and send the same install command again.

Output of each command:

## kustomize build common/knative/knative-serving/overlays/gateways | kubectl apply -f -
## 이건 `ensure CRDs are installed first` 에러 발생해서 몇 번 더 돌려봤음.
namespace/knative-serving unchanged
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/clusterdomainclaims.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/domainmappings.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
serviceaccount/activator unchanged
serviceaccount/controller unchanged
role.rbac.authorization.k8s.io/knative-serving-activator unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-activator-cluster unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-aggregated-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-core unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding unchanged
rolebinding.rbac.authorization.k8s.io/knative-serving-activator unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-activator-cluster unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-addressable-resolver unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin unchanged
configmap/config-autoscaler unchanged
configmap/config-certmanager unchanged
configmap/config-defaults unchanged
configmap/config-deployment unchanged
configmap/config-domain unchanged
configmap/config-features unchanged
configmap/config-gc unchanged
configmap/config-istio unchanged
configmap/config-leader-election unchanged
configmap/config-logging unchanged
configmap/config-network unchanged
configmap/config-observability unchanged
configmap/config-tracing unchanged
secret/net-istio-webhook-certs unchanged
secret/webhook-certs unchanged
service/knative-local-gateway unchanged
service/activator-service unchanged
service/autoscaler unchanged
service/controller unchanged
service/net-istio-webhook unchanged
service/webhook unchanged
deployment.apps/activator configured
deployment.apps/autoscaler configured
deployment.apps/controller configured
deployment.apps/net-istio-controller unchanged
deployment.apps/net-istio-webhook unchanged
deployment.apps/webhook unchanged
poddisruptionbudget.policy/activator-pdb configured
poddisruptionbudget.policy/webhook-pdb configured
horizontalpodautoscaler.autoscaling/activator unchanged
horizontalpodautoscaler.autoscaling/webhook unchanged
image.caching.internal.knative.dev/queue-proxy unchanged
certificate.networking.internal.knative.dev/routing-serving-certs unchanged
destinationrule.networking.istio.io/knative unchanged
gateway.networking.istio.io/knative-local-gateway unchanged
authorizationpolicy.security.istio.io/activator-service unchanged
authorizationpolicy.security.istio.io/autoscaler unchanged
authorizationpolicy.security.istio.io/controller unchanged
authorizationpolicy.security.istio.io/istio-webhook unchanged
authorizationpolicy.security.istio.io/webhook unchanged
peerauthentication.security.istio.io/net-istio-webhook unchanged
peerauthentication.security.istio.io/webhook unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.istio.networking.internal.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev configured
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.istio.networking.internal.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev configured

## kustomize build common/istio/cluster-local-gateway/base | kubectl apply -f -
serviceaccount/cluster-local-gateway-service-account created
role.rbac.authorization.k8s.io/cluster-local-gateway-sds created
rolebinding.rbac.authorization.k8s.io/cluster-local-gateway-sds created
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created
horizontalpodautoscaler.autoscaling/cluster-local-gateway created
gateway.networking.istio.io/cluster-local-gateway created
authorizationpolicy.security.istio.io/cluster-local-gateway created

## kustomize build common/knative/knative-eventing/base | kubectl apply -f -
namespace/knative-eventing created
customresourcedefinition.apiextensions.k8s.io/apiserversources.sources.knative.dev created
customresourcedefinition.apiextensions.k8s.io/brokers.eventing.knative.dev created
customresourcedefinition.apiextensions.k8s.io/channels.messaging.knative.dev created
customresourcedefinition.apiextensions.k8s.io/containersources.sources.knative.dev created
customresourcedefinition.apiextensions.k8s.io/eventpolicies.eventing.knative.dev created
customresourcedefinition.apiextensions.k8s.io/eventtypes.eventing.knative.dev created
customresourcedefinition.apiextensions.k8s.io/jobsinks.sinks.knative.dev created
customresourcedefinition.apiextensions.k8s.io/parallels.flows.knative.dev created
customresourcedefinition.apiextensions.k8s.io/pingsources.sources.knative.dev created
customresourcedefinition.apiextensions.k8s.io/sequences.flows.knative.dev created
customresourcedefinition.apiextensions.k8s.io/sinkbindings.sources.knative.dev created
customresourcedefinition.apiextensions.k8s.io/subscriptions.messaging.knative.dev created
customresourcedefinition.apiextensions.k8s.io/triggers.eventing.knative.dev created
serviceaccount/eventing-controller created
serviceaccount/eventing-webhook created
serviceaccount/job-sink created
serviceaccount/pingsource-mt-adapter created
role.rbac.authorization.k8s.io/knative-eventing-webhook created
clusterrole.rbac.authorization.k8s.io/addressable-resolver configured
clusterrole.rbac.authorization.k8s.io/broker-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/broker-subscriber unchanged
clusterrole.rbac.authorization.k8s.io/builtin-podspecable-binding unchanged
clusterrole.rbac.authorization.k8s.io/channel-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/channel-subscriber unchanged
clusterrole.rbac.authorization.k8s.io/channelable-manipulator configured
clusterrole.rbac.authorization.k8s.io/crossnamespace-subscriber configured
clusterrole.rbac.authorization.k8s.io/eventing-broker-filter unchanged
clusterrole.rbac.authorization.k8s.io/eventing-broker-ingress unchanged
clusterrole.rbac.authorization.k8s.io/eventing-config-reader unchanged
clusterrole.rbac.authorization.k8s.io/eventing-sources-source-observer unchanged
clusterrole.rbac.authorization.k8s.io/flows-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/jobsinks-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/knative-bindings-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-job-sink unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-edit unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-view unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-pingsource-mt-adapter unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-sources-controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-webhook unchanged
clusterrole.rbac.authorization.k8s.io/knative-flows-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-messaging-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-sinks-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-sources-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/meta-channelable-manipulator unchanged
clusterrole.rbac.authorization.k8s.io/podspecable-binding configured
clusterrole.rbac.authorization.k8s.io/service-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/serving-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/source-observer configured
rolebinding.rbac.authorization.k8s.io/eventing-webhook created
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-crossnamespace-subscriber unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-manipulator unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-resolver unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-source-observer unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-sources-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-podspecable-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-resolver unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-eventing-job-sink unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-eventing-pingsource-mt-adapter unchanged
configmap/config-br-default-channel created
configmap/config-br-defaults created
configmap/config-features created
configmap/config-kreference-mapping created
configmap/config-leader-election created
configmap/config-logging created
configmap/config-observability created
configmap/config-ping-defaults created
configmap/config-sugar created
configmap/config-tracing created
configmap/default-ch-webhook created
secret/eventing-webhook-certs created
service/eventing-webhook created
service/job-sink created
deployment.apps/eventing-controller created
deployment.apps/eventing-webhook created
deployment.apps/job-sink created
deployment.apps/pingsource-mt-adapter created
poddisruptionbudget.policy/eventing-webhook created
horizontalpodautoscaler.autoscaling/eventing-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/sinkbindings.webhook.sources.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.eventing.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.eventing.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.eventing.knative.dev created

NetworkPolicies

Installs the network policies.

kustomize build common/networkpolicies/base | kubectl apply -f -

Output:

networkpolicy.networking.k8s.io/cache-server created
networkpolicy.networking.k8s.io/centraldashboard created
networkpolicy.networking.k8s.io/default-allow-same-namespace created
networkpolicy.networking.k8s.io/jupyter-web-app created
networkpolicy.networking.k8s.io/katib-controller created
networkpolicy.networking.k8s.io/katib-db-manager created
networkpolicy.networking.k8s.io/katib-ui created
networkpolicy.networking.k8s.io/kserve created
networkpolicy.networking.k8s.io/kserve-models-web-app created
networkpolicy.networking.k8s.io/metadata-grpc-server created
networkpolicy.networking.k8s.io/metatada-envoy created
networkpolicy.networking.k8s.io/minio created
networkpolicy.networking.k8s.io/ml-pipeline created
networkpolicy.networking.k8s.io/ml-pipeline-ui created
networkpolicy.networking.k8s.io/model-registry created
networkpolicy.networking.k8s.io/model-registry-ui created
networkpolicy.networking.k8s.io/poddefaults created
networkpolicy.networking.k8s.io/pvcviewer-webhook created
networkpolicy.networking.k8s.io/spark-operator-webhook created
networkpolicy.networking.k8s.io/tensorboards-web-app created
networkpolicy.networking.k8s.io/training-operator-webhook created
networkpolicy.networking.k8s.io/volumes-web-app created

Kubeflow Roles

Creates the Kubeflow ClusterRoles: kubeflow-view, kubeflow-edit, and kubeflow-admin. Kubeflow components aggregate permissions into these ClusterRoles.

kustomize build common/kubeflow-roles/base | kubectl apply -f -

Output:

clusterrole.rbac.authorization.k8s.io/kubeflow-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-view created

Kubeflow Istio Resources

Creates the Kubeflow gateway: kubeflow-gateway and the ClusterRole kubeflow-istio-admin

kustomize build common/istio/kubeflow-istio-resources/base | kubectl apply -f -

Output:

clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-view created
gateway.networking.istio.io/kubeflow-gateway created

Kubeflow Pipelines

INFORMATION
From this step on, it eats a ton of compute resources, so brace yourself

ML workflow orchestration

When you need Pipelines:
- When you want to automate and reuse ML workflows
- When you need experiment tracking and version control
- When you build complex ML pipelines
When you don't need Pipelines:
- When you only use Jupyter Notebook
- When you only need model serving
- When you only run simple experiments

v1.10.2

WARNING
Do not use v1.10.2 because of the MinIO issue

Install Multi-User Kubeflow Pipelines, an official Kubeflow component.

This command installs Argo with the runasnonroot emissary executor.

It is still the user's responsibility to analyze the security issues of running containers with root privileges and to decide whether to run the main Kubeflow Pipelines containers as runasnonroot.

In general, it is strongly recommended to run all user-accessible OCI containers with "restricted Pod Security Standards" (Pod Security Standards restricted).

kustomize build applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -

You may get the "ensure CRDs are installed first" error, so be ready to run it several times...

v1.11.0

Kubeflow Pipelines offers two deployment options designed for different use cases and operational preferences.

The traditional database-based approach stores pipeline definitions in an external database, whereas
the Kubernetes native API mode uses Kubernetes custom resources to store and manage pipeline definitions.

As described here, the default artifact store is now SeaweedFS. The single-command installation using the kustomization in example sets up SeaweedFS as the default S3-compatible artifact store for pipelines.

Patching v1.10.2 to use SeaweedFS

This installation replaces the existing setup so that minio-service S3 traffic is routed to SeaweedFS, and patches the Argo Workflow controller to use SeaweedFS. If you are doing the step-by-step installation and want to use SeaweedFS as the pipeline artifact store, apply the following overlay instead of the MinIO-based overlay.

INFORMATION
This installation patches v1.10.2 to use SeaweedFS. So v1.10.2 has experimental/seaweedfs/istio, but v1.11.0 does not. In other words, v1.11.0 just uses it by default, so you don't need to worry about it.

kustomize build experimental/seaweedfs/istio | kubectl apply -f -

To switch back to MinIO, use the standard upstream pipeline overlay shown below.

INFORMATION
MinIO is scheduled to be removed in the next release.

Pipeline Definitions Stored in the Database

(Pipeline definitions are stored in the database)

Install Multi-User Kubeflow Pipelines, an official Kubeflow component.

kustomize build applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -

You'll get the error below, so keep retrying the request:

error: resource mapping not found for name: "kubeflow-pipelines-profile-controller" namespace: "kubeflow" from "STDIN": no matches for kind "DecoratorController" in version "metacontroller.k8s.io/v1alpha1"
ensure CRDs are installed first

First output:

# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2025/12/24 00:30:52 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
serviceaccount/argo created
serviceaccount/kubeflow-pipelines-cache created
serviceaccount/kubeflow-pipelines-container-builder created
serviceaccount/kubeflow-pipelines-metadata-writer created
serviceaccount/kubeflow-pipelines-viewer created
serviceaccount/meta-controller-service created
serviceaccount/metadata-grpc-server created
serviceaccount/ml-pipeline created
serviceaccount/ml-pipeline-persistenceagent created
serviceaccount/ml-pipeline-scheduledworkflow created
serviceaccount/ml-pipeline-ui created
serviceaccount/ml-pipeline-viewer-crd-service-account created
serviceaccount/ml-pipeline-visualizationserver created
serviceaccount/mysql created
serviceaccount/pipeline-runner created
serviceaccount/seaweedfs created
role.rbac.authorization.k8s.io/argo-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role created
role.rbac.authorization.k8s.io/ml-pipeline created
role.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role created
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role created
role.rbac.authorization.k8s.io/ml-pipeline-ui created
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
role.rbac.authorization.k8s.io/pipeline-runner created
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-edit created
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-view created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view created
clusterrole.rbac.authorization.k8s.io/argo-cluster-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-metacontroller created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-view created
clusterrole.rbac.authorization.k8s.io/ml-pipeline created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-ui created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
rolebinding.rbac.authorization.k8s.io/argo-binding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding created
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding created
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding created
configmap/kfp-launcher created
configmap/kubeflow-pipelines-profile-controller-code-42d46f9h7d created
configmap/kubeflow-pipelines-profile-controller-env-5252m69c4c created
configmap/metadata-envoy-configmap created
configmap/metadata-grpc-configmap created
configmap/ml-pipeline-ui-configmap created
configmap/pipeline-api-server-config-dc9hkg52h6 created
configmap/pipeline-install-config created
configmap/workflow-controller-configmap created
secret/mlpipeline-minio-artifact created
secret/mysql-secret created
service/cache-server created
service/kubeflow-pipelines-profile-controller created
service/metadata-envoy-service created
service/metadata-grpc-service created
service/minio-service created
service/ml-pipeline created
service/ml-pipeline-ui created
service/ml-pipeline-visualizationserver created
service/mysql created
service/seaweedfs created
priorityclass.scheduling.k8s.io/workflow-controller created
persistentvolumeclaim/mysql-pv-claim created
persistentvolumeclaim/seaweedfs-pvc created
deployment.apps/cache-server created
deployment.apps/kubeflow-pipelines-profile-controller created
deployment.apps/metadata-envoy-deployment created
deployment.apps/metadata-grpc-deployment created
deployment.apps/metadata-writer created
deployment.apps/ml-pipeline created
deployment.apps/ml-pipeline-persistenceagent created
deployment.apps/ml-pipeline-scheduledworkflow created
deployment.apps/ml-pipeline-ui created
deployment.apps/ml-pipeline-viewer-crd created
deployment.apps/ml-pipeline-visualizationserver created
deployment.apps/mysql created
deployment.apps/seaweedfs created
deployment.apps/workflow-controller created
statefulset.apps/metacontroller created
certificate.cert-manager.io/kfp-cache-cert created
issuer.cert-manager.io/kfp-cache-selfsigned-issuer created
destinationrule.networking.istio.io/metadata-grpc-service created
destinationrule.networking.istio.io/ml-pipeline created
destinationrule.networking.istio.io/ml-pipeline-mysql created
destinationrule.networking.istio.io/ml-pipeline-seaweedfs created
destinationrule.networking.istio.io/ml-pipeline-ui created
destinationrule.networking.istio.io/ml-pipeline-visualizationserver created
virtualservice.networking.istio.io/metadata-grpc created
virtualservice.networking.istio.io/ml-pipeline-ui created
networkpolicy.networking.k8s.io/seaweedfs created
authorizationpolicy.security.istio.io/metadata-grpc-service created
authorizationpolicy.security.istio.io/ml-pipeline created
authorizationpolicy.security.istio.io/ml-pipeline-ui created
authorizationpolicy.security.istio.io/ml-pipeline-visualizationserver created
authorizationpolicy.security.istio.io/mysql created
authorizationpolicy.security.istio.io/seaweedfs-service created
authorizationpolicy.security.istio.io/service-cache-server created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cache-webhook-kubeflow created
error: resource mapping not found for name: "kubeflow-pipelines-profile-controller" namespace: "kubeflow" from "STDIN": no matches for kind "DecoratorController" in version "metacontroller.k8s.io/v1alpha1"
ensure CRDs are installed first

Second output:

# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2025/12/24 00:36:07 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
serviceaccount/argo unchanged
serviceaccount/kubeflow-pipelines-cache unchanged
serviceaccount/kubeflow-pipelines-container-builder unchanged
serviceaccount/kubeflow-pipelines-metadata-writer unchanged
serviceaccount/kubeflow-pipelines-viewer unchanged
serviceaccount/meta-controller-service unchanged
serviceaccount/metadata-grpc-server unchanged
serviceaccount/ml-pipeline unchanged
serviceaccount/ml-pipeline-persistenceagent unchanged
serviceaccount/ml-pipeline-scheduledworkflow unchanged
serviceaccount/ml-pipeline-ui unchanged
serviceaccount/ml-pipeline-viewer-crd-service-account unchanged
serviceaccount/ml-pipeline-visualizationserver unchanged
serviceaccount/mysql unchanged
serviceaccount/pipeline-runner unchanged
serviceaccount/seaweedfs unchanged
role.rbac.authorization.k8s.io/argo-role unchanged
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline unchanged
role.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged
role.rbac.authorization.k8s.io/pipeline-runner unchanged
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-edit unchanged
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-metacontroller unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-edit configured
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-view configured
clusterrole.rbac.authorization.k8s.io/ml-pipeline unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged
rolebinding.rbac.authorization.k8s.io/argo-binding unchanged
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged
configmap/kfp-launcher unchanged
configmap/kubeflow-pipelines-profile-controller-code-42d46f9h7d unchanged
configmap/kubeflow-pipelines-profile-controller-env-5252m69c4c unchanged
configmap/metadata-envoy-configmap unchanged
configmap/metadata-grpc-configmap unchanged
configmap/ml-pipeline-ui-configmap unchanged
configmap/pipeline-api-server-config-dc9hkg52h6 unchanged
configmap/pipeline-install-config unchanged
configmap/workflow-controller-configmap unchanged
secret/mlpipeline-minio-artifact configured
secret/mysql-secret configured
service/cache-server unchanged
service/kubeflow-pipelines-profile-controller unchanged
service/metadata-envoy-service unchanged
service/metadata-grpc-service unchanged
service/minio-service unchanged
service/ml-pipeline unchanged
service/ml-pipeline-ui unchanged
service/ml-pipeline-visualizationserver unchanged
service/mysql unchanged
service/seaweedfs unchanged
priorityclass.scheduling.k8s.io/workflow-controller unchanged
persistentvolumeclaim/mysql-pv-claim unchanged
persistentvolumeclaim/seaweedfs-pvc unchanged
deployment.apps/cache-server configured
deployment.apps/kubeflow-pipelines-profile-controller unchanged
deployment.apps/metadata-envoy-deployment unchanged
deployment.apps/metadata-grpc-deployment unchanged
deployment.apps/metadata-writer configured
deployment.apps/ml-pipeline configured
deployment.apps/ml-pipeline-persistenceagent configured
deployment.apps/ml-pipeline-scheduledworkflow configured
deployment.apps/ml-pipeline-ui configured
deployment.apps/ml-pipeline-viewer-crd configured
deployment.apps/ml-pipeline-visualizationserver unchanged
deployment.apps/mysql unchanged
deployment.apps/seaweedfs unchanged
deployment.apps/workflow-controller unchanged
statefulset.apps/metacontroller configured
certificate.cert-manager.io/kfp-cache-cert unchanged
issuer.cert-manager.io/kfp-cache-selfsigned-issuer unchanged
decoratorcontroller.metacontroller.k8s.io/kubeflow-pipelines-profile-controller created
destinationrule.networking.istio.io/metadata-grpc-service unchanged
destinationrule.networking.istio.io/ml-pipeline unchanged
destinationrule.networking.istio.io/ml-pipeline-mysql unchanged
destinationrule.networking.istio.io/ml-pipeline-seaweedfs unchanged
destinationrule.networking.istio.io/ml-pipeline-ui unchanged
destinationrule.networking.istio.io/ml-pipeline-visualizationserver unchanged
virtualservice.networking.istio.io/metadata-grpc unchanged
virtualservice.networking.istio.io/ml-pipeline-ui unchanged
networkpolicy.networking.k8s.io/seaweedfs configured
authorizationpolicy.security.istio.io/metadata-grpc-service unchanged
authorizationpolicy.security.istio.io/ml-pipeline unchanged
authorizationpolicy.security.istio.io/ml-pipeline-ui unchanged
authorizationpolicy.security.istio.io/ml-pipeline-visualizationserver unchanged
authorizationpolicy.security.istio.io/mysql unchanged
authorizationpolicy.security.istio.io/seaweedfs-service unchanged
authorizationpolicy.security.istio.io/service-cache-server unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/cache-webhook-kubeflow configured

This step takes quite a long time. Keep an eye on the Pod status and wait it out.

kubectl get pod -n kubeflow

.. took about 35 minutes T_T

Pipeline Definitions Stored as Kubernetes Resources

(Pipeline definitions are stored as Kubernetes resources)

Kubeflow Pipelines can be deployed in Kubernetes native API mode, in which pipeline definitions are stored as Kubernetes custom resources (the Pipeline and PipelineVersion kinds) instead of using external storage. This mode improves integration with Kubernetes native tools and GitOps workflows.

Using the KFP SDK with Kubernetes Native API Mode

(Using the KFP SDK in Kubernetes native API mode)

For detailed pipeline compilation instructions, see the Kubeflow Pipelines compilation guide.

Differences in Kubernetes native API mode:

Pipeline definitions are stored in Kubernetes as the custom resource Pipeline .PipelineVersion
Pipeline validation is handled through a Kubernetes admission webhook.
The REST API transparently handles translation into Kubernetes API calls.

Advantages of Kubernetes native mode:

This approach is ideal for organizations that prefer Kubernetes native workflows and want to manage pipelines with standard Kubernetes tools and practices.
Pipeline definitions can be managed through various interfaces, such as kubectl commands, the Kubeflow Pipelines REST API, and the KFP UI, a user-friendly pipeline management tool.

KServe

INFORMATION
KFServing has been rebranded as KServe.

KServe is a component dedicated to model serving (Model Serving).

Deploys trained ML models to production environments
Provides auto-scaling, canary deployment, and more
Creates REST/gRPC API endpoints
When you need KServe:
- When you need to serve a trained model as an API
- When you need infrastructure for production model deployment

Install the KServe component:

kustomize build applications/kserve/kserve | kubectl apply --server-side --force-conflicts -f -

Install the Models web application:

kustomize build applications/kserve/models-web-app/overlays/kubeflow | kubectl apply -f -

"ensure CRDs are installed first" showed up a lot because ClusterServingRuntime didn't exist yet.
A clusterservingruntime.kserve-webhook-server.validator webhook failure was also printed

Output of each command from the final successful run after several retries:

## kustomize build applications/kserve/kserve | kubectl apply --server-side --force-conflicts -f -
customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/localmodelcaches.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/localmodelnodegroups.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/localmodelnodes.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied
serviceaccount/kserve-controller-manager serverside-applied
serviceaccount/kserve-localmodel-controller-manager serverside-applied
serviceaccount/kserve-localmodelnode-agent serverside-applied
role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kserve-localmodel-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kserve-localmodelnode-agent-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kserve-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kserve-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-kserve-view serverside-applied
rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kserve-localmodel-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kserve-localmodelnode-agent-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied
configmap/inferenceservice-config serverside-applied
secret/kserve-webhook-server-secret serverside-applied
service/kserve-controller-manager-metrics-service serverside-applied
service/kserve-controller-manager-service serverside-applied
service/kserve-webhook-server-service serverside-applied
deployment.apps/kserve-controller-manager serverside-applied
deployment.apps/kserve-localmodel-controller-manager serverside-applied
certificate.cert-manager.io/serving-cert serverside-applied
issuer.cert-manager.io/selfsigned-issuer serverside-applied
clusterservingruntime.serving.kserve.io/kserve-huggingfaceserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode serverside-applied
clusterservingruntime.serving.kserve.io/kserve-lgbserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-mlserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-paddleserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-pmmlserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-sklearnserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-tensorflow-serving serverside-applied
clusterservingruntime.serving.kserve.io/kserve-torchserve serverside-applied
clusterservingruntime.serving.kserve.io/kserve-tritonserver serverside-applied
clusterservingruntime.serving.kserve.io/kserve-xgbserver serverside-applied
clusterstoragecontainer.serving.kserve.io/default serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/localmodelcache.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied

## kustomize build applications/kserve/models-web-app/overlays/kubeflow | kubectl apply -f -
serviceaccount/kserve-models-web-app created
clusterrole.rbac.authorization.k8s.io/kserve-models-web-app-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/kserve-models-web-app-binding created
configmap/kserve-models-web-app-config created
service/kserve-models-web-app created
deployment.apps/kserve-models-web-app created
virtualservice.networking.istio.io/kserve-models-web-app created
authorizationpolicy.security.istio.io/kserve-models-web-app created

Katib

Hyperparameter tuning

kustomize build applications/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -

Output:

customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org created
serviceaccount/katib-controller created
serviceaccount/katib-ui created
clusterrole.rbac.authorization.k8s.io/katib-controller created
clusterrole.rbac.authorization.k8s.io/katib-ui created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-view created
clusterrolebinding.rbac.authorization.k8s.io/katib-controller created
clusterrolebinding.rbac.authorization.k8s.io/katib-ui created
configmap/katib-config created
configmap/trial-templates created
secret/katib-mysql-secrets created
service/katib-controller created
service/katib-db-manager created
service/katib-mysql created
service/katib-ui created
persistentvolumeclaim/katib-mysql created
deployment.apps/katib-controller created
deployment.apps/katib-db-manager created
deployment.apps/katib-mysql created
deployment.apps/katib-ui created
certificate.cert-manager.io/katib-webhook-cert created
issuer.cert-manager.io/katib-selfsigned-issuer created
virtualservice.networking.istio.io/katib-ui created
authorizationpolicy.security.istio.io/katib-ui created
mutatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org created
validatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org created

Central Dashboard

Install the Central Dashboard official Kubeflow component:

kustomize build applications/centraldashboard/overlays/oauth2-proxy | kubectl apply -f -

Output:

serviceaccount/centraldashboard created
role.rbac.authorization.k8s.io/centraldashboard created
clusterrole.rbac.authorization.k8s.io/centraldashboard created
rolebinding.rbac.authorization.k8s.io/centraldashboard created
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard created
configmap/centraldashboard-config created
configmap/centraldashboard-parameters created
service/centraldashboard created
deployment.apps/centraldashboard created
virtualservice.networking.istio.io/centraldashboard created
authorizationpolicy.security.istio.io/central-dashboard created

Admission Webhook

Install the Admission Webhook for PodDefaults:

kustomize build applications/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -

Output:

$ kustomize build applications/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org created
serviceaccount/admission-webhook-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding created
service/admission-webhook-service created
deployment.apps/admission-webhook-deployment created
certificate.cert-manager.io/admission-webhook-cert created
issuer.cert-manager.io/admission-webhook-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration created

Notebooks 1.0

Install the Notebook Controller official Kubeflow component.

kustomize build applications/jupyter/notebook-controller/upstream/overlays/kubeflow | kubectl apply -f -

Output:

# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org created
serviceaccount/notebook-controller-service-account created
role.rbac.authorization.k8s.io/notebook-controller-leader-election-role created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-admin created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-edit created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-view created
clusterrole.rbac.authorization.k8s.io/notebook-controller-role created
rolebinding.rbac.authorization.k8s.io/notebook-controller-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding created
configmap/notebook-controller-config-fhm9f7tdt5 created
service/notebook-controller-service created
deployment.apps/notebook-controller-deployment created

Install the official Kubeflow Jupyter web application component.

kustomize build applications/jupyter/jupyter-web-app/upstream/overlays/istio | kubectl apply -f -

Output:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/jupyter-web-app-service-account created
role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-admin created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-edit created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-view created
rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding created
configmap/jupyter-web-app-config-9c2fbg2gdc created
configmap/jupyter-web-app-logos created
configmap/jupyter-web-app-parameters-48gf6bbhmk created
service/jupyter-web-app-service created
deployment.apps/jupyter-web-app-deployment created
destinationrule.networking.istio.io/jupyter-web-app created
virtualservice.networking.istio.io/jupyter-web-app-jupyter-web-app created
authorizationpolicy.security.istio.io/jupyter-web-app created

Workspaces (Notebooks 2.0)

This feature is still in development. <- apparently it's still in development, at least as of v1.11.0

PVC Viewer

Install the PVC Viewer Controller, an official Kubeflow component.

kustomize build applications/pvcviewer-controller/upstream/base | kubectl apply -f -

Output:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
customresourcedefinition.apiextensions.k8s.io/pvcviewers.kubeflow.org created
serviceaccount/pvcviewer-controller-manager created
role.rbac.authorization.k8s.io/pvcviewer-leader-election-role created
clusterrole.rbac.authorization.k8s.io/pvcviewer-metrics-reader created
clusterrole.rbac.authorization.k8s.io/pvcviewer-proxy-role created
clusterrole.rbac.authorization.k8s.io/pvcviewer-role created
rolebinding.rbac.authorization.k8s.io/pvcviewer-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pvcviewer-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pvcviewer-proxy-rolebinding created
service/pvcviewer-controller-manager-metrics-service created
service/pvcviewer-webhook-service created
deployment.apps/pvcviewer-controller-manager created
certificate.cert-manager.io/pvcviewer-serving-cert created
issuer.cert-manager.io/pvcviewer-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/pvcviewer-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/pvcviewer-validating-webhook-configuration created

Profiles + KFAM

Installs the Profile Controller and Kubeflow Access-Management (KFAM), official Kubeflow components.

kustomize build applications/profiles/upstream/overlays/kubeflow | kubectl apply -f -

Output:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Warning: Detected changes to resource profiles.kubeflow.org which is currently being deleted.
customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org configured
serviceaccount/profiles-controller-service-account created
role.rbac.authorization.k8s.io/profiles-leader-election-role created
rolebinding.rbac.authorization.k8s.io/profiles-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-rolebinding created
configmap/namespace-labels-data-4df5t8mdgf created
configmap/profiles-config-5h9m86f79f created
service/profiles-kfam created
deployment.apps/profiles-deployment created
virtualservice.networking.istio.io/profiles-kfam created
authorizationpolicy.security.istio.io/profiles-kfam created

WARNING
"customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org" must print "created". Keep running it until it does~ Otherwise you'll get an error later in #User Namespaces

Volumes Web Application

Install the Volumes web application, an official Kubeflow component.

kustomize build applications/volumes-web-app/upstream/overlays/istio | kubectl apply -f -

Output:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/volumes-web-app-service-account created
clusterrole.rbac.authorization.k8s.io/volumes-web-app-cluster-role created
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-admin created
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-edit created
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-view created
clusterrolebinding.rbac.authorization.k8s.io/volumes-web-app-cluster-role-binding created
configmap/volumes-web-app-parameters-mbftc78hbk created
configmap/volumes-web-app-viewer-spec-gm954c98h6 created
service/volumes-web-app-service created
deployment.apps/volumes-web-app-deployment created
destinationrule.networking.istio.io/volumes-web-app created
virtualservice.networking.istio.io/volumes-web-app-volumes-web-app created
authorizationpolicy.security.istio.io/volumes-web-app created

Tensorboard Web Application

Install the Tensorboards web application official Kubeflow component.

kustomize build applications/tensorboard/tensorboards-web-app/upstream/overlays/istio | kubectl apply -f -

Output:

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/tensorboards-web-app-service-account created
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role created
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-admin created
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-edit created
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-view created
clusterrolebinding.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role-binding created
configmap/tensorboards-web-app-parameters-6ffg2tt572 created
service/tensorboards-web-app-service created
deployment.apps/tensorboards-web-app-deployment created
destinationrule.networking.istio.io/tensorboards-web-app created
virtualservice.networking.istio.io/tensorboards-web-app-tensorboards-web-app created
authorizationpolicy.security.istio.io/tensorboards-web-app created

Tensorboard Controller

Install the Tensorboard Controller official Kubeflow component.

kustomize build applications/tensorboard/tensorboard-controller/upstream/overlays/kubeflow | kubectl apply -f -

Output:

# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
customresourcedefinition.apiextensions.k8s.io/tensorboards.tensorboard.kubeflow.org created
serviceaccount/tensorboard-controller-controller-manager created
role.rbac.authorization.k8s.io/tensorboard-controller-leader-election-role created
clusterrole.rbac.authorization.k8s.io/tensorboard-controller-manager-role created
clusterrole.rbac.authorization.k8s.io/tensorboard-controller-metrics-reader created
clusterrole.rbac.authorization.k8s.io/tensorboard-controller-proxy-role created
rolebinding.rbac.authorization.k8s.io/tensorboard-controller-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-proxy-rolebinding created
configmap/tensorboard-controller-config-7hd244gf2d created
service/tensorboard-controller-controller-manager-metrics-service created
deployment.apps/tensorboard-controller-deployment created

Training Operator

Training Operator v1 (v1.10.2)

Install the Training Operator, an official Kubeflow component.

kustomize build applications/training-operator/upstream/overlays/kubeflow | kubectl apply --server-side --force-conflicts -f -

Output:

customresourcedefinition.apiextensions.k8s.io/jaxjobs.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/mpijobs.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/paddlejobs.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/xgboostjobs.kubeflow.org serverside-applied
serviceaccount/training-operator serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-training-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-training-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-training-view serverside-applied
clusterrole.rbac.authorization.k8s.io/training-operator serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/training-operator serverside-applied
secret/training-operator-webhook-cert serverside-applied
service/training-operator serverside-applied
deployment.apps/training-operator serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validator.training-operator.kubeflow.org serverside-applied

Trainer (Training Operator v2; v1.11.0)

Install Trainer (Training Operator v2), an official Kubeflow component.

kustomize build applications/trainer/upstream/overlays/kubeflow-platform | kubectl apply --server-side --force-conflicts -f -
# kustomize build applications/training-operator/upstream/overlays/kubeflow | kubectl apply --server-side --force-conflicts -f -

"ensure CRDs are installed first" shows up.... you know the drill, right?

First output:

customresourcedefinition.apiextensions.k8s.io/clustertrainingruntimes.trainer.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/trainingruntimes.trainer.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/trainjobs.trainer.kubeflow.org serverside-applied
serviceaccount/jobset-controller-manager serverside-applied
serviceaccount/kubeflow-trainer-controller-manager serverside-applied
role.rbac.authorization.k8s.io/jobset-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-controller-manager serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-view serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-view-cluster-runtimes serverside-applied
rolebinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-trainer-controller-manager serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-trainer-view-cluster-runtimes serverside-applied
configmap/jobset-manager-config serverside-applied
configmap/kubeflow-trainer-config serverside-applied
secret/jobset-webhook-server-cert serverside-applied
secret/kubeflow-trainer-webhook-cert serverside-applied
service/jobset-controller-manager-metrics-service serverside-applied
service/jobset-webhook-service serverside-applied
service/kubeflow-trainer-controller-manager serverside-applied
deployment.apps/jobset-controller-manager serverside-applied
deployment.apps/kubeflow-trainer-controller-manager serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validator.trainer.kubeflow.org serverside-applied
resource mapping not found for name: "deepspeed-distributed" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "mlx-distributed" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "torch-distributed" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "torchtune-llama3.2-1b" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "torchtune-llama3.2-3b" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first
resource mapping not found for name: "torchtune-qwen2.5-1.5b" namespace: "kubeflow" from "STDIN": no matches for kind "ClusterTrainingRuntime" in version "trainer.kubeflow.org/v1alpha1"
ensure CRDs are installed first

The #failed calling webhook "validator.clustertrainingruntime.trainer.kubeflow.org" issue occurred. Final output after fixing it and running again:

customresourcedefinition.apiextensions.k8s.io/clustertrainingruntimes.trainer.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/trainingruntimes.trainer.kubeflow.org serverside-applied
customresourcedefinition.apiextensions.k8s.io/trainjobs.trainer.kubeflow.org serverside-applied
serviceaccount/jobset-controller-manager serverside-applied
serviceaccount/kubeflow-trainer-controller-manager serverside-applied
role.rbac.authorization.k8s.io/jobset-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-controller-manager serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-view serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-trainer-view-cluster-runtimes serverside-applied
rolebinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-trainer-controller-manager serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-trainer-view-cluster-runtimes serverside-applied
configmap/jobset-manager-config serverside-applied
configmap/kubeflow-trainer-config serverside-applied
secret/jobset-webhook-server-cert serverside-applied
secret/kubeflow-trainer-webhook-cert serverside-applied
service/jobset-controller-manager-metrics-service serverside-applied
service/jobset-webhook-service serverside-applied
service/kubeflow-trainer-controller-manager serverside-applied
deployment.apps/jobset-controller-manager serverside-applied
deployment.apps/kubeflow-trainer-controller-manager serverside-applied
clustertrainingruntime.trainer.kubeflow.org/deepspeed-distributed serverside-applied
clustertrainingruntime.trainer.kubeflow.org/mlx-distributed serverside-applied
clustertrainingruntime.trainer.kubeflow.org/torch-distributed serverside-applied
clustertrainingruntime.trainer.kubeflow.org/torchtune-llama3.2-1b serverside-applied
clustertrainingruntime.trainer.kubeflow.org/torchtune-llama3.2-3b serverside-applied
clustertrainingruntime.trainer.kubeflow.org/torchtune-qwen2.5-1.5b serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validator.trainer.kubeflow.org serverside-applied

Spark Operator

Install the Spark Operator:

INFORMATION
The Ray component in the experimental folder is configured to disable Istio sidecar injection for head and worker pods to ensure compatibility with Istio CNI.

kustomize build applications/spark/spark-operator/overlays/kubeflow | kubectl apply --server-side --force-conflicts -f -

Output:

customresourcedefinition.apiextensions.k8s.io/scheduledsparkapplications.sparkoperator.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/sparkapplications.sparkoperator.k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/sparkconnects.sparkoperator.k8s.io serverside-applied
serviceaccount/spark-operator-controller serverside-applied
serviceaccount/spark-operator-webhook serverside-applied
role.rbac.authorization.k8s.io/spark-operator-controller serverside-applied
role.rbac.authorization.k8s.io/spark-operator-webhook serverside-applied
clusterrole.rbac.authorization.k8s.io/spark-operator-controller serverside-applied
clusterrole.rbac.authorization.k8s.io/spark-operator-webhook serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-spark-admin serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-spark-edit serverside-applied
clusterrole.rbac.authorization.k8s.io/kubeflow-spark-view serverside-applied
rolebinding.rbac.authorization.k8s.io/spark-operator-controller serverside-applied
rolebinding.rbac.authorization.k8s.io/spark-operator-webhook serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/spark-operator-controller serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/spark-operator-webhook serverside-applied
service/spark-operator-webhook-svc serverside-applied
deployment.apps/spark-operator-controller serverside-applied
deployment.apps/spark-operator-webhook serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/spark-operator-webhook serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/spark-operator-webhook serverside-applied

User Namespaces

Creates a new namespace for the default user (named kubeflow-user-example-com).

kubectl apply -k common/user-namespace/base

## 아래 명령으로 실행하면 `Invalid value: "$(profile-name)"` 같은 에러가 발생한다.
## kustomize build common/user-namespace/base | kubectl apply -f -

Output:

configmap/default-install-config-9h2h2b6hbk unchanged
profile.kubeflow.org/kubeflow-user-example-com created

If you get an RFC 1123-related error, see the #a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters section below.

Connect to Your Kubeflow Cluster

After installation, it may take a while for all Pods to become ready. Make sure all Pods are ready before trying to connect, or you may run into unexpected errors. Use the following commands to check that all Kubeflow-related Pods are ready.

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n auth
kubectl get pods -n oauth2-proxy
kubectl get pods -n knative-serving
kubectl get pods -n kubeflow
kubectl get pods -n kubeflow-user-example-com

Port-Forward

The default way to access Kubeflow is port-forwarding. It lets you get started quickly without any special requirements for your environment. Run the following command to port-forward Istio's Ingress-Gateway to local port 8080

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

After running the command, you can access the Kubeflow Central dashboard by following these steps.

Open a browser and go to http://localhost:8080/. The Dex login screen should appear.
Log in with the default user credentials. The default email address is [email protected] and the default password is 12341234.

Exposing via the k3s Traefik IngressRoute

Create a kubeflow-ingressroute.yaml file that exposes Kubeflow's Istio Gateway service through a Traefik IngressRoute:

First, use the following command to check the apiVersion:

kubectl api-resources | grep -E 'Middleware|IngressRoute'

When you want to register the host name kubeflow.mydomain.com as a Rule:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: kubeflow-ingress
  namespace: istio-system
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`kubeflow.mydomain.com`)
      kind: Rule
      services:
        - name: istio-ingressgateway
          port: 80
  tls:
    certResolver: default  # Let's Encrypt 사용 시

Check the Istio Gateway service:

kubectl get svc -n istio-system istio-ingressgateway

Output:

NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                    AGE
istio-ingressgateway   ClusterIP   10.43.51.45   <none>        15021/TCP,80/TCP,443/TCP   6h14m

Check that the Traefik deployment is running:

kubectl get deploy -n kube-system traefik

Output:

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
traefik   1/1     1            1           6h41m

Check the traefik service:

kubectl get svc -n kube-system traefik

Output:

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
traefik   LoadBalancer   10.43.176.104   192.168.88.103   80:32235/TCP,443:31309/TCP   6h41m

Add the domain (kubeflow.mydomain.com) to /etc/hosts or a DNS record, then apply:

kubectl apply -f kubeflow-ingressroute.yaml

The External IP is 192.168.88.103; connect through that interface at https://kubeflow.mydomain.com/ . <- k3s is configured to use HTTPS.

Additionally, if needed:

# Check IngressRoute 확인
kubectl get ingressroute -n istio-system

# Check Traefik routes (Traefik dashboard 필요)
kubectl port-forward -n kube-system deployment/traefik 9000:9000

HTTPS redirect Middleware (optional)

# https-redirect-middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: default
spec:
  redirectScheme:
    scheme: https
    permanent: true

Upgrading and Extending

A brief explanation of modifying and upgrading the Kubeflow platform in place is provided for advanced users.

Do not edit the manifest files directly. Use Kustomize overlays and components on top of the example.yaml file .
That way, you can upgrade just by referencing the new manifests, building with Kustomize, and running kubectl apply again.
You may need to adjust your overlays and components if necessary.
You may need to clean up old resources. To do that, add labels to all resources from the start.
With labels, you can use kubectl apply to list prunable resources .--prune--dry-run
Big changes happen from time to time.
- For example, the 1.9 release switched to oauth2-proxy, which needs extra care (istio-system must be cleaned up once).
- Or 1.9.1 -> 1.10 kubectl delete clusterrolebinding meta-controller-cluster-role-binding
But with basic knowledge of Kubernetes, you should be able to carry out the upgrade.

Troubleshooting

ensure CRDs are installed first

error: resource mapping not found for name: "<RESOURCE_NAME>" namespace: "<SOME_NAMESPACE>" from "STDIN": no matches for kind "<CRD_NAME>" in version "<CRD_FULL_NAME>"
ensure CRDs are installed first

This happens because kustomization applies CRDs and CRs very quickly, and the CRD has not been created yet (Established). See kubernetes/kubectl#1117 and helm/helm#4925Established for details.

If you get this error, reapply the component's manifests.

metadata.annotations: Too long: may not be more than xxx bytes

Error from server (Invalid): error when creating "applications/trainer/overlays": CustomResourceDefinition.apiextensions.k8s.io "clustertrainingruntimes.trainer.kubeflow.org" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes

Use --server-side to fix this.

Apply failed with 1 conflict

error: Apply failed with 1 conflict: conflict with "clusterrole-aggregation-controller": .rules
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

As it says, use --force-conflicts.

Warning: resource namespaces/kubeflow is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply

Warning: resource namespaces/kubeflow is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

The warning message printed midway translates as follows:

Resource namespaces/kubeflow is missing the ``kubectl.kubernetes.io/last-applied-configuration`` annotation required by ``kubectl apply``. ``kubectl apply`` should only be used on resources created declaratively via ``kubectl create --save-config`` or ``kubectl apply``. The missing annotation will be patched automatically.

It says it will be patched automatically, so let's ignore it for now.

Error from server (InternalError): Internal error occurred: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator"

The following error is printed while installing #KServe:

Error from server (InternalError): Internal error occurred: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": no endpoints available for service "kserve-webhook-server-service"

The CRDs and webhook configuration were applied successfully, but the endpoints (the actual pods) of kserve-webhook-server-service have not started yet, so validation fails when creating ClusterServingRuntime resources.

# kserve-controller-manager pod 상태 확인
kubectl get pods -n kubeflow -l control-plane=kserve-controller-manager

# Pod가 Running 상태가 될 때까지 대기
kubectl wait --for=condition=Ready pod -l control-plane=kserve-controller-manager -n kubeflow --timeout=300s

# webhook 서비스의 엔드포인트가 준비되었는지 확인
kubectl get endpoints kserve-webhook-server-service -n kubeflow

Several Pods are in the Pending state. kserve-controller-manager hasn't started, so there is no webhook endpoint. You need to find out why they are Pending.

# kserve-controller-manager의 Pending 원인 확인
kubectl describe pod kserve-controller-manager-5fbbbcdd64-wgc66 -n kubeflow

# 다른 Pending Pod들도 확인
kubectl describe pod metacontroller-0 -n kubeflow
kubectl describe pod minio-6d486b66cd-czfw5 -n kubeflow

You will mostly see messages like these:

Insufficient cpu/memory - not enough resources
no nodes available to schedule pods - not enough nodes
persistentvolumeclaim "xxx" not found - PVC problem
node(s) didn't match Pod's node affinity/selector - node selection problem

a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters

The following error is printed in #User Namespaces:

Error from server (Invalid): Profile.kubeflow.org "$(profile-name)" is invalid: metadata.name: Invalid value: "$(profile-name)": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
Retrying to apply resources

If the command you ran was:

kustomize build common/user-namespace/base | kubectl apply -f -

Using this instead

kubectl apply -k .

fixed it.

Error from server (NotFound): the server could not find the requested resource (patch profiles.kubeflow.org $(profile-name))

Error from server (NotFound): the server could not find the requested resource (patch profiles.kubeflow.org $(profile-name))

Problems in v1.10.2

When doing the final status check with kubectl get pod -n kubeflow, you can see a few problems.

minio-6d486b66cd-5hn6w                                   1/2     ImagePullBackOff   0             22m
ml-pipeline-65ff55599d-fmbjv                             1/2     CrashLoopBackOff   9 (30s ago)   22m

Checking ml-pipeline-65ff55599d-fmbjv, its status is CrashLoopBackOff. Check the events with the following command:

kubectl describe pod ml-pipeline-65ff55599d-fmbjv -n kubeflow

The last Events section looks like this:

  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  44m                    default-scheduler  Successfully assigned kubeflow/ml-pipeline-65ff55599d-fmbjv to zer0-z97n-gaming-5
  Normal   Pulled     44m (x2 over 44m)      kubelet            Container image "gcr.io/istio-release/proxyv2:1.26.1" already present on machine
  Normal   Created    44m (x2 over 44m)      kubelet            Created container: istio-validation
  Normal   Started    44m (x2 over 44m)      kubelet            Started container istio-validation
  Normal   Pulled     44m                    kubelet            Container image "gcr.io/istio-release/proxyv2:1.26.1" already present on machine
  Normal   Created    44m                    kubelet            Created container: istio-proxy
  Normal   Started    44m                    kubelet            Started container istio-proxy
  Warning  Unhealthy  44m (x3 over 44m)      kubelet            Startup probe failed:
  Normal   Created    42m (x5 over 44m)      kubelet            Created container: ml-pipeline-api-server
  Normal   Started    42m (x5 over 44m)      kubelet            Started container ml-pipeline-api-server
  Warning  BackOff    4m25s (x202 over 44m)  kubelet            Back-off restarting failed container ml-pipeline-api-server in pod ml-pipeline-65ff55599d-fmbjv_kubeflow(543df7c7-3a50-46d1-821d-f40bd9713112)
  Normal   Pulled     2m41s (x14 over 44m)   kubelet            Container image "ghcr.io/kubeflow/kfp-api-server:2.5.0" already present on machine

All you can see is "Startup probe failed".. Let's check the logs:

kubectl logs ml-pipeline-65ff55599d-fmbjv -n kubeflow -c ml-pipeline-api-server

The output is:

I1223 07:11:44.161982       7 client_manager.go:190] Initializing controller client...
I1223 07:11:44.162246       7 client_manager.go:204] Controller client initialized successfully.
I1223 07:11:44.162258       7 client_manager.go:210] Initializing client manager
I1223 07:11:44.162261       7 client_manager.go:211] Initializing DB client...
I1223 07:11:44.162288       7 config.go:58] Config DBConfig.MySQLConfig.ExtraParams not specified, skipping
I1223 07:11:44.363267       7 client_manager.go:214] DB client initialized successfully
I1223 07:11:44.365553       7 client_manager.go:229] Initializing Object store client...
F1223 07:11:44.366346       7 client_manager.go:550] Failed to check if object store bucket exists. Error: 503 Service Unavailable

Looking at the lines above, the DB connected, but then something about Object Store Bucket blah blah... it says the object storage isn't enabled (503 error)

Seeing that minio-6d486b66cd-5hn6w above is in ImagePullBackOff (Docker Image missing), MinIO was the problem... ugh;;

Edit the corresponding Deployment:

kubectl edit deployment minio -n kubeflow

Find the old MinIO image (gcr.io/ml-pipeline/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance) and replace it with the latest image (minio/minio:RELEASE.2023-09-04T19-57-37Z).

Insufficient cpu

Check the Event section with kubectl describe pod ...:

  Warning  FailedScheduling  4m51s  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

If you see that, it means you're out of CPU, so just give up, no way around it.

Data Dictionary initialization failed

CrashLoopBackOff showed up while installing #Katib, so I checked with logs:

2025-12-24 01:49:47+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.44-1.el9 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2025-12-24T01:49:49.514808Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2025-12-24T01:49:49.516480Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.44) starting as process 1
2025-12-24T01:49:49.579215Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-12-24T01:49:49.801600Z 1 [ERROR] [MY-012960] [InnoDB] Cannot create redo log files because data files are corrupt or the database was not shut down cleanly after creating the data files.
2025-12-24T01:49:49.801737Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2025-12-24T01:49:50.305004Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2025-12-24T01:49:50.315382Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2025-12-24T01:49:50.315419Z 0 [ERROR] [MY-010119] [Server] Aborting
2025-12-24T01:49:50.319269Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.44)  MySQL Community Server - GPL.

Tried reinstalling:

kustomize build applications/katib/upstream/installs/katib-with-kubeflow | kubectl delete -f -
kustomize build applications/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -

Still doesn't work...;;

Tried deleting the PVC and recreating it:

# Katib MySQL Deployment 일시 정지 (Scale down)
kubectl scale deployment katib-mysql -n kubeflow --replicas=0

# PVC 이름 확인
kubectl get pvc -n kubeflow | grep katib-mysql

# PVC 삭제 (보통 이름이 katib-mysql임)
kubectl delete pvc katib-mysql -n kubeflow

# 다시 배포 (Scale up)
kubectl scale deployment katib-mysql -n kubeflow --replicas=1

Checking the Event with describe:

persistentvolumeclaim "katib-mysql" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

Find the katib-mysql PVC and apply it:

kubectl apply -f applications/katib/upstream/components/mysql/pvc.yaml

For reference, its contents are:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: katib-mysql
  namespace: kubeflow
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Still doesn't work...;;

So I compared it with the previous version (v1.10.2) and found the mysql image version had changed from 8.0.29 to 8.0.

So I changed the image version to 8.0.29 and ran it -> still didn't work -> checked the logs

$ kubectl logs -n kubeflow katib-mysql-7766f7694f-mqmf9
2025-12-24 04:07:42+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.29-1.el8 started.
2025-12-24 04:07:43+00:00 [Note] [Entrypoint]: Initializing database files
2025-12-24T04:07:43.277091Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.29) initializing of server in progress as process 23
2025-12-24T04:07:43.402023Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-12-24T04:07:56.963758Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-12-24T04:08:29.260370Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2025-12-24 04:09:27+00:00 [Note] [Entrypoint]: Database files initialized
2025-12-24 04:09:27+00:00 [Note] [Entrypoint]: Starting temporary server
2025-12-24T04:09:27.983274Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 226
2025-12-24T04:09:28.149113Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-12-24T04:09:30.278561Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-12-24T04:09:33.026860Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2025-12-24T04:09:33.026910Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2025-12-24T04:09:33.141083Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2025-12-24T04:09:33.323942Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2025-12-24T04:09:33.346168Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server - GPL.
2025-12-24 04:09:33+00:00 [Note] [Entrypoint]: Temporary server started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.

04:07:42 - Initialization started
04:09:27 - Database file initialization complete (took about 1 min 45 s)
04:09:33 - Temporary server started (on port 0)
Currently loading timezone data - still in progress...

That's the summary. The MySQL 8.0 initialization stages are:

✅ Database files initialization
✅ Temporary server start
🔄 Initial setup tasks (timezone, root password, etc.) ← currently here
⏳ Temporary server shutdown
⏳ Actual MySQL server start (port 3306)

The current probe timings are shorter than the initialization time:

Readiness probe: initialDelaySeconds: 30 ← too short!
Liveness probe: initialDelaySeconds: 60 ← still not enough!

Changing them to 180 seconds (3 minutes) made it work. The changes made in v1.11.0 are:

diff --git a/applications/katib/upstream/components/mysql/mysql.yaml b/applications/katib/upstream/components/mysql/mysql.yaml
index 1dbd3d4e..35cc7547 100644
--- a/applications/katib/upstream/components/mysql/mysql.yaml
+++ b/applications/katib/upstream/components/mysql/mysql.yaml
@@ -24,7 +24,7 @@ spec:
         fsGroupChangePolicy: OnRootMismatch
       containers:
         - name: katib-mysql
-          image: mysql:8.0
+          image: mysql:8.0.29
           args:
             - --datadir
             - /var/lib/mysql/datadir
@@ -47,7 +47,7 @@ spec:
                 - "/bin/bash"
                 - "-c"
                 - "mysql -h 127.0.0.1 -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'"
-            initialDelaySeconds: 10
+            initialDelaySeconds: 180
             periodSeconds: 5
             failureThreshold: 10
           livenessProbe:
@@ -56,7 +56,7 @@ spec:
               - "/bin/bash"
               - "-c"
               - "mysql -h 127.0.0.1 -D ${MYSQL_DATABASE} -u root -p${MYSQL_ROOT_PASSWORD} -e 'SELECT 1'"
-            initialDelaySeconds: 10
+            initialDelaySeconds: 180
             periodSeconds: 5
             failureThreshold: 10
           volumeMounts:

failed calling webhook "validator.clustertrainingruntime.trainer.kubeflow.org"

During the #Trainer (Training Operator v2; v1.11.0) installation, when I ran the following command:

kustomize build applications/trainer/upstream/overlays/kubeflow-platform | kubectl apply --server-side --force-conflicts -f -

the following webhook timeout wouldn't go away:

Error from server (InternalError): Internal error occurred: failed calling webhook "validator.clustertrainingruntime.trainer.kubeflow.org": failed to call webhook: Post "https://kubeflow-trainer-controller-manager.kubeflow.svc:443/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime?timeout=10s": EOF

First, checked the service status ... OK:

$ kubectl get svc -n kubeflow kubeflow-trainer-controller-manager
NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
kubeflow-trainer-controller-manager   ClusterIP   10.43.189.218   <none>        8080/TCP,443/TCP   3h1m

Checked the Endpoint.

$ kubectl get endpoints -n kubeflow kubeflow-trainer-controller-manager
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                                  ENDPOINTS                         AGE
kubeflow-trainer-controller-manager   10.42.0.73:9443,10.42.0.73:8080   3h2m

Check the Training Operator Pod status

kubectl get pods -n kubeflow | grep trainer

Checking the logs:

{"level":"error","ts":"2025-12-24T01:37:10.40758628Z","logger":"cert-rotation","caller":"rotator/rotator.go:336","msg":"could not refresh CA and server certs","error":"Operation cannot be fulfilled on secrets \"kubeflow-trainer-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"..."}
{"level":"info","ts":"2025-12-24T01:37:10.426126643Z","logger":"cert-rotation","caller":"rotator/rotator.go:361","msg":"no cert refresh needed"}
{"level":"error","ts":"2025-12-24T01:37:10.426197365Z","logger":"cert-rotation","caller":"rotator/rotator.go:809","msg":"secret is not well-formed, cannot update webhook configurations","error":"Cert secret is not well-formed, missing ca.crt","errorVerbose":"Cert secret is not well-formed, missing ca.crt\n...","stacktrace":"..."}
...
2025/12/24 01:41:30 http: TLS handshake error from 127.0.0.6:52269: EOF
2025/12/24 01:41:30 http: TLS handshake error from 127.0.0.6:43371: EOF

Core problem: ca.crt is missing from the kubeflow-trainer-webhook-cert Secret, so the TLS certificate was not set up properly.

Check that the Secret's data field has all of ca.crt, tls.crt, and tls.key:

kubectl get secret -n kubeflow kubeflow-trainer-webhook-cert -o yaml

... ??? What?? They're all there? ... Just in case, deleted and reinstalled:

kustomize build applications/trainer/upstream/overlays/kubeflow-platform | kubectl delete -f -
kustomize build applications/trainer/upstream/overlays/kubeflow-platform | kubectl apply --server-side --force-conflicts -f -

... ??? Why does it work now?

Favorite site

author Zer0 sitemap

Kubeflow:Install

Prerequisites

v1.10.2

Kubeflow Components Versions

v1.11.0

Install

Uninstall

Checking resources that are actually running

Example installation

How to remove

Step-by-Step installation

Kubeflow namespace

Cert-Manager

What is kubeflow-issuer

Istio

Oauth2-proxy

Options 1

Options 2

Options 3

How to create and use tokens after installation

Dex

Connecting other identity providers

KNative

NetworkPolicies

Kubeflow Roles

Kubeflow Istio Resources

Kubeflow Pipelines

v1.10.2

v1.11.0

Patching v1.10.2 to use SeaweedFS

Pipeline Definitions Stored in the Database

Pipeline Definitions Stored as Kubernetes Resources

Using the KFP SDK with Kubernetes Native API Mode

KServe

Katib

Central Dashboard

Admission Webhook

Notebooks 1.0

Workspaces (Notebooks 2.0)

PVC Viewer

Profiles + KFAM

Volumes Web Application

Tensorboard Web Application

Tensorboard Controller

Training Operator

Training Operator v1 (v1.10.2)

Trainer (Training Operator v2; v1.11.0)

Spark Operator

User Namespaces

Connect to Your Kubeflow Cluster

Port-Forward

Exposing via the k3s Traefik IngressRoute

Upgrading and Extending

Troubleshooting

ensure CRDs are installed first

metadata.annotations: Too long: may not be more than xxx bytes

Apply failed with 1 conflict

Warning: resource namespaces/kubeflow is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply

Error from server (InternalError): Internal error occurred: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator"

a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters

Error from server (NotFound): the server could not find the requested resource (patch profiles.kubeflow.org $(profile-name))

Problems in v1.10.2

Insufficient cpu

Data Dictionary initialization failed

failed calling webhook "validator.clustertrainingruntime.trainer.kubeflow.org"

See also

Favorite site