Lightweight Kubernetes

lastmod 2026-10-27created 2022-07-0894 edits

K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.

Categories

• k8s

Secret

• SealedSecret

Volumes

• Longhorn

Sidecar

• git-sync

Requirements

• Requirements | K3s

K3s is very lightweight, but has some minimum requirements as outlined below.

Whether you're configuring K3s to run in a container or as a native Linux service, each node running K3s should meet the following minimum requirements.

These requirements are baseline for K3s and its packaged components, and do not include resources consumed by the workload itself.

Prerequisites

Two nodes cannot have the same hostname.

If multiple nodes will have the same hostname, or if hostnames may be reused by an automated provisioning system, use the --with-node-id option to append a random suffix for each node,

or devise a unique name to pass with --node-name or $K3S_NODE_NAME for each node you add to the cluster.

Firewall

Turn off the ufw firewall.

Hardware

Networking

The k3s server needs port 6443 to be accessible by all nodes.

• Nodes need to be able to reach other nodes over UDP port 8472 when using the Flannel VXLAN backend,
• or over UDP port 51820 (and 51821 if IPv6 is used) when using the Flannel WireGuard backend.

Nodes should not listen on any other port.

K3s uses reverse tunneling so that nodes make outbound connections to the server and all kubelet traffic runs through that tunnel.

However, if you do not use Flannel and provide your own custom CNI, the ports needed by Flannel are not needed by K3s.

If you wish to utilize the metrics server, all nodes must be accessible to each other on port 10250.

If you plan on achieving high availability with embedded etcd, server nodes must be accessible to each other on ports 2379 and 2380.

Inbound Rules for K3s Nodes

• Requirements | K3s # Inbound Rules for K3s Nodes

Typically, all outbound traffic is allowed.

Additional changes to the firewall may be required depending on the OS used.

iptables settings

iptables rules for K3s Server nodes:

#!/bin/bash

# K3s Server Node iptables Rules
# K3s 서버 노드 iptables 규칙

# Allow K3s API Server / K3s API 서버 허용
iptables -A INPUT -p tcp --dport 6443 -j ACCEPT

# Allow kubelet metrics / kubelet 메트릭 허용
iptables -A INPUT -p tcp --dport 10250 -j ACCEPT

# Allow etcd for HA (Server to Server only) / HA를 위한 etcd 통신 (서버간만)
iptables -A INPUT -p tcp --dport 2379:2380 -s <SERVER_IP_RANGE> -j ACCEPT

# Allow Flannel VXLAN (if using VXLAN backend) / Flannel VXLAN 허용 (VXLAN 백엔드 사용시)
iptables -A INPUT -p udp --dport 8472 -j ACCEPT

# Allow Flannel WireGuard IPv4 (if using WireGuard backend) / Flannel WireGuard IPv4 허용
iptables -A INPUT -p udp --dport 51820 -j ACCEPT

# Allow Flannel WireGuard IPv6 (if using WireGuard with IPv6) / Flannel WireGuard IPv6 허용
iptables -A INPUT -p udp --dport 51821 -j ACCEPT

# Allow Spegel distributed registry (if enabled) / Spegel 분산 레지스트리 허용
iptables -A INPUT -p tcp --dport 5001 -j ACCEPT

# Allow Pod network / Pod 네트워크 허용 (기본 CIDR: 10.42.0.0/16)
iptables -A INPUT -s 10.42.0.0/16 -j ACCEPT

# Allow Service network / Service 네트워크 허용 (기본 CIDR: 10.43.0.0/16)
iptables -A INPUT -s 10.43.0.0/16 -j ACCEPT

# Save rules / 규칙 저장
# Debian/Ubuntu:
# netfilter-persistent save
# RHEL/CentOS:
# service iptables save

iptables rules for K3s Agent nodes:

#!/bin/bash

# K3s Agent Node iptables Rules
# K3s 에이전트 노드 iptables 규칙

# Allow kubelet metrics / kubelet 메트릭 허용
iptables -A INPUT -p tcp --dport 10250 -j ACCEPT

# Allow Flannel VXLAN (if using VXLAN backend) / Flannel VXLAN 허용
iptables -A INPUT -p udp --dport 8472 -j ACCEPT

# Allow Flannel WireGuard IPv4 (if using WireGuard backend) / Flannel WireGuard IPv4 허용
iptables -A INPUT -p udp --dport 51820 -j ACCEPT

# Allow Flannel WireGuard IPv6 (if using WireGuard with IPv6) / Flannel WireGuard IPv6 허용
iptables -A INPUT -p udp --dport 51821 -j ACCEPT

# Allow Spegel distributed registry (if enabled) / Spegel 분산 레지스트리 허용
iptables -A INPUT -p tcp --dport 5001 -j ACCEPT
iptables -A INPUT -p tcp --dport 6443 -j ACCEPT

# Allow Pod network / Pod 네트워크 허용
iptables -A INPUT -s 10.42.0.0/16 -j ACCEPT

# Allow Service network / Service 네트워크 허용
iptables -A INPUT -s 10.43.0.0/16 -j ACCEPT

# Save rules / 규칙 저장
# netfilter-persistent save (Debian/Ubuntu)
# service iptables save (RHEL/CentOS)

Requirements: Large Clusters

Hardware requirements are based on the size of your K3s cluster. For production and large clusters, we recommend using a high-availability setup with an external database. The following options are recommended for the external database in production:

• MySQL
• PostgreSQL
• etcd

CPU and Memory

The following are the minimum CPU and memory requirements for nodes in a high-availability K3s server:

Disk

Cluster performance depends on database performance. To ensure optimal speed, we recommend always using SSD disks to back your K3s cluster. On cloud providers, we also recommend using the minimum size that allows the maximum IOPS.

Network

You should consider increasing the subnet size for the cluster CIDR so that you don't run out of IPs for the pods. You can do this by passing the --cluster-cidr option to the K3s server upon starting.

Database

K3s supports a variety of databases, including MySQL, PostgreSQL, MariaDB, and etcd. See Cluster Datastore for details.

The following is a sizing guide for the database resources needed to run large clusters:

Installation

• (Machine translation) K3s : Quick-Start Guide – 098

# 기본 설치
curl -sfL https://get.k3s.io | sh -

# 권한 설치
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

After running this installation:

• The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.
• Additional utilities will be installed, including kubectl, crictl, ctr, k3s-killall.sh, and k3s-uninstall.sh.
• A kubeconfig file will be written to /etc/rancher/k3s/k3s.yaml and the kubectl installed by K3s will automatically use it.

The service runs as k3s server ....

• Service file /etc/systemd/system/k3s.service
• Environment variable file /etc/systemd/system/k3s.service.env

Changing the data storage location

The default location is /var/lib/rancher/k3s. Add a command-line argument such as --data-dir /mnt/myDisk/... at install time. Note that the node-token location changes too.

Installing as a worker node

• Add new worker node on an existing k3s cluster
• Unable to setup cluster with 1 server and 1 agent · Issue #4839 · k3s-io/k3s

To install on worker nodes and add them to the cluster, run the installation script with the K3S_URL and K3S_TOKEN environment variables. Here is an example showing how to join a worker node:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

Setting the parameter causes K3s to run in worker mode. (The service runs as k3s agent ....)

The K3s agent will register with the K3s server listening at the supplied URL.

The value to use for K3S_TOKEN is stored at /var/lib/rancher/k3s/server/node-token on your server node.

• Service file /etc/systemd/system/k3s-agent.service
• Environment variable file /etc/systemd/system/k3s-agent.service.env
• The uninstall script is saved at /usr/local/bin/k3s-agent-uninstall.sh.

Run kubectl get nodes on the master node to see the added node, as follows:

NAME           STATUS   ROLES                  AGE   VERSION
api-your-run   Ready    control-plane,master   11m   v1.29.4+k3s1
answerver      Ready    <none>                 13s   v1.29.4+k3s1

High Availability Embedded etcd

• High Availability Embedded etcd | K3s

curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --cluster-init

To run K3s in this mode, you must have an odd number of server nodes. We recommend starting with three nodes.

To get started, first launch a server node with the --cluster-init flag to enable clustering, and a token that will be used as a shared secret to join additional servers to the cluster.

• Network related flags: --cluster-dns, --cluster-domain, --cluster-cidr, --service-cidr
• Flags controlling the deployment of certain components: --disable-helm-controller, --disable-kube-proxy, --disable-network-policy and any component passed to --disable
• Feature related flags: --secrets-encryption

My case

See Blackhole#k3s.

WSL

• Install K3s on Windows - Autocloud: GitOps Kubernetes Bootstrapper
• k3s on WSL2
• Basic kubectl setup and troubleshooting guide after installing k3s on WSL | Choi Minwoo

Configuration Options

• Configuration Options | K3s
• server | K3s
• agent | K3s

--server {서버URL}

Option that specifies "which cluster to join".

After --cluster-init, this option must be specified from the second cluster server onward.

If set to https://10.128.0.2:6443, new nodes can no longer join once that node dies. If set to https://kube.yourdomain.com:6443 instead, it can be replaced by an LB on the domain. The downside is the hassle of setting up the domain.

--tls-san {IP or Hostname}

TLS-SAN = TLS Subject Alternative Name. A list of domains/IPs to add to the API server's TLS certificate. Used to avoid certificate errors when accessing the cluster in multiple ways. Can be specified multiple times.

Must be applied identically on all server nodes. e.g. --tls-san kube.yourdomain.com --tls-san 136.115.171.248 --tls-san 136.112.206.68 --tls-san 104.154.24.148

--node-ip vs --node-external-ip

• --node-ip
  • Internal cluster communication (actual traffic) - e.g. 10.128.0.2 (Private IP)
  • IP actually used for Kubernetes internal communication
  • Pod-to-pod communication
  • Node-to-node communication (etcd, kubelet, etc.)
  • Used as the Service endpoint
• --node-external-ip
  • Metadata for external access (for display) - e.g. 136.115.171.248 (Public IP)
  • Mainly for displaying information
  • Shown when querying node information in kubectl
  • For reference when accessing the node over SSH from outside
  • Can be referenced by LoadBalancer type services

Can be checked with the following command:

$ kubectl get nodes -o wide

Output:

NAME                       STATUS   ROLES                       AGE     VERSION        INTERNAL-IP   EXTERNAL-IP       OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
instance-20251229-070757   Ready    control-plane,etcd,master   4h24m   v1.33.6+k3s1   10.128.0.2    136.115.171.248   Ubuntu 24.04.3 LTS   6.14.0-1021-gcp   containerd://2.1.5-k3s1.33
instance-20251229-071544   Ready    control-plane,etcd,master   4h23m   v1.33.6+k3s1   10.128.0.3    136.112.206.68    Ubuntu 24.04.3 LTS   6.14.0-1021-gcp   containerd://2.1.5-k3s1.33
instance-20251229-072735   Ready    control-plane,etcd,master   4h23m   v1.33.6+k3s1   10.128.0.4    104.154.24.148    Ubuntu 24.04.3 LTS   6.14.0-1021-gcp   containerd://2.1.5-k3s1.33

How agent node registration works

• Architecture | K3s

Agent nodes are registered with a websocket connection initiated by the k3s agent process, and the connection is maintained by a client-side load balancer running as part of the agent process. This load balancer maintains stable connections to all servers in the cluster, providing a connection to the supervisor that tolerates outages of individual servers.

Agents register with the server using the node cluster secret along with a randomly generated password for the node, stored at /etc/rancher/node/password. The server stores the passwords for individual nodes as Kubernetes secrets, and any subsequent attempts must use the same password. Node password secrets are stored in the kube-system namespace with names using the template <host>.node-password.k3s. This is done to protect the integrity of node IDs.

If the /etc/rancher/node directory of an agent is removed, or you wish to rejoin a node using an existing name, the node should be deleted from the cluster. This will clean up both the old node entry and the node password secret, and allow the node to (re)join the cluster. Note

If you frequently reuse hostnames but are unable to remove the node password secrets, a unique node ID can be automatically appended to the hostname by launching K3s servers or agents with the --with-node-id flag. When enabled, the node ID is also stored in /etc/rancher/node/.

Features

K3s is a fully compliant Kubernetes distribution with the following enhancements:

• Packaged as a single binary.
• Lightweight storage backend based on sqlite3 as the default storage mechanism. etcd3, MySQL, and Postgres are also available.
• Wrapped in a simple launcher that handles a lot of the complexity of TLS and options.
• Secure by default with reasonable defaults for lightweight environments.
• Simple but powerful "batteries-included" features have been added, such as:
  • Local storage provider
  • Service load balancer
  • Helm controller
  • Traefik ingress controller.
  • Longhorn open-source distributed block storage system.
• Operation of all Kubernetes control plane components is encapsulated in a single binary and process. This allows K3s to automate and manage complex cluster operations like distributing certificates.
• External dependencies have been minimized (just a modern kernel and cgroup mounts needed). K3s packages the required dependencies, including:
  • containerd
  • Flannel (CNI)
  • CoreDNS
  • Traefik (Ingress)
  • Klipper-lb (Service LB)
  • Embedded network policy controller
  • Embedded local-path-provisioner
  • Host utilities (iptables, socat, etc)

Default installed packages

• CoreDNS
• Traefik
• local-storage

Commands

• server - Run management server
• agent - Run node agent
• kubectl - Run kubectl
• crictl - Run crictl
• ctr - Run ctr
• check-config - Run config check
• etcd-snapshot - Trigger an immediate etcd snapshot
• secrets-encrypt - Control secrets encryption and keys rotation
• certificate - Certificates management
• completion - Install shell completion script
• help - Shows a list of commands or help for one command

Remove

/usr/local/bin/k3s-uninstall.sh

Cluster Datastore

• Cluster Datastore | K3s
• Building multi kubernetes with K3s

K3s has two types:

• K3s server, which includes the control plane
• K3s agent, which acts only as a worker node without the control plane.

There are three ways to configure a K3s cluster.

• #Single K3s server with an embedded DB
• #Multiple K3s servers with an embedded DB
• #Multiple K3s servers with an external DB

Single K3s server with an embedded DB

The figure below shows how to configure a cluster with a single K3s server.

The control plane runs on just a single node, and K3s agents (workers) are configured around it.

Users can access the cluster only through the single endpoint of the K3s server.

As anyone can easily guess, configuring the control plane as a single node is a big problem for high availability.

If the K3s server dies, the cluster can no longer be accessed or managed, so operations such as pod scheduling will stop.

Naturally, this can be solved by configuring multiple K3s servers.

Multiple K3s servers can be configured either with the internal embedded DB or with an external DB, a feature specific to K3s.

Multiple K3s servers with an embedded DB

To solve the problems described above you can configure multiple K3s servers; first, the configuration using the embedded DB is described.

To form the cluster, you basically need an odd number of servers, three or more. These servers provide the control plane and the Kubernetes API Server.

The embedded DB approach uses sqlite on a single node, but etcd on multiple nodes.

Multiple K3s servers with an external DB

Unlike the embedded DB configuration, using an external DB forms the cluster with a DB that exists outside the K3s servers.

MySQL, PostgreSQL, or an external etcd can be used.

Checking the installation status

sudo systemctl status k3s

Checking the cluster status

Check with kubectl. (kubectl is available once k3s is installed.)

# node 상태 확인
sudo kubectl get node

# pod, svc 상태 확인
sudo kubectl get all

# 모든 Namespace 상태 확인
sudo kubectl get -A all

Example output of sudo kubectl get -A all:

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-6d668d687-sk8z4                   1/1     Running     0          3d17h
kube-system   pod/helm-install-traefik-btnrc                0/1     Completed   1          3d17h
kube-system   pod/helm-install-traefik-crd-thplf            0/1     Completed   0          3d17h
kube-system   pod/local-path-provisioner-869c44bfbd-khd2x   1/1     Running     0          3d17h
kube-system   pod/metrics-server-7bfffcd44-bp7lr            1/1     Running     0          3d17h
kube-system   pod/svclb-traefik-33133399-bhx7l              2/2     Running     0          3d17h
kube-system   pod/traefik-865bd56545-995lq                  1/1     Running     0          3d17h

NAMESPACE         NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default           service/kubernetes                   ClusterIP      10.43.0.1       <none>           443/TCP                      3d17h
knative-serving   service/autoscaler-bucket-00-of-01   ClusterIP      10.43.128.142   <none>           8080/TCP                     91s
kube-system       service/kube-dns                     ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       3d17h
kube-system       service/metrics-server               ClusterIP      10.43.145.180   <none>           443/TCP                      3d17h
kube-system       service/traefik                      LoadBalancer   10.43.105.140   192.168.88.100   80:30837/TCP,443:31962/TCP   3d17h

NAMESPACE     NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik-33133399   1         1         1       1            1           <none>          3d17h

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                  1/1     1            1           3d17h
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           3d17h
kube-system   deployment.apps/metrics-server           1/1     1            1           3d17h
kube-system   deployment.apps/traefik                  1/1     1            1           3d17h

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-6d668d687                   1         1         1       3d17h
kube-system   replicaset.apps/local-path-provisioner-869c44bfbd   1         1         1       3d17h
kube-system   replicaset.apps/metrics-server-7bfffcd44            1         1         1       3d17h
kube-system   replicaset.apps/traefik-865bd56545                  1         1         1       3d17h

NAMESPACE     NAME                                 STATUS     COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik       Complete   1/1           33s        3d17h

Explanation of the installed items

Namespace

• kube-system (Kubernetes system components) - the system namespace responsible for the core functions of the cluster.
• default
• knative-serving

Pods

• coredns (cluster-internal DNS server) - translates Service names to IPs, resolves cluster-internal domains (e.g. my-service.default.svc.cluster.local → actual Service IP)
• helm-install-traefik & helm-install-traefik-crd - initial installation Jobs (one-time tasks)
  • traefik-crd: installs Traefik's Custom Resource Definition (CRD)
  • traefik: deploys the Traefik Ingress Controller with Helm
• local-path-provisioner (dynamic storage provisioner) - automatically creates a PV on a host path when a PersistentVolumeClaim (PVC) is created (default path: /var/lib/rancher/k3s/storage/)
• metrics-server (resource metrics collector) - collects and provides CPU/memory usage
  • Use: required by kubectl top nodes, kubectl top pods, HPA (Horizontal Pod Autoscaler), etc.
• svclb-traefik-xxxxx - k3s-specific Service Load Balancer (svclb) - L4 load balancer for LoadBalancer type Services
  • Feature: provided by default in k3s (LoadBalancer can be used without a cloud)
• traefik (Ingress Controller & Reverse Proxy) - routes HTTP/HTTPS traffic to services inside the cluster. Default ports are 80 (HTTP) and 443 (HTTPS).

Services

• kubernetes (default namespace) - Kubernetes API server endpoint
• autoscaler-bucket-00-of-01 (knative-serving) - service related to the Knative autoscaler
• kube-dns - Service endpoint for CoreDNS
• metrics-server - Metrics Server API endpoint
• traefik (LoadBalancer) - entry point for external traffic - EXTERNAL-IP: 192.168.88.100 (assigned by svclb)

DaemonSet

• svclb-traefik - deploys a LoadBalancer Pod on every node. When a node is added, it is automatically deployed to that node as well

Deployments

Each Deployment manages the Pods described above:

• coredns: DNS server
• local-path-provisioner: storage provisioner
• metrics-server: metrics collection
• traefik: Ingress Controller

ReplicaSets

The ReplicaSets created by each Deployment, which manage the number of Pod replicas.

• coredns-6d668d687 is the ReplicaSet managed by the coredns Deployment

Jobs

• helm-install-traefik-crd & helm-install-traefik - one-time tasks that install Traefik with Helm during cluster initialization

Accessing the cluster with kubectl from outside

• Cluster Access | K3s

Copy /etc/rancher/k3s/k3s.yaml to ~/.kube/config on a machine located outside the cluster.

Then replace the value of the server field with the IP or name of your K3s server. kubectl can now manage your K3s cluster.

scp user@remote-server:/etc/rancher/k3s/k3s.yaml ~/.kube/config
vim ~/.kube/config

Cluster access

See Kubectl#Cluster access.

Remote access to Kubernetes from a local environment

# 클러스터의 인증서 및 사용자 비밀번호 등 인증하는데 필요한 정보
cat /etc/rancher/k3s/k3s.yaml

# 해당 내용을 복사하여 로컬 ~/.kube/<클러스터 이름>로 붙여넣기
# 단, name 및 ip 주소를 해당 서버의 고정 ip로 변경할 것!

# 환경변수 설정
export KUBECONFIG=$HOME/.kube/config-k3s:$HOME/.kube/config:$KUBECONFIG

# 변경한 config 적용
kubectl config use-context <name>

Docker login

docker login <registry>

# 권한 토큰 정보 확인
cat ~/.docker/config.json

Creating a secret based on existing Docker credentials

kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=/home/ubuntu/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

Creating a pod that uses the secret

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred

NVIDIA container runtime support

• Advanced Options / Configuration | K3s
• (kubernetes) Using an Nvidia GPU in k3s
• NVIDIA device plugin for Kubernetes - required when adding a runtime class
• NVIDIA Container Toolkit

K3s will automatically detect and configure the NVIDIA container runtime if it is present when K3s starts.

First, install the NVIDIA Container Toolkit

• https://nvidia.github.io/libnvidia-container/
• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Install the prerequisites for the instructions below:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2

Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

sudo apt-get update

Install the NVIDIA Container Toolkit packages:

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.1-1
  sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

• Install K3s, or restart it if already installed: curl -ksL get.k3s.io | sh -
• Confirm that the NVIDIA container runtime has been found by k3s: grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

This will automatically add nvidia and/or nvidia-experimental runtimes to the containerd configuration, depending on what runtime executables are found.

Adding a RuntimeClass definition to the cluster

• Use the NVIDIA device plugin for Kubernetes.

You must add a RuntimeClass definition to your cluster, and deploy Pods that explicitly request the appropriate runtime by setting runtimeClassName: nvidia in the Pod spec:

# NVIDIA device plugin — 노드가 GPU 를 **스케줄 가능한 자원**으로 광고하게 한다.
# (plan 17 S9)
#
# 왜 필요한가: `RuntimeClass nvidia` 만으로도 파드에 GPU 를 밀어 넣을 수는 있지만,
# 그러면 클러스터가 GPU 점유를 모른다. GPU 가 하나뿐인 노드에서 서빙 엔드포인트를 둘
# 만들면 둘 다 같은 8 GB 를 잡고 서로 OOM 낸다. 이 플러그인이 `nvidia.com/gpu` 를
# 광고하면 두 번째 엔드포인트는 런타임에 죽는 대신 Pending 으로 대기한다 — 같은 실패라도
# 스케줄러가 아는 실패가 낫다.
#
# 원본: https://github.com/NVIDIA/k8s-device-plugin (v0.19.3, Apache-2.0)
# 로컬 수정: `runtimeClassName: nvidia` 한 줄. k3s 의 기본 런타임은 nvidia 가 아니라서
# 이게 없으면 플러그인 컨테이너가 NVML 을 못 열고 GPU 를 0개로 보고한다.
#
# 버전 하한: v0.18.2 ("Ignore errors getting device memory using NVML"). 통합 메모리
# GPU 는 `nvmlDeviceGetMemoryInfo` 에 `Not Supported` 를 돌려주는데, 그 이전 버전은
# 이것을 치명적 오류로 보고 디바이스 열거 자체를 포기한다 — GPU 를 정상 인식하고도
# `error building device map ... error getting device memory: Not Supported` 로 죽는다.
#
# 적용 (sudo 불필요 — kubeconfig 만 있으면 된다):
#     kubectl apply -f scripts/serving/nvidia-device-plugin.yml
#
# 확인 (노드가 여럿이면 `items[0]` 은 GPU 노드가 아닐 수 있다 — 전부 본다):
#     kubectl get nodes -o custom-columns='NAME:.metadata.name,ALLOC:.status.allocatable.nvidia\.com/gpu'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.19.3
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

Apply on the server:

kubectl apply -f ./nvidia-device-plugin.yml
kubectl -n kube-system rollout status daemonset nvidia-device-plugin-daemonset
kubectl -n kube-system logs -l name=nvidia-device-plugin-ds --tail=30

Check that it has been applied to the node:

## 숫자 1 이 출력되어야 한다.
kubectl get node -o jsonpath='{.items[0].status.allocatable.nvidia\.com/gpu}'

Note that the NVIDIA container runtime is also frequently used with the NVIDIA Device Plugin and GPU Feature Discovery, which must be installed separately, with modifications to ensure that pod specs include runtimeClassName: nvidia, as mentioned above.

Check that the GPU is recognized correctly in containerd.

sudo ctr image pull docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04
sudo ctr run --rm -t \
    --runc-binary=/usr/bin/nvidia-container-runtime \
    --env NVIDIA_VISIBLE_DEVICES=all \
    docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 \
    cuda-11.0.3-base-ubuntu20.04 nvidia-smi

For reference, ctr is the containerd CLI.

NVIDIA container runtime support where k3s is already installed

Just restart the service.

# k3s 재시작 / restart k3s
sudo systemctl restart k3s

# containerd 설정에 nvidia 런타임이 추가되었는지 확인
# check if nvidia runtime was auto-detected in containerd config
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

The output looks like this:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

k3s only adds the containerd configuration and does not create the RuntimeClass resource, so create it yourself:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Since runtimeClassName: nvidia must be specified in the Pod, specify it for the device plugin DaemonSet as well when installing:

# Helm으로 device plugin 설치 (runtimeClassName 지정)
# install device plugin via Helm with runtimeClassName
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --set runtimeClassName=nvidia

After that, specify runtimeClassName: nvidia + resources.limits."nvidia.com/gpu": 1 in any Pod that uses the GPU.

Running an HTTPS web service with k3s

• Building single-node Kubernetes with k3s - build single-node Kubernetes with k3s, then open ports and deploy with Traefik
• k3s series - Easily running an HTTPS web service on Kubernetes | si.mpli.st
• HTTPS using Letsencrypt and Traefik with k3s

You need to know the following:

• Kustomize
• cert-manager
• traefik

Tutorial

Follow along step by step with Kubernetes:Basic.

Single Node

• K3s single-node cluster for noobs | Logan Marchione
• Building Single Node Cluster Kubernetes (Installing K3S Server)

Swap memory support

• 1.28 Enable swap support · Issue #8811 · k3s-io/k3s

K3s has always supported swap. But it seems to have been a problem in kubelet for someone?

curl -sL get.k3s.io | sh -s - server --kubelet-arg=feature-gates=NodeSwap=true,CloudDualStackNodeIPs=true

Troubleshooting

tls: failed to verify certificate

See Kubectl#tls: failed to verify certificate.

In k3s, add the domain to ExeStart in the server's service file (/etc/systemd/system/k3s.service), like --tls-san api.your.run:

ExecStart=/usr/local/bin/k3s \
    server \
    --tls-san api.your.run \
    --write-kubeconfig-mode 644

For reference, the --tls-san in the flag name is an abbreviation of the following. (It's not "scan" (SCAN), not scan)

• TLS - Transport Layer Security
• SAN - Subject Alternative Name

Agent node installation failure

• Unable to setup cluster with 1 server and 1 agent · Issue #4839 · k3s-io/k3s

If you see an error like the following when checking with journalctl -u k3s:

Dec 25 18:11:38 ocvm-a1-1 k3s[8221]: time="2021-12-25T18:11:38Z" level=error msg="unable to verify hash for node 'ocvm-a1-2': hash does not match"

Try removing the password used for agent registration:

kubectl get secret -n kube-system
kubectl delete secret -n kube-system <host>.node-password.k3s

See #How agent node registration works for details.

Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get https://127.0.0.1:6444/cacerts

• Configuring a K3s cluster on an Ubuntu 20.04 server – nc2u blog

When k3s agent registration failed, the following error was found with journalctl -u k3s:

May 23 16:17:51 server k3s[2139]: time="2024-05-23T16:17:51+09:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
...

Ran sudo netstat -l -n -p -t | grep 6444 to see if the server is alive:

tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      2139/k3s agent

It's alive... Checked the HTTP connection with curl -v -khttps://127.0.0.1:6444/:

*   Trying 127.0.0.1:6444...
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

See OpenSSL#error:0A000126:SSL routines::unexpected eof while reading.

Is Ubuntu 22.04 the problem? ... Let's downgrade to Ubuntu 20.04 and check.

See also

• Cluster manager
• Kubernetes
• minikube

Favorite site

• K3s: Lightweight Kubernetes

Beginner's guide

• (Kubernetes) Building a multi-node Kubernetes cluster with k3s
• [Recommended] k3s series - Easily setting up a Kubernetes environment | si.mpli.st
• Building single-node Kubernetes with k3s
• k3s series - Easily running an HTTPS web service on Kubernetes | si.mpli.st

Guide

• Local Kubernetes Cluster - Byungchan's Blog - Building a Kubernetes Cluster with Multipass and K3S
• (Kubernetes) Building a single-node Kubernetes cluster with k3s