Skip to content
Search

Kubernetes

lastmod 2026-08-23created 2015-12-19127 edits

The de facto standard of Container Orchestration.

Docker plays a pivotal role in a collection of powerful, innovative tools and technologies for bringing containerized software development to the open source community. Kubernetes, which Google unveiled last June, is an open-source container management tool used to accelerate development and simplify operations.

Google has long used containers in its internal operations. At DockerCon in the summer of 2014, Google open-sourced Kubernetes, which was developed to meet the needs of the explosively growing Docker ecosystem. Through collaboration with other organizations and projects, including Red Hat and CoreOS, the Kubernetes project maintainers have grown Kubernetes into the No. 1 downloaded project on Docker Hub. The Kubernetes team plans to expand the project and grow the community to help software developers save the time they spend managing infrastructure and use it to build apps.

Categories

Core Components

A Kubernetes cluster consists of a control plane and one or more worker nodes. Here's a brief overview of the main components:

Control Plane Components

Manage the overall state of the cluster:

  • kube-apiserver - The core component server that exposes the Kubernetes HTTP API.
  • etcd - Consistent and highly-available key value store for all API server data.
  • kube-scheduler - Looks for Pods not yet bound to a node, and assigns each Pod to a suitable node.
  • kube-controller-manager - Runs controllers to implement Kubernetes API behavior.
  • cloud-controller-manager (optional) - Integrates with underlying cloud provider(s).

Node Components

Run on every node, maintaining running pods and providing the Kubernetes runtime environment:

  • kubelet - Ensures that Pods are running, including their containers.
  • kube-proxy (optional) - Maintains network rules on nodes to implement Services.
  • Container runtime - Software responsible for running containers. Read Container Runtimes to learn more.

Lightweight

Cloud Service

Container Network Interface (CNI)

  • Ingress
  • Flannel
  • ACI
  • Calico
  • Canal
  • Cilium
  • CNI-Genie
  • Contiv
  • Multus
  • NSX-T
  • Nuage
  • Romana
  • Weave Net

Libraries

Block Storage

Object Storage

Configuration

certificate

Secret

Sidecar

Dashboard / Management

  • Kubetail - Real-time Kubernetes logging dashboard
  • KuView - Web-based real-time Kubernetes dashboard
  • KDash - Fast and simple Kubernetes dashboard
  • Kubeshark - API traffic viewer for Kubernetes
  • Headlamp - Easy-to-use, extensible Kubernetes web UI

Services

ETC

Why do we need Kubernetes?

In a production environment, you need to manage the containers that run your applications and make sure there is no downtime. For example, if a container goes down, another container needs to be started to minimize downtime.

What if this behavior could be handled by the system?

Having these problems managed by the system is the role of Kubernetes!

Features

According to the official Kubernetes website, it provides the following features.

  • Service discovery and load balancing - Expose containers using a DNS name or their own IP address
  • Storage orchestration - Automatically mount a storage system of your choice, such as local storage or public cloud providers
  • Automated rollouts and rollbacks - Describe the desired state, and change the current state to the desired state at a configured rate
  • Automatic bin packing - Provides the CPU and memory (RAM) each container needs
  • Self-healing - Restarts failed containers and replaces containers
  • Secret and configuration management - Store and manage sensitive information such as passwords, OAuth tokens, and SSH keys

Patterns

  1. Foundational patterns
    • Health probe pattern
    • Predictable demands pattern
    • Automated placement pattern
  2. Structural patterns
    • Init container pattern
    • Sidecar pattern
  3. Behavioral patterns
    • Batch job pattern
    • Stateful service pattern
    • Service discovery pattern
  4. Advanced patterns
    • Controller pattern
    • Operator pattern

Cordon vs Drain

Cordon
Excludes a specific node from the scheduler so that no pods are assigned to it. Pods already deployed on the node stay where they are.
Drain
Excludes a specific node from the scheduler so that no pods are assigned to it, and moves the pods already deployed there to other nodes. Useful when updating a node.

HTTP connection

See Kube-apiserver#HTTPS Endpoint.

Checklis for using Kubernetes in production

Running applications in production can be tricky. This post proposes an opinionated checklist for going to production with a web service (i.e. application exposing HTTP API) on Kubernetes.

This checklist was shaped by my own limited experience in Zalando, so please see it only as template for your own checklist adopted to your environment. Some items can be considered optional or aspirational, depending on context.

General

  • Application's name, description, purpose, and owning team is clearly documented (e.g. in a central application registry or wiki)
  • Application's criticality level was defined (e.g. "tier 1" if the app is highly critical for the business)
  • Development team has sufficient knowledge/experience with the technology stack
  • Responsible 24/7 on-call team is identified and informed
  • Go-Live plan exists incl. steps for potential rollback

Application

  • Application's code repository (git) has clear instructions on how to develop, how to configure, and how to contribute changes (important for emergency fixes)
  • Code dependencies are pinned (i.e. hotfix changes do not accidentally pull in new libraries)
  • All relevant code is instrumented with OpenTracing or OpenTelemetry
  • OpenTracing/OpenTelemetry semantic conventions are followed (incl. additional company conventions)
  • All outgoing HTTP calls have a defined timeout
  • HTTP connection pools are configured with sane values according to expected traffic
  • Thread pools and/or non-blocking async code is correctly implemented/configured
  • Database connection pools are sized correctly
  • Retries and retry policies (e.g. backoff with jitter) are implemented for dependent services
  • Circuit breakers are implemented
  • Fallbacks for circuit breakers are defined according to business requirements
  • Load shedding / rate limiting mechanisms are implemented (could be part of provided infrastructure)
  • Application metrics are exposed for collection (e.g. to be scraped by Prometheus)
  • Application logs go to stdout/stderr
  • Application logs follow good practices (e.g. structured logging, meaningful messages), log levels are clearly defined, and debug logging is disabled for production by default (with option to turn on)
  • Application container crashes on fatal errors (i.e. it does not enter some unrecoverable state or deadlock)
  • Application design/code was reviewed by a senior/principal engineer

Security & Compliance

  • Application can run as unprivileged user (non-root)
  • Application does not require a writable container filesystem (i.e. can be mounted read-only)
  • HTTP requests are authenticated and authorized (e.g. using OAuth)
  • Mechanisms to mitigate Denial Of Service (DOS) attacks are in place (e.g. ingress rate limiting, WAF)
  • A security audit was conducted
  • Automated vulnerability checks for code / dependencies are in place
  • Processed data is understood, classified (e.g. PII), and documented
  • Threat model was created and risks are documented
  • Other applicable organizational rules and compliance standards are followed

CI/CD

  • Automated code linting is run on every change
  • Automated tests are part of the delivery pipeline
  • No manual operations are needed for production deployments
  • All relevant team members can deploy and rollback
  • Production deployments have smoke tests and optionally automatic rollbacks
  • Lead time from code commit to production is fast (e.g. 15 minutes or less including test runs)

Kubernetes

  • Development team is trained in Kubernetes topics and knows relevant concepts
  • Kubernetes manifests use the latest API version (e.g. apps/v1 for Deployment)
  • Container runs as non-root and uses a read-only filesystem
  • A proper Readiness Probe was defined, see blog post about Readiness/Liveness Probes
  • No Liveness Probe is used, or there is a clear rationale to use a Liveness Probe, see blog post about Readiness/Liveness Probes
  • Kubernetes deployment has at least two replicas
  • A Pod Disruption Budget was defined (or is automatically created, e.g. by pdb-controller)
  • Horizontal autoscaling (HPA) is configured if adequate
  • Memory and CPU requests are set according to performance/load tests
  • Memory limit equals memory requests (to avoid memory overcommit)
  • CPU limits are not set or impact of CPU throttling is well understood
  • Application is correctly configured for the container environment (e.g. JVM heap, single-threaded runtimes, runtimes not container-aware)
  • Single application process runs per container
  • Application can handle graceful shutdown and rolling updates without disruptions, see this blog post
  • Pod Lifecycle Hook (e.g. "sleep 20" in preStop) is used if the application does not handle graceful termination
  • All required Pod labels are set (e.g. Zalando uses "application", "component", "environment")
  • Application is set up for high availability: pods are spread across failure domains (AZs, default behavior for cross-AZ clusters) and/or application is deployed to multiple clusters
  • Kubernetes Service uses the right label selector for pods (e.g. not only matches the "application" label, but also "component" and "environment" for future extensibility)
  • There are no anti-affinity rules defined, unless really required (pods are spread across failure domains by default)
  • Optional: Tolerations are used as needed (e.g. to bind pods to a specific node pool)

See also this curated checklist of Kubernetes production best practices.

Monitoring

  • Metrics for The Four Golden Signals are collected
  • Application metrics are collected (e.g. via Prometheus scraping)
  • Backing data store (e.g. PostgreSQL database) is monitored
  • SLOs are defined
  • Monitoring dashboards (e.g. Grafana) exist (could be automatically set up)
  • Alerting rules are defined based on impact, not potential causes

Testing

  • Breaking points were tested (system/chaos test)
  • Load test was performed which reflects the expected traffic pattern
  • Backup and restore of the data store (e.g. PostgreSQL database) was tested

24/7 On-Call

  • All relevant 24/7 personnel is informed about the go-live (e.g. other teams, SREs, or other roles like incident commanders)
  • 24/7 on-call team has sufficient knowledge about the application and business context
  • 24/7 on-call team has necessary production access (e.g. kubectl, kube-web-view, application logs)
  • 24/7 on-call team has expertise to troubleshoot production issues with the tech stack (e.g. JVM)
  • 24/7 on-call team is trained and confident to perform standard operations (scale up, rollback, ..)
  • Runbooks are defined for application-specific incident handling
  • Runbooks for overload scenarios have pre-approved business decisions (e.g. what customer feature to disable to reduce load)
  • Monitoring alerts to page the 24/7 on-call team are set up
  • Automatic escalation rules are in place (e.g. page next level after 10 minutes without acknowledgement)
  • Process for conducting postmortems and disseminating incident learnings exists
  • Regular application/operational reviews are conducted (e.g. looking at SLO breaches)

Documentation

Kubernetes Hardening Guidance from the US National Security Agency (NSA)
https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF
CTR_KUBERNETES_HARDENING_GUIDANCE.PDF
Key points
  • Scan containers and pods for vulnerabilities or misconfigurations
  • Run containers and pods with the least privilege possible
  • Use network separation to control the damage that can occur
  • Use firewalls to limit unneeded network connectivity, and use encryption
  • Use strong authentication and authorization to limit user and administrator access and restrict the attack surface.
  • Use Log Auditing so administrators can monitor activity and be alerted to potentially malicious activity
  • Periodically review all k8s settings and use Vulnerability scans to make sure risks are mitigated and security patches are applied
[Translation] 10 easy mistakes to make in Kubernetes
https://coffeewhale.com/kubernetes/mistake/2020/11/29/mistake-10/

Building a web server with k8s

iptables in Kubernetes

See Iptables#iptables in Kubernetes for details.

Server Side Apply 

kubectl apply --server-side=true -f ...

List of Container Orchestration

See also

  • Cluster manager
  • Docker
  • Meshery - Open-source cloud native manager
  • clopus-watcher - A Claude Code-style autonomous 24/7 on-call engineer - An AI-based monitoring tool aimed at automating incident response in Kubernetes environments

Favorite site

Documentation

How to install

See Kubernetes:Install#Favorite site for details.

Guide

Tutorials

Article

Videos

Online Tools

  • kyaml2go - Kubernetes client-go code generator for resource YAMLs
  • KUBERNETES INSTANCE CALCULATOR - A calculator that helps you easily find suitable instances for your Kubernetes cluster.
  • Kubernetes Spec Explorer - Reference Guide and Documentation
    • Easily browse the documentation for all Kubernetes resources/properties/types/examples
    • Tree view with the schema, types, and descriptions of all resources
    • Shows the change history (properties added/removed/modified) since version X
    • Supports all versions since 1.25, including the newly released 1.32

References

  1. Getting_Started_with_Kubernetes_-_What_is_Kubernetes.pdf ↩

author Zer0 sitemap