Skip to content
Search

MLOps

lastmod 2026-10-27created 2022-08-1744 edits

MLOps or ML Ops is a paradigm that aims to deploy and maintain machine learning models in production reliably and efficiently. The word is a compound of "Korean: 머신 러닝" and the continuous development practice of DevOps in the software field.

Categories

Terms

Testing
In addition to the usual unit and integration tests, data validation, quality evaluation of the trained model, and model validation are also needed.
Deployment
It is not enough to deploy an ML model trained offline; the process of retraining and validating new models must be automated.
Production
Unlike software systems, which can generally reach optimal performance through optimization of algorithms and logic, ML models can additionally degrade from the continuously evolving data profile alone.
In other words, performance can be compromised for more diverse reasons than in conventional software systems, so you must continuously track the data's summary statistics and monitor the model's online performance, sending alerts or rolling back when values deviate from expectations.
CI (Continuous Integration)
CI must test and validate not only code and components but also data, data schemas, and models.
CD (Continuous Delivery)
You must deploy the entire ML training pipeline, not a single software package.
CT (Continuous Training)
A property unique to ML systems; the stage of automatically training and evaluating models.

Pipeline · Process

MLOps (Machine Learning Operations)
ML engineering framework that automates the data→training→evaluation→deployment→monitoring→retraining cycle from a CI/CD perspective
CI (Continuous Integration)
Pipeline stage that automatically runs training, validation, and model registration when code or data changes
CD (Continuous Deployment)
Stage that automatically rolls out a model approved for promotion through image build→container deployment→health check
CT (Continuous Training)
Loop that automatically retrains and redeploys on the latest data when performance degradation or data drift is detected in operation
KPI (Key Performance Indicator)
Key performance indicators that quantitatively evaluate MLOps quality, such as lead time, automation rate, response time, and availability
Human-in-the-loop
Approach that requires operator approval at key decision points in the automated flow, such as Production promotion
End-to-End
Fully automated flow connected from data collection to serving and monitoring without manual work

Model · Registry

Run
A single training attempt; hyperparameters, metrics, artifacts, and the source commit are recorded together
Model Registry
Central repository that manages the versions, stages, metadata, and artifacts of registered models in one place (e.g., MLflow)
SemVer (Semantic Versioning)
Scheme that assigns versions by the rule breaking change (MAJOR).feature addition (MINOR).bug fix (PATCH) (e.g., 1.2.3)
Dev (Development)
Development state just registered in the registry, before Staging (corresponds to None in MLflow)
Staging
Stage that validates performance, compatibility, and failure scenarios before Production promotion
Production
Active version put into field service, receiving real-time inference traffic
Archived
Past version no longer served but kept for history tracking and reproduction

Performance metrics

Acc (Accuracy)
Ratio of correct answers among all predictions (TP+TN)/N; can be distorted on class-imbalanced data
F1 Score
Harmonic mean of precision and recall 2PR/(P+R); evaluates classification performance on imbalanced data in a balanced way
Loss
Value quantifying the difference between prediction and ground truth; training proceeds in the direction of minimizing it
RPS (Requests Per Second)
Number of requests the serving API handles per second; indicates processing capacity and load level
P95 (95th Percentile Latency)
The 95th percentile of the response time distribution; evaluates tail latency excluding the top 5% of outliers

Drift · Quality

Data Drift
Phenomenon where the distribution of input features in operation shifts relative to training time (e.g., seasonal change, sensor replacement)
Concept Drift
Phenomenon where the ground-truth label itself changes for the same input, degrading model performance (e.g., change in judgment criteria)
PSI (Population Stability Index)
Index quantifying the difference between two distributions; below 0.1 is judged stable, 0.1\~0.25 caution, and ≥0.25 a significant change
Ground Truth
Correct-answer data from actual results obtained through on-site inspection or re-measurement, serving as the reference for evaluating prediction performance

Deployment strategies

Rolling (Rolling Update)
Deploys without service interruption by sequentially replacing old-version instances with the new version a fixed fraction at a time
Canary (Canary Deployment)
Expands gradually by raising the new version's traffic share in steps, like 5%→25%→100%, while watching metrics
Side-by-side comparison
Runs inference on Staging and Production with the same data to directly compare accuracy, response time, and resources
Rollback
Action that immediately returns traffic to the previous stable Production version on health check failure or an alert trigger

Resources · Alerts

CPU (Central Processing Unit)
General-purpose compute resource used for general preprocessing and lightweight model inference (allocated in vCPUs)
MEM (Memory)
Main memory resource needed for loading model weights and batch processing (allocated in GiB)
GPU (Graphics Processing Unit)
Parallel compute device for accelerating deep learning training and large-scale inference (e.g., NVIDIA A100, T4)
Critical
Severe incident at the service-outage/company-wide-impact level, requiring immediate on-call paging and recovery action
Warning
Caution level, such as nearing a threshold or a temporary spike in errors, requiring preventive action and root-cause investigation
Info
Informational alert for logging, such as deployment completion or configuration changes, requiring no immediate action

Data science steps for ML

From first defining the business use case and success criteria to deploying the ML model to production, every ML project, whether manual or automated, involves the following steps.

  1. Data Extraction
    • Extract relevant data from data sources
  2. Data Analysis
    • Perform exploratory data analysis (EDA) to understand the data
    • Understand the data schema and characteristics the model needs
  3. Data Preparation
    • Split the data into training, validation, and test sets
  4. Model Training
    • Implement various algorithms, tune and apply hyperparameters
    • The output is a trained model.
  5. Model Evaluation
    • Evaluate the model on a holdout test set
    • The output is the model's performance evaluation metrics.
  6. Model Validation
    • Validate that model performance exceeds the baseline and is suitable for deployment
  7. Model Serving
    • Microservice with a REST API to provide online predictions
    • Batch prediction system
    • Embedded model in mobile services
  8. Model Monitoring
    • Monitor the model's prediction performance

Data extraction and preprocessing

Labeling

Experiment tracking / Experiment Mamanagement

Machine learning projects require running many experiments with various models, parameters, and training data. Experiment tracking tools record all the information needed for each experiment (weights and hyperparameters, code, training results, etc.). This lets you track the versions and results of experiment components and compare multiple experiments.

  • Kedro - Open-source Python framework for creating reproducible, maintainable, modular data science code.
  • ModelDB - Open-source system for machine learning model versioning, metadata, and experiment management.
  • MLflow - Open-source platform for managing the end-to-end machine learning lifecycle.
  • Determined - Open-source deep learning training platform that makes building models fast and easy.
  • Weights & Biases - Tool for tracking and visualizing every part of the machine learning pipeline, from datasets to production models.
  • Polyaxon - Platform for building, training, and monitoring large-scale deep learning applications.
  • ClearML - End-to-end platform with an auto-magical suite that streamlines ML workflows.
  • TensorBoard - Visualization toolkit for optimizing and debugging ML models.
  • Losswise - Saves results by creating a git commit every epoch. Also provides visualization.
  • Comet.ml
  • Aim - An easy-to-use & supercharged open-source experiment tracker.

INFORMATION

TensorBoard is one of the most widely used tools for analyzing ML experiment results. However, most of TensorBoard's features are limited to a single experiment, so tracking multiple experiments requires combining it with other tools.

Experimentation

Experimentation is the process of analyzing data, building prototype models, and implementing training. Most of these tools provide the Jupyter Notebook environment often used in data science.

  • Kubeflow - Makes ML workflows on Kubernetes simple, portable, and scalable.
  • Jupyter - Open-source web application for creating and sharing documents containing live code, equations, visualizations, and text.
  • Polynote - Data science notebook with first-class Scala support.
  • Spyder - Free, open-source scientific environment for Python, written in Python.
  • Apache Zepellin - Web-based notebook supporting data-driven interactive data analysis and collaborative documents using SQL, Scala, Python, R, and more.

Data versioning

Data versioning tools let you manage different versions of datasets in a well-organized way. This gives ML teams insights such as identifying how data changes affect model performance and understanding how datasets evolve. Data versioning supports reproducing and tracking ML model lineage.

  • Dolt - Git for data. You can run SQL queries or manipulate data through a command-line interface.
  • DVC (Data Version Control) - Open-source version control system for machine learning project reproducibility.
  • Weights & Biases (wandb) - Tool for tracking and visualizing every part of the machine learning pipeline, from datasets to production models.
  • Git LFS - Open-source tool for versioning large files with Git.
  • Pachyderm - Tool for data versioning and pipelines for MLOps.
  • lakeFS - Git for object storage buckets, working at any data size.

Hyperparameter tuning

How do you tune hyperparameters? Deep learning is literally full of hyperparameters. Finding the optimal setting in a multidimensional space made of countless hyperparameters is not easy. Hyperparameter search is an iterative task constrained by computing power, money, and time. That is why using software that finds hyperparameters helps a lot.

  • Hyperopt - Python library that makes finding hyperparameters easy
  • SigOpt - Optiomization-as-a-Service API
  • Ray Tune
  • Weights & Biases
  • Katib - Runs as Kubeflow Components and provides Hyperparameter tunning

Machine learning pipeline (Pipeline orchestration)

A machine learning pipeline (ML Pipeline) structures and automates complex ML training and inference tasks step by step in production environments. When data/concept drift occurs, it automatically triggers the pipeline to retrain the model on the latest data.

  • Apache Airflow - Platform created by the Apache community to programmatically author, schedule, and monitor workflows.
  • Argo Workflows - Open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
  • Luigi - Python package that helps build complex pipelines of batch jobs.
  • Kubeflow - Makes ML workflows on Kubernetes simple, portable, and scalable.
  • Kedro - Open-source Python framework for creating reproducible, maintainable, modular data science code.
  • Nextflow - Enables scalable and reproducible scientific workflows using software containers.
  • Dagster - Orchestration platform for developing, producing, and observing data assets.
  • Apache Beam - Batch and streaming data processing tool for mission-critical production workloads.
  • ZenML - Extensible open-source MLOps framework for creating production-ready machine learning pipelines.
  • Flyte - Workflow automation platform for mission-critical data and large-scale machine learning processes.
  • Prefect - Open-source workflow management system designed for modern infrastructure.
  • Ray - Open-source tool that makes it easy to scale compute-intensive Python workloads.
  • DVC (Data Version Control) - is an open-source version control system for machine learning project reproducibility.
  • Polyaxon - Platform for building, training, and monitoring large-scale deep learning applications.
  • ClearML - End-to-end platform with an auto-magical suite that streamlines ML workflows.
  • Pachyderm - Tool for data versioning and pipelines for MLOps.
  • Orchest - Data pipeline creation tool that combines Jupyter notebooks and Python scripts with a visual pipeline editor.

Runtime engine

A runtime engine provides the environment where ML code runs. It is installed on a Kubernetes cluster to provide an API for parallel execution in multi-node environments, or provides its own cluster as the runtime environment.

  • Ray - Ray is an open-source tool that makes it easy to scale compute-intensive Python workloads.
  • Nuclio - Nuclio is a high-performance serverless framework focused on data-, I/O-, and compute-intensive workloads.
  • Dask - Dask is a flexible library for parallel computing in Python.
  • Horovod - Horovod makes it easy to train models in parallel on multiple GPUs.
  • Apache Spark - Apache Spark is a data processing framework for large datasets and distributed computing.

Artifact tracking

Stores and tracks datasets, models, and intermediate outputs across experiments and ML pipelines.

  • Kubeflow - Kubeflow makes ML workflows on Kubernetes simple, portable, and scalable.
  • MLflow - MLflow is an open-source platform for managing the end-to-end machine learning lifecycle.
  • Polyaxon - Polyaxon is a platform for building, training, and monitoring large-scale deep learning applications.
  • ClearML - ClearML is an end-to-end platform with an auto-magical suite that streamlines ML workflows.

INFORMATION

The outputs of each step of an ML pipeline or of ML experiment runs are called artifacts. These include trained model checkpoint files, CSV/Parquet/JSON files of transformed data, run log files, and so on.

INFORMATION

There are no dedicated artifact tracking tools. Artifact tracking is included as a feature of ML pipeline tools or ML experiment tracking tools.

Model registry

A model registry is a centralized model repository. It provides versioning and tracking of trained models and stores information about model input/output interfaces and runtime packages for deployment.

  • ModelDB - Open-source system for machine learning model versioning, metadata, and experiment management.
  • MLflow - Open-source platform for managing the end-to-end machine learning lifecycle.
  • Determined - Open-source deep learning training platform that makes building models fast and easy.
  • Weights & Biases - Tool for tracking and visualizing every part of the machine learning pipeline, from datasets to production models.
  • ClearML - End-to-end platform with an auto-magical suite that streamlines ML workflows.

Model serving

Creates a REST API or gRPC endpoint and responds with inference results when it receives requests. These are tools used when the model's deployment target is the cloud.

  • Seldon Core - Open-source platform for deploying machine learning models at scale on Kubernetes.
  • BentoML - Open platform that simplifies ML model deployment and lets you serve models at production scale in minutes.
  • NVIDIA Triton (NVIDIA Triton Inference Server) - Open-source inference serving software that standardizes model deployment and execution and delivers fast, scalable AI in production.
  • TensorFlow Serving - Flexible, high-performance serving system for machine learning models, designed for production environments.
  • KServe - Highly scalable, standards-based model inference platform on Kubernetes.
  • FastAPI - Modern, high-performance web framework for building APIs with Python.
  • TorchServe - Flexible, easy-to-use tool for serving and scaling PyTorch models in production.
  • Ray - Open-source tool that makes it easy to scale compute-intensive Python workloads.
  • Cog - Open-source tool that lets you package machine learning models into standard, production-ready containers.
  • ModelDB - Open-source system for machine learning model versioning, metadata, and experiment management.
  • MLflow - Open-source platform for managing the end-to-end machine learning lifecycle.
  • ClearML - End-to-end platform with an auto-magical suite that streamlines ML workflows.
  • Nuclio - High-performance serverless framework focused on data-, I/O-, and compute-intensive workloads.

Model monitoring

Model monitoring tools detect performance degradation, bias, and data drift in deployed models.

  • Prometheus - Open-source system monitoring and alerting toolkit.
  • Grafana - Open-source tool for online system analytics and monitoring.
  • Evidently - Tool for analyzing and tracking data and ML model quality throughout the model lifecycle.
  • Alibi Detect - Open-source Python library focused on outlier, adversarial, and drift detection.
  • ModelDB - Open-source system for machine learning model versioning, metadata, and experiment management.
  • ClearML - End-to-end platform with an auto-magical suite that streamlines ML workflows.

Machine learning/deep learning frameworks (ML/DL framework)

Frameworks that make developing complex machine learning/deep learning models easy and efficient.

  • Tensorflow - End-to-end open-source platform for machine learning.
  • PyTorch - Optimized tensor library for deep learning using GPUs and CPUs.
  • Keras - Deep learning API written in Python, running on the machine learning platform TensorFlow.
  • scikit-learn - Python module for machine learning built on top of SciPy.
  • JAX - NumPy on CPU, GPU, and TPU, with automatic differentiation for high-performance machine learning research.

Korean: 피처 스토어 (Feature store)

A feature store is a central repository for features. A feature store transforms raw data into features, stores them, and provides them for model training and inference.

  • Feast - Feast is an open-source feature store. It is the fastest way to operationalize analytic data for model training and online inference.
  • Hopsworks - Hopsworks is a data-intensive open-source AI platform used to develop and operate machine learning models at scale.
  • Tecton - Tecton is a fully managed feature platform built to orchestrate the full lifecycle of features, from transformation to online serving, with enterprise-grade SLAs.
  • Rasgo - Rasgo is a web app for exploring, transforming, and visualizing data within a cloud data warehouse.

INFORMATION

If you need feature engineering, consider adopting a feature store.

Data validation

Outlier data and invalid data harm ML models. These are tools that help with data validation.

  • TFDV (TensorFlow Data Validation) - Identifies anomalies in training and serving data and can inspect data to generate a schema automatically.
  • Deequ - Library built on Apache Spark for defining "unit tests for data" that measure data quality in large datasets.
  • DVT (Data Validation Tool) - Open-source Python CLI tool based on the Ibis framework that compares heterogeneous data source tables with multi-level validation functions.

Korean: 설명 가능한 AI (XAI, eXplainable AI)

XAI is a set of tools and techniques that help people easily understand why an AI model made a particular decision and how it works.

INFORMATION

XAI is being actively researched, and there are many articles on why it is needed, but there do not seem to be many real-world applications yet.

All-in-One Solutions

All-in-one solutions provide everything covered so far.

See also

Favorite site

Stack & Tools compare

Tutorials

author Zer0 sitemap