MLflow:Serving

lastmod 2026-07-16created 2026-07-102 edits

MLflow built-in serving

• Serving Machine Learning Model. With MLFlow and Docker | by Xin Cheng | Medium
• MLFlow Serving
• MLflow Serving

The simplest and fastest method, provided by MLflow itself. Useful for quickly setting up a serving environment locally or in Docker.

• How it works: when you run a command of the form mlflow models serve -m runs:/<run_id>/model -p 5000, MLflow automatically sets up the model's dependencies (Conda/Pip environment) and launches a Flask/FastAPI-based REST API server.
• Pros: requires almost no code and integrates directly with the MLflow Tracking Server and Model Registry.
• Cons: high availability (HA), autoscaling, complex routing, etc. must be implemented yourself, so it is too heavy to use on its own in large-scale production environments.
• Recommended for: PoC (proof of concept), internal demos, or low-traffic internal services.

MLflow built-in Docker build + container deployment (most recommended)

Packages MLflow's built-in serving feature as a Docker image and runs it on a lightweight container orchestration system you already have.

• How it works: the mlflow models build-docker command makes MLflow build a self-contained Docker image that includes the model and its runtime environment. Deploy this image to AWS ECS, Google Cloud Run, or Docker Compose on a single lightweight VM.
• Pros: no need for complex K8s infrastructure (Istio, Cert-Manager, etc.) as with KServe, and you can make direct use of the autoscaling and load balancing of the lightweight container services offered by cloud providers.
• Recommended for: when you want to reduce the burden of managing K8s while still ensuring stability in production.

How input data is passed

Key summary: the MLflow server (mlflow models serve) does not distinguish between images, text, and audio. The client must always convert data into a JSON-serializable form (numeric arrays or base64 strings) before sending it, and interpreting "this is an image" depends entirely on the logic inside pyfunc.PythonModel.predict().

1. The Model Signature defines the input format

When an MLflow model is saved, its signature (input/output schema) is saved with it. This signature determines the JSON schema of /invocations requests.

import mlflow
from mlflow.models.signature import infer_signature

# 학습 시 signature 추론 및 저장 / infer and log signature at training time
signature = infer_signature(train_x, model.predict(train_x))
mlflow.pyfunc.log_model(
    artifact_path="model",
    python_model=my_model,
    signature=signature
)

2. Request JSON formats (3 common types regardless of model type)

Format 1: dataframe_split (column-based, common for tabular models)

{
  "dataframe_split": {
    "columns": ["feature1", "feature2"],
    "data": [[1.0, 2.0]]
  }
}

Format 2: instances (tensor-based, common for images, etc.)

{
  "instances": [[[0.1, 0.2, "..."], "..."]]
}

Format 3: inputs (simple array)

{
  "inputs": [[1.0, 2.0, 3.0]]
}

3. How each modality is actually handled

Images

(a) Convert to a numpy array on the client before sending

import numpy as np
from PIL import Image

img = np.array(Image.open("photo.jpg").resize((224, 224))) / 255.0
payload = {"inputs": img.tolist()}  # shape: (224, 224, 3) -> JSON list

(b) Handle base64 decoding inside the model's pyfunc wrapper (recommended approach)

import base64
from io import BytesIO
from PIL import Image

class ImageModel(mlflow.pyfunc.PythonModel):
    def predict(self, context, model_input):
        # model_input은 base64 문자열 컬럼을 가진 DataFrame
        images = []
        for b64_str in model_input["image_base64"]:
            img_bytes = base64.b64decode(b64_str)
            img = Image.open(BytesIO(img_bytes))
            images.append(preprocess(img))
        return self.model.predict(images)

Request example:

import base64
import requests

with open("photo.jpg", "rb") as f:
    b64_img = base64.b64encode(f.read()).decode("utf-8")

requests.post(
    "http://localhost:5000/invocations",
    json={"dataframe_split": {"columns": ["image_base64"], "data": [[b64_img]]}}
)

Text

The simplest case. Pass an array of strings as is, and handle tokenizing inside the pyfunc wrapper.

{
  "dataframe_split": {
    "columns": ["text"],
    "data": [["안녕하세요, 이 문장을 분류해주세요"]]
  }
}

Audio

As with images, base64 encoding is common, and decoding is done inside the wrapper with librosa or similar.

{
  "dataframe_split": {
    "columns": ["audio_base64", "sample_rate"],
    "data": [["<base64 encoded wav bytes>", 16000]]
  }
}

4. Key takeaways

5. How to check the Signature

# 배포된 모델의 signature 확인 / check signature of deployed model
mlflow models predict -m "models:/my_model/1" --env-manager conda --input-path input.json

Alternatively, you can check it by looking up model.metadata.signature in the MLflow Model Registry UI/API.

6. Caveats: Point Cloud / 3D data

For models that take large 3D arrays such as point clouds as input (e.g. welding QC PTv3/Pointcept), this approach can be tricky. Serializing 3D arrays to JSON makes the payload larger and slower, so in such cases it is better to consider gRPC or a separate binary protocol.

See also

• MLflow
• MLOps

Favorite site

• ML Model Serving | MLflow AI Platform
• Command-Line Interface # build-docker