Docker Compose

lastmod 2025-02-14created 2016-05-3122 edits

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services. Then, with a single command, you create and start all the services from your configuration.

How to run

Using Compose is basically a three-step process:

Define your app’s environment with a Dockerfile so it can be reproduced anywhere.
Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
Run docker compose up and the Docker compose command starts and runs your entire app. You can alternatively run docker-compose up using the docker-compose binary.

A docker-compose.yml looks like this:

version: "3.9"  # optional since v1.27.0
services:
  web:
    build: .
    ports:
      - "5000:5000"
    volumes:
      - .:/code
      - logvolume01:/var/log
    links:
      - redis
  redis:
    image: redis
volumes:
  logvolume01: {}

For more information about the Compose file, see the Compose file reference.

Compose has commands for managing the whole lifecycle of your application:

Start, stop, and rebuild services
View the status of running services
Stream the log output of running services
Run a one-off command on a service

How to install

Install Docker Compose | Docker Documentation

On Linux, you can download the Docker Compose binary from the Compose repository release page on GitHub. Follow the instructions from the link, which involve running the curl command in your terminal to download the binaries. These step-by-step instructions are also included below.

WARNING
For alpine, the following dependency packages are needed: py-pip, python-dev, libffi-dev, openssl-dev, gcc, libc-dev, and make.

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.26.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Healthcheck

Do it as follows:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 10s
  timeout: 5s
  retries: 5

The dependency check is as follows.

depends_on:
  postgres-database:
    condition: service_healthy

curl Example:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080"]
  interval: 1m30s
  timeout: 10s
  retries: 3
  start_period: 40s

Using wget:

healthcheck:
   test: wget --no-verbose --tries=1 --spider http://localhost || exit 1
   interval: 5m
   timeout: 3s
   retries: 3
   start_period: 2m

Using Python's urllib:

healthcheck:
   test: python -c "import urllib.request as r; exit(0 if r.urlopen('http://www.python.org/').status == 201 else 1)"
   interval: 5m
   timeout: 3s
   retries: 3
   start_period: 2m

How to scale

You can adjust the number of replicas of a service using the scale command.

docker-compose up -d --scale service_name=N

Used like this, it adjusts the number of replicas of a specific service.

Environment variables

Just have a .env file.

To raise an error when an environment variable is missing, use ${USER_VAR:?err}.

How to build from a Compose file

Compose file build reference

services:
  foo:
    build:
      context: ./dir
      dockerfile: backend.Dockerfile

How to extend by referencing another Compose file

Extend your Compose file | Docker Docs

If the compose.yaml file is as follows:

services:
  web:
    extends:
      file: common-services.yml
      service: webapp

The compose-services.yaml file referenced above is as follows:

services:
  webapp:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - "/data"

In the case above, web in the compose.yaml file receives all settings, i.e. everything inside webapp in the compose-services.yaml file.

Furthermore, you can define or override configuration locally in compose.yaml:

services:
  web:
    extends:
      file: common-services.yml
      service: webapp
    environment:
      - DEBUG=1
    cpu_shares: 5

  important_web:
    extends: web
    cpu_shares: 10

How to specify the user

Using current user when running container in docker-compose - Stack Overflow

  my-service:
    image: ubuntu:latest
    user: ${MY_UID}:${MY_GID}
    volumes:
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro

When running:

MY_UID="$(id -u)" MY_GID="$(id -g)" docker-compose up

How to restart

Docker Compose - Summary of automatic restart on reboot/error

api-server:
    container_name: api-server
    build:
        context: api-server
        dockerfile: Dockerfile
    image: product-server:latest
    networks:
        - web-network
    restart: on-failure:5

no: The container does not restart automatically
on-failure[:max-retries]: Restarts the container if it exits with a non-zero exit code, up to the maximum number of attempts the Docker daemon may make to restart the container
always: Always restarts the container when it stops
unless-stopped: Always restarts the container unless it is stopped arbitrarily or by the Docker daemon

Restart policy

version: "3.8"
services:
  redis:
    image: redis:alpine
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s

delay: Duration between restart attempts
max_attempts: Maximum number of attempts outside the restart period
window: Period used to decide whether a restart has succeeded

Log file driver

Compose file version 3 reference | Docker Docs # logging

version: "3.8"
services:
  some-service:
    image: some-service
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

Enabling GPU access with Compose

Run a sample CUDA container:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

If you want to save a command like the one above in a docker-compose.yml file:

capabilities: This value specifies as a list of strings (eg. capabilities: [gpu]). You must set this field in the Compose file. Otherwise, it returns an error on service deployment.
count: This value, specified as an integer or the value all, represents the number of GPU devices that should be reserved (providing the host holds that number of GPUs). If count is set to all or not specified, all GPUs available on the host are used by default.
device_ids: This value, specified as a list of strings, represents GPU device IDs from the host. You can find the device ID in the output of nvidia-smi on the host. If no device_ids are set, all GPUs available on the host are used by default.
driver: This value is specified as a string, for example driver: 'nvidia'
options: Key-value pairs representing driver specific options.

Example of a Compose file for running a service with access to 1 GPU device:

services:
  test:
    image: nvidia/cuda:12.3.1-base-ubuntu20.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

To enable access only to GPU-0 and GPU-3 devices:

services:
  test:
    image: tensorflow/tensorflow:latest-gpu
    command: python -c "import tensorflow as tf;tf.test.gpu_device_name()"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0', '3']
            capabilities: [gpu]

Legacy

For reference, using the runtime attribute is an old method used in the Compose v2.3 format.

services:
  test:
    image: nvidia/cuda:10.2-base
    command: nvidia-smi
    runtime: nvidia

Troubleshooting

YAML invalid type

ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.db.environment contains {"POSTGRES_USER": "user"}, which is an invalid type, it should be a string

This mistake often happens when editing variables such as environment; you need to clearly distinguish between a sequence and key-value.

services:
  db:
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: 0000

The above is key-value.

services:
  db:
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=0000

The above is a sequence. Note that the equals (=) character is used.

depends_on status check

Control startup and shutdown order in Compose

The dependency check in depends_on does not check whether the image is actually up. You need to write a separate script that communicates with it.