🔧 Dynamically start Docker Compose Services with a simple Makefile

Efficiently Managing Docker Compose Services with a Dynamic Makefile

As developers, we often find ourselves juggling multiple services when working on projects, especially when using Docker and Docker Compose. Sometimes, the services required in a production environment aren't necessary for local development. Other times, we might need to toggle specific services on or off based on the task at hand. Managing this can become cumbersome, but with a cleverly crafted Makefile, we can simplify the process significantly.

In this post, I'll walk you through a Makefile that dynamically manages Docker Compose files and services, allowing you to switch between environments and toggle services effortlessly.

The Challenge

When working with Docker Compose, you might have multiple services defined across different YAML files. For example:

• docker-compose.yml (the main Compose file)
• docker-compose.dev.yml (development-specific configurations)
• docker-compose.prod.yml (production-specific configurations)
• Additional service files like docker-compose.nginx.yml, docker-compose.smtp4dev.yml, etc.

Managing these files manually can be error-prone and time-consuming. You might need to include or exclude certain services or switch between development and production environments. Without automation, this can lead to complex docker-compose commands and potential mistakes.

The Solution: A Dynamic Makefile

The Makefile below addresses these challenges by:

• Allowing you to toggle services on or off.
• Dynamically including Docker Compose files based on enabled services.
• Providing commands to start, stop, and destroy containers in different environments.
• Ensuring that your .env file is correctly set up.

The Makefile

# Makefile
FOLDER := ./docker

# The services that can be toggled
SERVICES := nginx smtp4dev

# Function to dynamically add Docker Compose files based on enabled services
define compose_files
  $(foreach service,$(SERVICES),$(if $($(shell echo $(service) | tr a-z A-Z)), -f $(FOLDER)/docker-compose.$(service).yml))
endef

# Function to dynamically add Docker Compose dev files based on enabled services and file existence
define compose_dev_files
  $(foreach service,$(SERVICES), \
  $(if $($(shell echo $(service) | tr a-z A-Z)), \
    $(if $(shell [ -f $(FOLDER)/docker-compose.$(service).dev.yml ] && echo yes), -f $(FOLDER)/docker-compose.$(service).dev.yml)))
endef

# Function to dynamically add Docker Compose prod files based on enabled services and file existence
define compose_prod_files
  $(foreach service,$(SERVICES), \
  $(if $($(shell echo $(service) | tr a-z A-Z)), \
    $(if $(shell [ -f $(FOLDER)/docker-compose.$(service).prod.yml ] && echo yes), -f $(FOLDER)/docker-compose.$(service).prod.yml)))
endef

# Ensure .env file exists and matches .env.example
check-env:
    @if [ ! -f "$(FOLDER)/.env" ]; then \
        echo "Error: .env file does not exist. Creating one now from .env.example ..."; \
        cp $(FOLDER)/.env.example $(FOLDER)/.env; \
    fi
    @echo "Checking .env file for missing variables..."
    @awk -F '=' 'NR==FNR {a[$$1]; next} !($$1 in a) {print "Missing env var: " $$1}' $(FOLDER)/.env $(FOLDER)/.env.example

init:
    cp $(FOLDER)/.env.example $(FOLDER)/.env

# Start command: runs docker-compose with the main file and any additional service files
start: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml $(call compose_files) up -d

# Dev command: runs docker-compose with the main file, dev file, and any additional service dev files (if they exist)
dev: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) up -d

# Start-prod command: runs docker-compose with the main file, prod file, and any additional service prod files (if they exist)
start-prod: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.prod.yml $(call compose_files) $(call compose_prod_files) up -d

# Stop command: stops the running containers
stop:
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) $(call compose_prod_files) down

# Destroy command: stops the running containers and removes the volumes
destroy:
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) $(call compose_prod_files) down -v

Breaking Down the Makefile

Let's delve into how this Makefile works and how you can leverage it in your projects.

Variables and Services

FOLDER := ./docker

# The services that can be toggled
SERVICES := nginx smtp4dev

• FOLDER: Specifies the directory where your Docker Compose files reside.
• SERVICES: A list of services that you might want to toggle on or off.

Dynamic Functions

compose_files

define compose_files
  $(foreach service,$(SERVICES),$(if $($(shell echo $(service) | tr a-z A-Z)), -f $(FOLDER)/docker-compose.$(service).yml))
endef

This function:

• Iterates over each service in SERVICES.
• Checks if an environment variable corresponding to the uppercase service name is set (e.g., NGINX, SMTP4DEV).
• If the variable is set, it includes the corresponding Docker Compose file.

compose_dev_files and compose_prod_files

These functions are similar to compose_files but specifically target development and production files, respectively. They also check if the files exist before including them.

Environment Check

check-env:
    @if [ ! -f "$(FOLDER)/.env" ]; then \
        echo "Error: .env file does not exist. Creating one now from .env.example ..."; \
        cp $(FOLDER)/.env.example $(FOLDER)/.env; \
    fi
    @echo "Checking .env file for missing variables..."
    @awk -F '=' 'NR==FNR {a[$$1]; next} !($$1 in a) {print "Missing env var: " $$1}' $(FOLDER)/.env $(FOLDER)/.env.example

This target ensures that your .env file exists and contains all the necessary variables as defined in .env.example.

Make Commands

start

start: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml $(call compose_files) up -d

Starts the containers using the main Docker Compose file and any additional service files based on the toggled services.

dev

dev: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) up -d

Starts the containers in development mode, including development-specific configurations and service files.

start-prod

start-prod: check-env
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.prod.yml $(call compose_files) $(call compose_prod_files) up -d

Starts the containers in production mode, including production-specific configurations and service files.

stop and destroy

stop:
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) $(call compose_prod_files) down

destroy:
    @docker compose -f $(FOLDER)/docker-compose.yml -f $(FOLDER)/docker-compose.dev.yml $(call compose_files) $(call compose_dev_files) $(call compose_prod_files) down -v

• stop: Stops all running containers.
• destroy: Stops all running containers and removes associated volumes.

How to Use This Makefile

Toggling Services

To enable or disable services, set environment variables corresponding to the uppercase service names.

For example, to enable nginx and disable smtp4dev:

make dev NGINX=1

Alternatively, you can set these variables directly in your shell or include them in your .env file.

Switching Environments

• Development: Use make dev to start services with development configurations.
• Production: Use make start-prod to start services with production configurations.
• Default: Use make start to start services with the default configurations.

Examples

Starting Services in Development Mode with nginx Enabled

make dev NGINX=1

Stopping All Services

make stop

Destroying All Services and Volumes

make destroy

Running Database Migrations

make migrate-up

Benefits of This Approach

• Flexibility: Easily toggle services on or off without modifying Docker Compose files.
• Environment Management: Seamlessly switch between development and production environments.
• Automation: Simplify complex docker-compose commands into easy-to-remember make commands.
• Consistency: Ensure that all team members use the same commands and configurations.

Conclusion

Managing multiple Docker Compose files and services doesn't have to be a headache. By using a dynamic Makefile, you can streamline your workflow, reduce errors, and focus on what matters most—writing great code.

Feel free to adapt and extend this Makefile to suit your project's needs. Happy coding!

Note: Remember to adjust the SERVICES variable and other configurations to match your specific project setup.