Setup Celery Worker with Supervisord on elastic beanstalk via .ebextensions

Introduction: The Backbone of Scalable Applications

Building a robust, scalable application often means dealing with tasks that require more than a single server or thread can handle efficiently. Whether it's processing images, sending emails, or performing data-heavy computations, offloading these tasks to a task queue is a best practice. For . Each infographic generation request is computationally intensive, involving AI-based topic research, design optimization, and sourcing vector graphics. To maintain a seamless user experience, these tasks must be offloaded to a background worker that can handle multiple requests concurrently. Celery’s asynchronous task handling and scalability made it the obvious choice.

Why Supervisord?

While Elastic Beanstalk can manage web servers natively, it doesn’t have built-in support for background processes like Celery workers. Enter Supervisord. It acts as a supervisor for the Celery worker process, ensuring that it runs continuously and restarts automatically if it fails. This reliability is crucial for processing infographic generation requests without interruptions.

With the stage set, let’s dive into the technical details of configuring Celery, Supervisord, and eb_extensions on Elastic Beanstalk to create a scalable and efficient task queue for your application.

Step-by-Step: Setting Up Celery with Supervisord on Elastic Beanstalk

In this section, we'll walk through the .ebextensions files required to set up Celery with Supervisord on Elastic Beanstalk. Each step is explained in detail, with tips to help you avoid common pitfalls.

1. Installing Supervisord

File: 01_install_supervisord.config

This file installs Supervisord and sets up a non-root user for running processes securely.

commands:

  01_install_pip:

    command: "yum install -y python3-pip"

    ignoreErrors: true

  02_install_supervisor:

    command: "/usr/bin/pip3 install supervisor"

  03_create_nonroot_user:

    command: "useradd -r -M -s /sbin/nologin nonrootuser || true"

    ignoreErrors: true

Explanation:

Install pip: Ensures Python's package manager is available.

Install Supervisor: Uses pip to install Supervisord, a lightweight and powerful process manager.

Create non-root user: Adds a restricted user (nonrootuser) with no login shell or home directory. Running processes as a non-root user is a security best practice.

💡 Tip: Always use ignoreErrors: true when commands might fail during repeated deployments. This ensures your deployment won’t fail if the user or package already exists.

2. Cleaning Up Stale Processes

File: 02_cleanup_existing_supervisord.config

This file handles cleanup of old Supervisord instances and socket files that might linger between deployments.

commands:

  kill_existing_supervisord:

    command: "pkill supervisord || true"

    ignoreErrors: true

  remove_stale_socket:

    command: "rm -f /tmp/supervisor.sock"

    ignoreErrors: true

Explanation:

Kill existing Supervisord: Ensures no stray Supervisord processes are running. The || true part ensures this command won't throw errors if no process is found.

Remove stale socket: Deletes any old Supervisord socket files, which could prevent Supervisord from starting.

💡 Tip: Cleaning up sockets and processes is essential in environments like Elastic Beanstalk, where deployments can sometimes leave behind remnants of previous configurations.

3. Configuring Celery with Supervisord

File: 03_celery_configuration.config

This file creates the Supervisord configuration file and starts the Celery worker process.

files:

  "/etc/supervisord.conf":

    mode: "000644"

    owner: root

    group: root

    content: |

      [unix_http_server]

      file=/tmp/supervisor.sock

      chmod=0770

      chown=root:nonrootuser



      [supervisord]

      logfile=/var/log/supervisord.log

      logfile_maxbytes=50MB

      logfile_backups=10

      loglevel=info

      pidfile=/tmp/supervisord.pid

      nodaemon=false

      minfds=1024

      minprocs=200

      user=root



      [rpcinterface:supervisor]

      supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface



      [supervisorctl]

      serverurl=unix:///tmp/supervisor.sock



      [program:celery]

      command=celery -A application.celery worker --loglevel=INFO

      directory=/var/app/current

      autostart=true

      autorestart=true

      startsecs=10

      stopwaitsecs=600

      stdout_logfile=/var/log/celery_worker.log

      stderr_logfile=/var/log/celery_worker.err.log

      environment=PATH="/var/app/venv/staging-LQM1lest/bin:$PATH"

      user=nonrootuser

Explanation:

Unix socket for control: The unix_http_server section creates a secure socket for interacting with Supervisord.

Logging: Logs are stored in /var/log/supervisord.log, with a rotation policy to prevent disk usage from spiraling out of control.

Celery program block:

Command: Runs the Celery worker with the application configuration.

Autostart and autorestart: Ensures Celery starts automatically on deployment and restarts if it fails.

Logs: Logs Celery’s output to /var/log/celery_worker.log and /var/log/celery_worker.err.log.

Environment: Ensures the correct Python virtual environment is used.

💡 Tip: Use directory=/var/app/current to point Supervisord to the application’s deployment directory, which is updated with each Elastic Beanstalk deployment.

4. Starting Supervisord

File: 03_celery_configuration.config (continued)

container_commands:

  01_start_supervisor:

    command: "supervisord -c /etc/supervisord.conf"

Explanation:

Container commands: These run after your application is deployed but before the environment is marked as ready. Starting Supervisord here ensures your Celery worker is running when the app goes live.

💡 Tip: Elastic Beanstalk processes container commands in alphabetical order, so prefix your commands with numbers like 01_ to control the execution order.

Fun Tricks with eb_extensions

Debugging Made Easy: If something doesn’t work, add a temporary container command to print environment variables or list directory contents:

container_commands:

  99_debug:

    command: "env > /tmp/env_vars.log && ls -al /var/app/current > /tmp/deployment_files.log"

Check the logs in /var/log/eb-activity.log.

Reuse Common Configs: Store shared configuration snippets in a separate YAML file, then include them in multiple .ebextensions files using the include directive (unofficially supported).

This setup ensures your Celery workers are managed efficiently with Supervisord, scaling alongside your Elastic Beanstalk application. Whether you're handling infographic generation or any other background task, this approach offers reliability, scalability, and peace of mind.