Automated Nginx and Fluentd Deployment on AWS EC2 using Ansible: A DevOps Guide

Table of Content

1. 
   

   Introduction

   
   
   
   
   
   • Brief overview of the problem: Managing web servers and logs at scale
   
   
   • Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination
   
   
   • Why This Stack Works Together
   
   
   
   



2. Prerequisites



3. 
   

   Infrastructure Setup

   
   
   
   
   
   • Create security group
   
   
   • Security Group configuration for web traffic
   
   
   • Verify the security group rules
   
   
   • Create your key pair
   
   
   • Launch EC2 instance using created security group
   
   
   • Copy keypair to ssh directory
   
   
   
   


4. 
   

   Ansible Configuration

   
   
   
   
   
   • Folder Structure
   
   
   • Prepare workspace environment
   
   
   • Create the host.yaml file
   
   
   • Test ansible connection
   
   
   
   



5. Set Up Roles




6. Common Task




7. Nginx Playbook




8. Security Playbook




9. Log Management FluentD Playbook




10. Deployment and Testing

Introduction

Brief overview of the problem: Managing web servers and logs at scale

In today's cloud environments, managing web servers and logs at scale presents significant challenges.

DevOps teams struggle with manual server configurations, inconsistent deployments, and the overwhelming task of processing massive log data for insights and security analysis.

Our automated solution combines AWS, Ansible, Nginx, and Fluentd to streamline these operations efficiently.

Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination

AWS EC2 + Ansible + Nginx + Fluentd creates a robust web infrastructure by combining cloud scalability with automation and efficient logging.

AWS EC2 provides the flexible compute resources, Ansible automates deployment and configuration tasks, Nginx serves as a high-performance web server, and Fluentd handles comprehensive log collection and processing.

Why This Stack Works Together

AWS EC2 (Infrastructure)

• Provides on-demand, scalable computing resources


• Offers multiple instance types to match workload needs


• Integrates seamlessly with other AWS services


• Enables global deployment with multiple regions

Ansible (Automation)

• Automates server configuration and application deployment


• Uses simple YAML syntax for easy maintenance


• Requires no agents on managed servers (agentless)


• Ensures consistent configurations across all servers

Nginx (Web Server)

• Delivers high-performance web serving capabilities


• Handles concurrent connections efficiently


• Provides reverse proxy and load balancing features


• Offers robust security features and SSL/TLS support

Prerequisites

1. Basic Knowledge of AWS & Ansible


2. 
   AWS CLI installation on local machine


3. 
   Configure AWS CLI


4. Install Ansible

Infrastructure Setup

Before we proceed, with setting up infrastructure. It is best we confirm Ansible is installed on our machine and AWS CLI is well configured.

ansible --version

aws sts get-caller-identity

Now we can go ahead with setting up infrastructure.

Create security group

Next step is to create the security group, copy and paste the below code in your terminal

aws ec2 create-security-group \

    --group-name nginx-web-server-sg \

    --description "Security group for Nginx web server and SSH access"

This command would create a security group and output the

• GroupId


• and SecurityGroupArn;
  

{

    "GroupId": "sg-0ee2b6c700c11e902",

    "SecurityGroupArn": "arn:aws:ec2:us-east-1:910883278292:security-group/sg-0ee2b6c700c11e902"

}

You can login to your AWS console to confirm the creation of the Security Group

Security Group configuration for web traffic

Lets Add the necessary inbound rules for HTTP (80), HTTPS (443), and SSH (22):

• Export the Security Group ID to a variable (from the previous command output; Replace with your actual Security Group ID)
  

export SG_ID="sg-0ee2b6c700c11e902"

confirm you have exported your SG_ID

echo $SG_ID

• Allow HTTP (port 80)
  

aws ec2 authorize-security-group-ingress \

    --group-id $SG_ID \

    --protocol tcp \

    --port 80 \

    --cidr 0.0.0.0/0

The above command should output similar results (Remember your security ID is different from mine)

{

    "Return": true,

    "SecurityGroupRules": [

        {

            "SecurityGroupRuleId": "sgr-081264c3c8fe3e58c",

            "GroupId": "sg-0ee2b6c700c11e902",

            "GroupOwnerId": "910883278292",

            "IsEgress": false,

            "IpProtocol": "tcp",

            "FromPort": 80,

            "ToPort": 80,

            "CidrIpv4": "0.0.0.0/0",

            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-081264c3c8fe3e58c"

        }

    ]

}

1. Allow HTTPS (port 443)
   

aws ec2 authorize-security-group-ingress \

    --group-id $SG_ID \

    --protocol tcp \

    --port 443 \

    --cidr 0.0.0.0/0

Command output should be similar

{

    "Return": true,

    "SecurityGroupRules": [

        {

            "SecurityGroupRuleId": "sgr-06591dee28547a3eb",

            "GroupId": "sg-0ee2b6c700c11e902",

            "GroupOwnerId": "910883278292",

            "IsEgress": false,

            "IpProtocol": "tcp",

            "FromPort": 443,

            "ToPort": 443,

            "CidrIpv4": "0.0.0.0/0",

            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06591dee28547a3eb"

        }

    ]

}

• Allow SSH (port 22) - Best practice is to limit this to your IP
  

aws ec2 authorize-security-group-ingress \

    --group-id $SG_ID \

    --protocol tcp \

    --port 22 \

    --cidr 0.0.0.0/0

Note: for a more secure environment, its advisable to only allow your specific IPs instead of using 0.0.0.0/0 as the cidr block

Command output should be similar

{

    "Return": true,

    "SecurityGroupRules": [

        {

            "SecurityGroupRuleId": "sgr-06c1f6c3205247aea",

            "GroupId": "sg-0ee2b6c700c11e902",

            "GroupOwnerId": "910883278292",

            "IsEgress": false,

            "IpProtocol": "tcp",

            "FromPort": 22,

            "ToPort": 22,

            "CidrIpv4": "105.113.64.38/32",

            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06c1f6c3205247aea"

        }

    ]

}

Verify the security group rules

aws ec2 describe-security-groups --group-ids $SG_ID

Due to the how lengthy the command outputs for the security group verification is, I wont be posting it here.

Create your key pair

This command would create your key pair, and save it on your local machine

aws ec2 create-key-pair --key-name nginx-server-key --query 'KeyMaterial' --output text > nginx-server-key.pem

Confirm the created key

cat nginx-server-key.pem

Launch EC2 instance using created security group

aws ec2 run-instances \

    --image-id ami-0e1bed4f06a3b463d \

    --instance-type t2.micro \

    --key-name nginx-server-key \

    --security-group-ids $SG_ID \

    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nginx-fluentd-server}]'

• *Image AMI: * ami-0e1bed4f06a3b463d : installs Ubuntu 22


• *Instance Type: * vCPUs: 1, Memory: 1 GiB


• 
  Key Pair: Key=Name,Value=nginx-fluentd-server : identifies as the name of the server

Copy keypair to ssh directory

cp nginx-server-key.pem ~/.ssh

SSH into EC2 instance

You need this step to confirm you can actually log into the machine.

• Change key pair permissions
  chmod 400 ~/.ssh/nginx-server-key.pem
  


• Connect using ssh
  ssh -i ~/.ssh/nginx-server-key.pem ubuntu@<instance-ip-address>
  

Once you can log into the instance via ssh, we can move to the next section.

Ansible Configuration

Folder Structure

.

├── hosts.yaml

├── site.yml

└── roles/

    ├── common/

    ├── └── defaults/

    │   |   └── main.yml

    │   └── tasks/

    │   |    └── main.yml

    │   └── handlers/

    │       └── main.yml

    ├── nginx/

    │   └── tasks/

    │       └── main.yml

    ├── security/

    |   └── defaults/

    │   |   └── main.yml

    │   └── tasks/

    │       └── main.yml

    └── fluentd/

        └── defaults/

        |   └── main.yml

        └── tasks/

        |   └── main.yml

        └── files/

        |   └── denylist.txt

        └── templates/

            └── td-agent.conf.j2

Prepare workspace environment

Open your preferred terminal or use vscode terminal, create a new folder called AWS-Nginx-Ansible-FluentD and navigate into that folder

mkdir AWS-Nginx-Ansible-FluentD

cd AWS-Nginx-Ansible-FluentD

Create the host.yaml file

The hosts.yaml file is an Ansible inventory file that defines the target servers (hosts) Ansible will manage.

This file tells Ansible where and how to connect to the managed nodes for executing tasks or running playbooks.

touch host.yaml

add the below code

---

nginx_server:

  hosts:

    nginx-server-1:

      ansible_host: <your-instance-public-ip>

      ansible_user: ubuntu

      ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/nginx-server-key.pem"

      ansible_ssh_common_args: "-o StrictHostKeyChecking=no"

Note: change the value of ansible_host to your created instance IP

Test ansible connection

Run the below command in the project folder terminal

ansible nginx-server -i hosts.yaml -m ping

If connection is successfull, you should see the below result

nginx-server-1 | SUCCESS => {

    "ansible_facts": {

        "discovered_interpreter_python": "/usr/bin/python3.10"

    },

    "changed": false,

    "ping": "pong"

}

Set Up Roles

In Ansible, roles are a way to organize and reuse configurations by breaking them into modular, reusable components.

Each role contains specific;

• Tasks,


• Variables,


• Templates,


• Files,


• and handlers
  required to perform a particular function, such as installing a web server or configuring a database.

By using roles, you can structure your playbooks more efficiently, making them cleaner, more scalable, and easier to maintain

1. Create site.yaml: This file is the root directory for all roles.
   

touch site.yaml

1. Add the below snippet to the playbook
   

---

- name: Configure Web Server with Nginx and Fluentd

  hosts: nginx_server

  become: yes



  roles:

    - common

    - nginx

    - security

    - fluentd

Common Task

Create Required directories and files

• In the root directory, create a folder called common
  

mkdir common

• Create defaults, handler & task folders
  

mkdir defaults handler tasks

• Create the main files in defaults, handler & tasks
  

touch defaults/main.yaml handler/main.yaml tasks/main.yaml

Common Playbook

• 
  variable definition: Add the below code to common\defaults\main.yaml
  The content of the file defines settings for your Ansible tasks. In simple terms, it’s configuring paths and settings for Nginx and Fluentd, as well as setting a rule for how long logs are retained.
  

---

nginx_html_root: /var/www/html

fluentd_config_dir: /etc/td-agent

log_retention_days: 5

• 
  Handler Task List: Add the below code to common\handler\main.yaml
  Restart Nginx: Restarts the nginx service to apply any changes or ensure it is running.
  Restart Fluentd: Restarts the fluentd service to reload its configuration or ensure it is operational.
  

---

- name: restart nginx

  service:

    name: nginx

    state: restarted



- name: restart fluentd

  service:

    name: fluentd

    state: restarted

• 
  Main Common Task: Add the below code to common\tasks\main.yaml
  The content of the file updates the package cache on systems using apt (like Debian or Ubuntu).
  

---

- name: Update apt cache

  apt:

    update_cache: yes

    cache_valid_time: 3600

update_cache: yes: Refreshes the list of available packages from repositories.

cache_valid_time: 3600: Ensures the cache is valid for 3600 seconds (1 hour), skipping updates if the cache is still recent.

Nginx Playbook

• In the root directory, create a folder called nginx
  

mkdir nginx

• Create nginx task folder
  

mkdir nginx/tasks

• Create the nginx tasks files
  

touch nginx/tasks/main.yaml

• 
  Nginx Task playbook: Add the below code to nginx/tasks/main.yaml
  

---

- name: Install Nginx

  apt:

    name: nginx

    state: present



- name: Create simple HTML page

  copy:

    content: |

      <!DOCTYPE html>

      <html>

      <head><title>Hello, World!</title></head>

      <body><h1>Hello, World!</h1></body>

      </html>

    dest: "{{ nginx_html_root }}/index.html"

    mode: '0644'

  notify: restart nginx



- name: Enable and start Nginx

  service:

    name: nginx

    state: started

    enabled: yes



- name: Configure logrotate for Nginx

  template:

    src: nginx-logrotate.j2

    dest: /etc/logrotate.d/nginx

    mode: '0644'

This is an Ansible Playbook Task List that sets up and configures Nginx with the following steps:

1. 
   Install Nginx: Ensures the Nginx package is installed on the target machine.


2. 
   Create a Simple HTML Page: Copies a basic “Hello, World!” HTML file to the Nginx web root (nginx_html_root), with proper permissions (0644). It also triggers a restart nginx handler when changes occur.


3. 
   Enable and Start Nginx: Ensures the Nginx service is running (started) and configured to start automatically on system boot (enabled).


4. 
   Configure Logrotate for Nginx: Deploys a log rotation configuration file for Nginx using a Jinja2 template (nginx-logrotate.j2), setting appropriate permissions (0644).

Security Playbook

• In the root directory, create a folder called security
  

mkdir security

• Create security defaults & task folder
  

mkdir security/defaults security/tasks 

• Create the security defaults & tasks files
  

touch security/defaults/main.yaml security/tasks/main.yaml

• 
  Security variable definition: Add the below code to security/defaults/main.yaml
  

This file sets firewall rules to allow essential traffic and disables unnecessary services to enhance security.

---

ufw_rules:

  - { rule: 'allow', port: '80', proto: 'tcp' }

  - { rule: 'allow', port: '443', proto: 'tcp' }

  - { rule: 'allow', port: '22', proto: 'tcp' }



disabled_services:

  - rpcbind

  - cups

  - avahi-daemon

• 
  Security Task playbook: Add the below code to security/tasks/main.yaml
  

---

- name: Install UFW

  apt:

    name: ufw

    state: present



- name: Configure UFW rules

  ufw:

    rule: "{{ item.rule }}"

    port: "{{ item.port }}"

    proto: "{{ item.proto }}"

    state: enabled

  with_items: "{{ ufw_rules }}"



- name: Disable unnecessary services

  service:

    name: "{{ item }}"

    state: stopped

    enabled: no

  with_items: "{{ disabled_services }}"

  ignore_errors: yes

These tasks install and configure a firewall for secure access while stopping and disabling unneeded services to improve security and performance.

FluentD Playbook

Fluentd is used to centralize and manage log data efficiently, enabling better monitoring, troubleshooting, and analytics for applications and systems.

• In the root directory, create a folder called security
  

mkdir fluentd

• Create fluentd defaults, files, task & templates folder
  

mkdir fluentd/defaults fluentd/files fluentd/tasks  fluentd/templates 

• Create the fluentd defaults, files, task & templates files
  

touch fluentd/defaults/main.yaml fluentd/files/denylist.txt fluentd/tasks/main.yaml fluentd/templates/td-agent.conf.j2

• 
  fluentd variable definition: Add the below code to fluentd/defaults/main.yaml
  

fluentd_config_dir: "/etc/fluentd"

This variable tells Ansible (or any script using it) where to find or manage Fluentd’s configuration files, which typically include settings for log inputs, filters, and outputs.

• 
  Fluentd files: To deny a list of IP addresses, you can add them to the fluentd/files/denylist.txtfile.
  

192.168.1.100

10.0.0.50

• 
  Fluentd Task Playbook: Add the below code to fluentd/tasks/main.yaml
  

---

- name: Download and run Fluentd installation script

  shell: |

    curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh

  args:

    executable: /bin/bash



- name: Verify Fluentd installation

  command: fluentd --version

  register: fluentd_version



- debug:

    var: fluentd_version.stdout



- name: Create Fluentd config directory

  file:

    path: "{{ fluentd_config_dir }}"

    state: directory

    mode: '0755'

  become: yes



- name: Create Fluentd config

  template:

    src: td-agent.conf.j2

    dest: "{{ fluentd_config_dir }}/td-agent.conf"

    mode: '0644'

  notify: restart fluentd



- name: Enable and start Fluentd

  service:

    name: fluentd

    state: started

    enabled: yes

This playbook automates installing Fluentd, setting up its configuration, and ensuring the service is up and running.

• 
  Fluentd Configuration Jinja Templates: Add the below code to fluentd/templates/td-agent.conf.j2
  

# Input for Nginx access logs

<source>

  @type tail

  path /var/log/nginx/access.log

  pos_file /var/log/td-agent/nginx.access.pos

  tag nginx.access

  <parse>

    @type nginx

  </parse>

</source>



# Input for Nginx error logs

<source>

  @type tail

  path /var/log/nginx/error.log

  pos_file /var/log/td-agent/nginx.error.pos

  tag nginx.error

  <parse>

    @type regexp

    expression /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).*?: (?<message>.*)$/

    time_format %Y/%m/%d %H:%M:%S

  </parse>

</source>



# Filter to check IPs against denylist

<filter nginx.access>

  @type grep

  <regexp>

    key remote_addr

    pattern /^(?!#{File.readlines("#{ENV['FLUENT_CONFIG_DIR'] || '/etc/td-agent'}/denied_ips/denylist.txt").map(&:strip).join('|')}).*$/

  </regexp>

</filter>



# Route normal logs (non-denied IPs)

<match nginx.access>

  @type file

  path /var/log/td-agent/nginx_access

  append true

  <buffer>

    timekey 1d

    timekey_use_utc true

    timekey_wait 10m

  </buffer>

  <format>

    @type json

  </format>

</match>



# Route denied IP logs to audit file

<match nginx.access>

  @type copy

  <store>

    @type file

    path /var/log/td-agent/audit/denylist_audit

    append true

    <buffer>

      timekey 1d

      timekey_use_utc true

      timekey_wait 10m

    </buffer>

    <format>

      @type json

      include_time_key true

      time_key timestamp

    </format>

  </store>

</match>



# Handle error logs

<match nginx.error>

  @type file

  path /var/log/td-agent/nginx_error

  append true

  <buffer>

    timekey 1d

    timekey_use_utc true

    timekey_wait 10m

  </buffer>

  <format>

    @type json

  </format>

</match>

This configuration collects and processes Nginx access and error logs. It checks access logs against a denylist, routes normal logs and denied IP logs to separate files, and stores error logs in a structured JSON format for easy analysis.

This setup is ideal for monitoring, auditing, and managing log data efficiently.

If you’ve made it this far, well done! Congratulations!

Deployment and Testing

Running the Playbook

Check syntax

ansible-playbook -i hosts.yaml site.yaml --syntax-check

Run playbook

ansible-playbook -i hosts.yaml site.yaml

Ansible would run through all the task listed in the site.yaml file, and run each of the playbooks sequentially.

If you face an error with TASK [security : Disable unnecessary services] theres no need to panic, it just means, you don't have the unnecessary services to disable and you playbook, would continue to the other task

Verification Steps

Check Nginx status

ansible webservers -i hosts.yaml -m shell -a "systemctl status nginx"  

or input the instance public ip on a browser, to get the Hello, World! page

Check Fluentd status

ansible webservers -i hosts.yaml -m shell -a "systemctl status td-agent"

Test log processing

First make a curl request to the server

ansible nginx_server -i hosts.yaml -m shell -a "curl http://localhost"

Then check the tail of the fluentd logs

ansible nginx_server -i hosts.yaml -m shell -a "tail  /var/log/fluent/fluentd.log"

Conclusion

This solution provides a robust, automated approach to deploying and managing web servers on AWS.

By combining Ansible's automation capabilities with AWS services, we create a scalable and maintainable infrastructure that follows DevOps best practices.

🎉 Congratulations! If you've made it this far, you've just learned how to:

• Launch and configure an EC2 instance like a pro 🚀


• Set up a secure web server that would make security experts nod in approval 🔒


• Create a logging system that catches every digital whisper 🔍


• Automate everything so smoothly that your future self will thank you ⚡

Fun fact: The automation you just built will save you approximately 127 cups of coffee worth of manual configuration time per year! ☕