It's midnight. You get an incident call on your phone that your application's web server has crashed, and users are seeing the dreaded 500 Internal server error. You stumble to your laptop, sleepy-eyes, to run the restart command or to run your restart script.
This is the "Old Way." It's manual, it's reactive, and it ruins your sleep.
In the world of DevOps, we don't just fix things; we build things that fix themselves (self-healing). In this article, we're going to build a simple automation that detects a web server crash and restarts the service before you even roll over in bed.
The "Healing" Architecture
To build this, we need five simple AWS resources:
- The Sensor (Amazon CloudWatch): The eye that monitors your server.
- The Tripwire (CloudWatch Alarms): The logic that says, "Wait, something is wrong."
- The Router (Amazon EventBridge): The nervous system that delivers the alert to our code.
- The Brain (AWS Lambda): The function that decides what to do.
- The Hands (AWS Systems Manager): The tool that actually reaches out and restarts the service.
The flow looks like this:
Start a service: SSH into your server and install Nginx:
sudo dnf update -y
sudo dnf install nginx -y
sudo systemctl enable --now nginx
Open your browser to the instance's public IP. You should see the "Welcome to Nginx" page.
Step 2: The "Eyes" Installation (CloudWatch Agent)
The default EC2 metrics can't see inside your server. We need the agent to monitor the Nginx process.
Install the Agent: On your EC2, run:
sudo dnf install amazon-cloudwatch-agent -y
Configure the "Nginx Watcher": CloudWatch doesn't know what Nginx is until you tell it. Create a configuration file to monitor the process:
sudo nano /opt/aws/amazon-cloudwatch-agent/bin/config.json
Paste this configuration:
{
"agent": {
"metrics_collection_interval": 60
},
"metrics": {
"namespace": "CWAgent",
"metrics_collected": {
"procstat": [
{
"exe": "nginx",
"measurement": ["pid_count"]
}
]
}
}
}
Save the file and start the agen, you have to tell the agent to load the file:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
The agent is now running. Verify with:
sudo systemctl status amazon-cloudwatch-agent
After a minute or two, you should see a procstat_lookup pid_count metric appear in CloudWatch under the CWAgent namespace.
Confirm AWSLambdaBasicExecutionRole is attached (it usually is by default). Then click Add permissions → Create inline policy, switch to the JSON tab, and paste:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ssm:SendCommand",
"Resource": [
"arn:aws:ec2:REGION:ACCOUNT_ID:instance/i-YOUR_INSTANCE_ID",
"arn:aws:ssm:REGION::document/AWS-RunShellScript"
]
}]
}
Replace REGION, ACCOUNT_ID, and the instance ID with your values.
Step 4: Set the Tripwire (The Alarm)
Now we tell AWS when to trigger that code.
Go to CloudWatch > Alarms and click "Create Alarm."
Select metric: CWAgent > procstat > pid_count (filter for your Nginx process).
Condition: Lower than 1 (meaning Nginx is not running).
Missing data treatment: Treat missing data as breaching. This is important — when Nginx dies, the procstat plugin may stop reporting the metric entirely rather than reporting zero. Without this setting, the alarm may never fire.
Notification: You can skip the SNS step entirely. With EventBridge, you don't configure any alarm action CloudWatch automatically publishes every alarm state change to the default event bus.
Give the alarm a memorable name like nginx-down-alarm. You'll reference it in the next step.
Step 5: Route the Alarm to Lambda with EventBridge
Here's where EventBridge shines. CloudWatch Alarms publish state-change events to the default event bus automatically, we just need a rule that catches the ones we care about and sends them to our Lambda.
Go to EventBridge > Rules and click "Create rule."
Name: nginx-down-rule
Event bus: default
Rule type: Rule with an event pattern
Event source: AWS services
Event pattern: Use the custom pattern editor and paste this:
{
"source": ["aws.cloudwatch"],
"detail-type": ["CloudWatch Alarm State Change"],
"detail": {
"alarmName": ["nginx-down-alarm"],
"state": {
"value": ["ALARM"]
}
}
}
EventBridge will automatically add the invoke permission on your function. No subscriptions, no topic management.
Step 6: Testing the Self-Heal
We're going to kill Nginx and watch it come back.
- Confirm your server's IP shows the Nginx welcome page in a browser.
- SSH into your server and stop the service:
sudo systemctl stop nginx
- Refresh the browser; the site is down.
- Wait. Within 1–2 minutes, the metric drops, the alarm trips, EventBridge fires, Lambda runs, and SSM restarts Nginx.
- Refresh again; The site is back up, and you didn't lift a finger.
If something doesn't fire, check the alarm history in CloudWatch first, then your Lambda's CloudWatch Logs. EventBridge also has a "Monitoring" tab on each rule showing invocation counts and failures, which is handy for debugging.
Why This Matters for Your Career
If you can walk an interviewer through this project, you're demonstrating skills that hiring managers genuinely look for in entry-level DevOps roles. You aren't just "using AWS"; you are demonstrating:
Event-Driven Architecture: Triggering actions based on state changes.
Least Privilege: Giving the Lambda only the permissions it needs.
MTTR (Mean Time To Recovery): You just reduced your recovery time from "whenever I wake up" to "under 120 seconds."
A Note on Cost
Most of what we used in this article fits inside the AWS Free Tier, the EC2 t3.micro, Lambda invocations, and EventBridge events from AWS services (like CloudWatch alarm state changes) are all free. One caveat: CloudWatch custom metrics (which is what procstat produces) are only free for the first 10 metrics, so a single procstat metric is fine, but the cost can scale up if you expand this pattern broadly.
When you're done experimenting, terminate the EC2 instance, delete the CloudWatch alarm, and remove the EventBridge rule to avoid any surprise bills.
SOCIAL SHARE CARD GENERATOR