Building Secure Jenkins-Slack Integration with AWS Lambda - Part 1: Complete Setup Guide

Ever wished you could trigger your CI/CD pipelines directly from Slack without exposing Jenkins to the public internet? In this comprehensive two-part series, we'll build a production-ready solution that lets your team trigger Jenkins jobs using Slack slash commands while maintaining enterprise-grade security.

This is Part 1 of our series, focusing on architecture design, infrastructure setup, and initial implementation. Part 2 will dive deep into troubleshooting real-world issues and production enhancements.

What We'll Build

This solution enables teams to trigger Jenkins CI/CD pipelines from Slack using a secure, serverless architecture. The key innovation is keeping Jenkins completely private while providing a public API endpoint through AWS Lambda and API Gateway.

Tech Stack Overview

• 
  Infrastructure: Terraform (IaC), AWS VPC, EC2, Lambda, API Gateway, ALB, NAT Gateway


• 
  Backend: Go 1.20 (Lambda functions), Jenkins (CI/CD orchestration)


• 
  Storage/Secrets: AWS SSM Parameter Store


• 
  Integration: Slack API (Slash Commands, Webhooks)


• 
  Monitoring: CloudWatch Logs


• 
  OS: Amazon Linux 2

Architecture Deep Dive

Let's start with the complete architecture flow:

┌─────────────────────────────────────────────────────────────────┐

│                        Architecture Flow                        │

└─────────────────────────────────────────────────────────────────┘



👥 Slack User                    🌐 API Gateway

/run_test environment=staging ──→ /prod/trigger

                                        │

                                        ▼

                                ⚡ AWS Lambda

                                jenkins-trigger

                                        │

                    ┌───────────────────┼───────────────────┐

                    │                   │                   │

                    ▼                   ▼                   ▼

            🔐 AWS SSM Parameter    🔧 Jenkins (Private EC2)    🛡️ Bastion Host

            Store                   CI/CD Orchestration        SSH Access Gateway

            Jenkins URL &           ◄─────────────────────────┘

            Credentials             │

                    │               │

                    └───────────────┼───────────────────────────┐

                                    │                           │

                                    ▼                           ▼

                            ⚖️ Internal ALB              Job Status

                            Load Balancer                Notifications

                                    │                           │

                                    └───────────────────────────┘

                                                              │

                                                              ▼

                                                        👥 Slack User

                                                        (Status Updates)



Legend:

🟠 AWS Services (API Gateway, Lambda, SSM, ALB, Bastion)

🟣 Slack Integration

🔴 Jenkins (Private EC2)

Key Components Explained

🔒 VPC with Public/Private Subnets: Jenkins runs in a private subnet with no internet access, while the bastion host provides secure SSH access from the public subnet.

⚡ Lambda Function in VPC: The Lambda function runs in the private subnet to communicate with Jenkins while being invoked by API Gateway from the internet.

🌐 API Gateway: Provides a public HTTPS endpoint that Slack can reach, while the Lambda function handles the secure communication with Jenkins.

🔐 SSM Parameter Store: Stores Jenkins credentials and configuration securely, eliminating the need for hardcoded secrets.

🛡️ Bastion Host: Provides secure SSH access to Jenkins for administrative tasks without exposing Jenkins directly.

⚖️ Internal Load Balancer: Distributes traffic to Jenkins instances and provides health checks.

Security Benefits

• 
  Zero Public Exposure: Jenkins never faces the internet


• 
  Encrypted Secrets: All credentials stored in SSM with encryption


• 
  Network Isolation: Private subnets with controlled access


• 
  Least Privilege: IAM roles with minimal required permissions

Prerequisites

Before we start, ensure you have:

• AWS account with appropriate permissions (EC2, Lambda, API Gateway, SSM, VPC)


• Terraform >= 1.0.0 installed


• AWS CLI configured with credentials


• Go 1.20+ installed (for Lambda function builds)


• Slack workspace with admin privileges


• SSH key pair for EC2 access


• Basic understanding of Jenkins, Lambda, and Terraform

Step 0: Initial Setup

Configure AWS CLI

First, configure your AWS CLI with proper credentials:

# Configure AWS CLI

aws configure



# Enter your AWS Access Key ID, Secret Access Key, region (us-east-1), and output format (json)

# Verify configuration

aws sts get-caller-identity

Create SSH Key Pair

Create an EC2 key pair for SSH access:

# Option 1: Create via AWS CLI

aws ec2 create-key-pair --key-name jenkins-slack-key --query 'KeyMaterial' --output text > jenkins-slack-key.pem

chmod 400 jenkins-slack-key.pem



# Option 2: Create via AWS Console

# Go to EC2 → Key Pairs → Create Key Pair → Name: jenkins-slack-key → Download .pem file

Step-by-Step Implementation

Step 1: Clone and Setup

git clone https://github.com/kasukur/jenkins-slack-aws-integration

cd jenkins-slack-aws-integration

Step 2: Build Lambda Functions

Build the Lambda function binaries:

# Navigate to Lambda function directories

cd lambda/jenkins_trigger

go mod tidy

GOOS=linux GOARCH=amd64 go build -o bootstrap main.go

zip jenkins_trigger.zip bootstrap



cd ../slack_verification

go mod tidy

GOOS=linux GOARCH=amd64 go build -o bootstrap main.go

zip slack_verification.zip bootstrap



# Copy zips to terraform directory

cp jenkins_trigger.zip ../../terraform/

cp slack_verification.zip ../../terraform/

The repository structure:

├── terraform/           # Infrastructure as Code

│   ├── main.tf         # Provider and key pair

│   ├── vpc.tf          # VPC, subnets, security groups

│   ├── jenkins.tf      # Jenkins EC2 and ALB

│   ├── lambda.tf       # Lambda functions

│   ├── api_gateway.tf  # API Gateway configuration

│   ├── iam.tf          # IAM roles and policies

│   └── variables.tf    # Variable definitions

├── lambda/             # Lambda function source code

│   ├── jenkins_trigger/    # Main Lambda function

│   └── slack_verification/ # Slack signature verification

└── jenkins/            # Jenkins pipeline definitions

    └── Jenkinsfile     # Test pipeline configuration

Step 3: Configure Slack App

Create Slack App

1. 
   Go to Slack API: https://api.slack.com/apps
   


2. 
   Click "Create New App" → "From scratch"


3. 
   App Name: Jenkins CI/CD Integration
   


4. 
   Workspace: Select your workspace


5. Click "Create App"

Configure Slash Command

1. 
   In your app settings, go to "Slash Commands"
   


2. Click "Create New Command"


3. 
   Fill in the details:
   
   
   
   
   • 
     Command: /run_test
     
   
   
   • 
     Request URL: https://your-api-gateway-url/prod/trigger (we'll get this after deployment)
   
   
   • 
     Short Description: Trigger Jenkins test pipeline
     
   
   
   • 
     Usage Hint: [environment] [test_suite]
     
   
   
   
   


4. Click "Save"

Enable Incoming Webhooks

1. 
   Go to "Incoming Webhooks" in your app settings


2. 
   Toggle "Activate Incoming Webhooks" to On


3. Click "Add New Webhook to Workspace"


4. 
   Select a channel (e.g., #devops)


5. Click "Allow"


6. 
   Copy the webhook URL (starts with https://hooks.slack.com/services/)

Get App Credentials

1. 
   Go to "Basic Information" in your app settings


2. 
   In "App Credentials" section, copy:
   
   
   
   
   • 
     App-Level Token (starts with xapp-)
   
   
   • 
     Signing Secret (long random string)
   
   
   
   

Install App to Workspace

1. 
   Go to "Install App" in your app settings


2. Click "Install to Workspace"


3. 
   Review permissions and click "Allow"


4. 
   Copy the Bot User OAuth Token (starts with xoxb-)

Save these values for the Terraform configuration:

• App-Level Token (xapp-...)


• Signing Secret


• Webhook URL


• Bot User OAuth Token (xoxb-...)

Step 4: Prepare Terraform Variables

Copy the example configuration:

cp terraform/terraform.tfvars.example terraform/terraform.tfvars

Edit terraform/terraform.tfvars with your values:

# Basic Configuration

aws_region = "us-east-1"

project_name = "jenkins-slack-demo"



# Network Configuration

vpc_cidr = "10.0.0.0/16"

public_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]

private_subnet_cidrs = ["10.0.101.0/24", "10.0.102.0/24"]



# SSH Access

key_name = "your-key-pair-name"

public_key = "ssh-rsa AAAAB3NzaC1yc2E... [email protected]"



# Jenkins Configuration

jenkins_admin_username = "admin"

jenkins_admin_password = "your-secure-password"



# Slack Configuration

slack_app_token = "xapp-your-app-token"

slack_signing_secret = "your-signing-secret"

slack_webhook_url = "https://hooks.slack.com/services/your/webhook/url"

slack_bot_token = "xoxb-your-bot-token"

slack_channel = "#devops"

Step 5: Deploy Infrastructure

cd terraform



# Initialize Terraform

terraform init



# Review the plan (should show ~25 resources)

terraform plan



# Deploy the infrastructure

terraform apply

Important: Save the outputs displayed after successful deployment:

api_gateway_url = "https://<your-api-gateway-id>.execute-api.<region>.amazonaws.com/prod/trigger"

bastion_public_ip = "<bastion-public-ip>"

jenkins_url = "http://<internal-alb-url>"

Step 6: Update Slack Configuration

1. Go back to your Slack app settings


2. Update the slash command Request URL with the API Gateway URL from Step 5


3. Save the changes

Step 7: Configure Jenkins

Access Jenkins

Set up SSH tunnel through the bastion host:

# SSH to bastion host with port forwarding

ssh -i your-key.pem -L 8080:<jenkins_private_ip>:8080 ec2-user@<bastion_public_ip> -N



# In another terminal, access Jenkins via localhost

# Open browser: http://localhost:8080

Initial Jenkins Setup

Wait for Jenkins initialization (5-10 minutes), then:

1. 
   Get the initial admin password:
   

   # SSH to Jenkins instance

   ssh -i your-key.pem ec2-user@<bastion_public_ip>

   ssh ec2-user@<jenkins_private_ip>



   # Get initial password

   sudo cat /var/lib/jenkins/secrets/initialAdminPassword

1. 
   Access Jenkins at http://localhost:8080
   


2. Enter the initial admin password


3. Install suggested plugins


4. 
   Create admin user:
   
   
   
   
   • Username: admin
     
   
   
   • Password: <your-secure-password> (use a strong password)
   
   
   • Full name: Administrator
     
   
   
   • Email: [email protected]
     
   
   
   
   

Install Slack Plugin

1. 
   Go to: Manage Jenkins → Manage Plugins → Available


2. 
   Search for: "Slack Notification"


3. 
   Check the box and click "Install without restart"
   


4. 
   Wait for installation to complete


5. 
   Go to: Manage Jenkins → Configure System


6. 
   Find "Slack" section:
   
   
   
   
   • 
     Workspace: Your workspace name
   
   
   • 
     Default Channel: #devops (or your preferred channel)
   
   
   • 
     Integration Token: Paste your Bot User OAuth Token (xoxb-...)
   
   
   
   


7. 
   Click "Test Connection" - should show "Success"


8. Click "Save"

Create Pipeline Job

1. Click "New Item"


2. 
   Enter item name: run_test
   


3. 
   Select "Pipeline" and click "OK"
   


4. 
   In Pipeline section, select "Pipeline script"
   


5. 
   Copy the content from jenkins/Jenkinsfile into the script area


6. Click "Save"

Step 8: Update SSM Parameters

Verify the Jenkins URL in SSM Parameter Store:

aws ssm get-parameter --name "/jenkins-slack-demo/jenkins_url" --region us-east-1

If needed, update it with the correct internal ALB URL:

aws ssm put-parameter \

  --name "/jenkins-slack-demo/jenkins_url" \

  --value "http://<internal-alb-url>" \

  --overwrite \

  --region us-east-1

Verification and Testing

Test 1: Verify Infrastructure

# Check EC2 instances

aws ec2 describe-instances --filters "Name=tag:Name,Values=jenkins-slack-demo-*" --region us-east-1



# Check Lambda functions

aws lambda list-functions --region us-east-1



# Check API Gateway

curl -X POST https://your-api-gateway-url/prod/trigger

Test 2: Verify Lambda Function

Test the Lambda function directly:

# Test Lambda function

aws lambda invoke \

  --function-name jenkins-slack-demo-jenkins-trigger \

  --payload '{}' \

  /tmp/lambda-response.json && cat /tmp/lambda-response.json

Test 3: Verify Jenkins Connectivity

Test Jenkins accessibility from Lambda:

# SSH to Jenkins and verify it's running

ssh -i your-key.pem ec2-user@<bastion_public_ip>

ssh ec2-user@<jenkins_private_ip>



# Check Jenkins status

sudo systemctl status jenkins

curl -u admin:<your-secure-password> http://localhost:8080/api/json

Test 4: Test Slack Command

In any Slack channel where your app is installed, type:

/run_test

Expected behavior:

• Immediate response in Slack: "Job triggered successfully!"


• Jenkins job appears in the dashboard


• Job runs with default parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)

Test 5: Verify Jenkins Job Execution

1. 
   Check Jenkins dashboard for the run_test job


2. 
   Verify job parameters (ENVIRONMENT=dev, TEST_SUITE=smoke)


3. 
   Check job logs for successful execution


4. 
   Confirm Slack receives success/failure notifications

Test 6: End-to-End Test with Parameters

Test with custom parameters:

/run_test environment=staging test_suite=regression

Monitor the complete flow:

1. 
   Slack → API Gateway → Lambda


2. 
   Lambda → Jenkins (job trigger with parameters)


3. 
   Jenkins → Slack (status notification)

Key Code Components

Lambda Function (Go)

The Lambda function handles the core integration logic:

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {

    // Get Jenkins URL and credentials from SSM

    jenkinsURL, err := getSSMParameter(ctx, jenkinsURLParam, false)

    credsJSON, err := getSSMParameter(ctx, jenkinsCredsParam, true)



    // Get CSRF crumb for Jenkins security

    crumbField, crumb, err := getJenkinsCrumb(client, jenkinsURL, creds.Username, creds.Password)



    // Trigger Jenkins job

    jobURL := fmt.Sprintf("%s/job/run_test/buildWithParameters", jenkinsURL)

    data := url.Values{}

    data.Set("ENVIRONMENT", "dev")

    data.Set("TEST_SUITE", "smoke")



    // Make authenticated request with crumb

    req.Header.Set("Jenkins-Crumb", crumb)

    req.SetBasicAuth(creds.Username, creds.Password)



    return events.APIGatewayProxyResponse{

        StatusCode: 200,

        Body:       `{"message": "Job triggered! Status: 201"}`,

    }, nil

}

Jenkins Pipeline

The Jenkinsfile defines the test pipeline:

pipeline {

    agent any



    parameters {

        string(name: 'ENVIRONMENT', defaultValue: 'dev', description: 'Environment to run tests against')

        string(name: 'TEST_SUITE', defaultValue: 'smoke', description: 'Test suite to run')

    }



    stages {

        stage('Prepare') {

            steps {

                echo "Preparing to run tests in ${params.ENVIRONMENT} environment"

                slackSend(color: '#FFFF00', message: "STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'")

            }

        }



        stage('Run Tests') {

            steps {

                echo "Running ${params.TEST_SUITE} tests against ${params.ENVIRONMENT}"

                // Your actual test commands here

            }

        }

    }



    post {

        success {

            slackSend(color: '#00FF00', message: "SUCCESS: Job completed successfully")

        }

        failure {

            slackSend(color: '#FF0000', message: "FAILED: Job failed")

        }

    }

}

What's Next?

Congratulations! You've successfully built the foundation of a secure Jenkins-Slack integration. At this point, you should have:

✅ Working Infrastructure: VPC, Jenkins, Lambda, API Gateway


✅ Basic Integration: Slack can trigger Jenkins jobs


✅ Security: Private Jenkins with secure access patterns


✅ Monitoring: CloudWatch logs for observability

Coming Up in Part 2

In Part 2 of this series, we'll dive deep into:

🔧 Real-World Troubleshooting: Common issues you'll encounter and battle-tested solutions


🛠️ Production Enhancements: Security improvements, monitoring, and scaling considerations


🚨 Critical Fixes: Jenkins authentication issues, CSRF token problems, and Slack integration challenges


📊 Advanced Features: Parameter parsing, job status callbacks, and multi-environment support

Key Takeaways

This architecture demonstrates several important DevOps and AWS best practices:

✅ Security First: Jenkins is never exposed to the internet, following zero-trust principles

✅ Infrastructure as Code: Everything is defined in Terraform, making it reproducible and version-controlled

✅ Serverless Integration: Lambda provides cost-effective, scalable integration without managing servers

✅ Secret Management: SSM Parameter Store eliminates hardcoded credentials

✅ Network Isolation: VPC with public/private subnets provides proper network segmentation

✅ Observability: CloudWatch logs provide visibility into system behavior

Ready to tackle the real-world challenges? Check out Part 2 where we'll solve the tricky issues that make or break production deployments!

Have questions or suggestions for improvements? Feel free to reach out or contribute to the project repository!

Cover image by @dlxmedia.hu from unsplash