DevOps: Jenkins & AWS Series, Part 4: Jenkins Notifications and AWS SNS Integration (Terraform):- A Step-by-Step Guide

8 min readApr 15, 2023

Jenkins and AWS SNS Integration: A Step-by-Step Guide

Introduction

Welcome to the fourth part of our DevOps: Jenkins & AWS series! In this installment, we’ll focus on integrating Jenkins with Amazon Simple Notification Service (SNS) to send notifications through various channels, such as email, SMS, or even AWS Lambda functions. This step-by-step guide will provide an in-depth introduction to the benefits of using AWS SNS for Jenkins notifications, as well as an overview of the integration process.

Amazon SNS is a powerful and flexible messaging service that allows you to decouple microservices, distributed systems, and serverless applications. By integrating Jenkins with AWS SNS, you can streamline your notification workflow and easily manage the distribution of messages to various endpoints. This integration enables you to keep your team informed about the status of builds and deployments, fostering better collaboration and communication.

In the subsequent sections of this guide, we’ll provide a detailed walkthrough of the process to set up Jenkins and AWS SNS integration. We’ll cover the creation of an SNS topic, setting up IAM permissions, installing and configuring the necessary Jenkins plugins, and finally, adding the AWS SNS notification step to your Jenkins pipeline.

By following this guide, you’ll be able to harness the power of AWS SNS in your Jenkins pipelines and improve your team’s responsiveness to build and deployment events. Stay tuned for the next parts of this series, where we’ll explore even more ways to optimize your DevOps workflow using Jenkins and AWS services.

Prerequisites
Create an SNS Topic
Create an IAM User with SNS Permissions
Install and Configure the AWS SDK Plugin in Jenkins
Add the AWS SNS Notification Step to Your Jenkins Pipeline
Testing AWS SNS Integration & Output
Conclusion

Prerequisites

Before we begin, make sure you have the following:

A Jenkins instance up and running
An AWS account with access to SNS and IAM

Create an SNS Topic

a. Log in to the AWS Management Console and navigate to the SNS service.

b. Click “Topics” on the left sidebar and then click the “Create topic” button.

c. Enter a name and display name for your topic, then click “Create Topic.”

Create an IAM User with SNS Permissions

a. Navigate to the IAM service in the AWS Management Console.

b. Click “Users” on the left sidebar and then click the “Add user” button.

c. Enter a user name, select “Programmatic access” for the access type, and click “Next: Permissions.”

d. Click “Attach existing policies directly” and search for “AmazonSNSFullAccess” in the policy search box. Check the box next to the policy and click “Next: Tags.”

e. Optionally, add tags for the user and click “Next: Review.”

f. Review the user details and click “Create user.”

g. Download the CSV containing the user’s Access Key ID and Secret Access Key. You’ll need these for configuring Jenkins.

Terraform Automation

I’ll provide you with a Terraform script that follows best practices, including the use of variables and outputs. This script will create the required AWS resources for Jenkins and AWS SNS integration:

variables.tf:

variable "region" {
  description = "The AWS region to deploy resources in."
  default     = "us-east-1"
}
variable "sns_topic_name" {
  description = "The name of the SNS topic to create."
  default     = "jenkins-notifications"
}
variable "email_address" {
  description = "The email address to subscribe to the SNS topic."
  default     = "varunmanik1@gmailc.om"
}

main.tf:

provider "aws" {
  region = var.region
}
resource "aws_sns_topic" "jenkins_notifications" {
  name = var.sns_topic_name
}
resource "aws_sns_topic_subscription" "jenkins_email_subscription" {
  topic_arn = aws_sns_topic.jenkins_notifications.arn
  protocol  = "email"
  endpoint  = var.email_address
}

outputs.tf:

output "sns_topic_arn" {
  description = "The ARN of the created SNS topic."
  value       = aws_sns_topic.jenkins_notifications.arn
}

This Terraform script is now modular and follows best practices:

Variables are defined in a separate variables.tf file, making it easy to customize the script for different environments or projects.
The main.tf file contains the AWS resources, and the provider is configured using the region variable.
The outputs.tf file defines the output for the SNS topic ARN, which can be used as an input for other Terraform modules or displayed in the console after applying the configuration.

To use this Terraform script, make sure to run terraform init, followed by terraform apply.

Install and Configure the AWS SDK Plugin in Jenkins

a. Log in to your Jenkins instance and go to “Manage Jenkins” > “Manage Plugins.”

b. Click the “Available” tab and search for “AWS SDK.” Install the plugin and restart Jenkins if prompted.

c. Go to “Manage Jenkins” > “Configure System.”

d. Scroll down to the “AWS SDK” section and click the “Add” button.

e. Enter the Access Key ID and Secret Access Key you downloaded earlier for the IAM user. Optionally, set a name for the profile.

f. Save the configuration.

Add the AWS SNS Notification Step to Your Jenkins Pipeline

a. Create a new pipeline job or modify an existing one.

b. In the pipeline script section, add the following code:

import groovy.json.JsonOutput

pipeline {
    agent any

    environment {
        AWS_REGION = 'us-east-1'
        AWS_ACCOUNT_ID = '1234567890'
        SNS_TOPIC_NAME = 'jenkins-notifications'
    }

    stages {
        stage('Hello') {
            steps { 
                sh 'echo "hello-world"'
            }}
        stage('Send SNS Notification') {
            steps {
                withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'aws-sns', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
                    script {
                        def snsTopicArn = "arn:aws:sns:${AWS_REGION}:${AWS_ACCOUNT_ID}:${SNS_TOPIC_NAME}"
                        def message = "Jenkins job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' has been executed."
                        def messageJson = JsonOutput.toJson([default: message])

                        writeFile file: 'message.json', text: messageJson
                        sh "aws sns publish --region ${AWS_REGION} --topic-arn ${snsTopicArn} --message file://message.json"
                        sh "rm -f message.json"
                    }
                }
            }
        }
    }
}

I’ll break down the pipeline script into sections and explain each line:

Import the JsonOutput class from the groovy.json package, which is used to convert a map into a JSON string:

import groovy.json.JsonOutput

Define the Jenkins pipeline with a single agent:

pipeline {
    agent any

Set environment variables for the AWS region, AWS account ID, and SNS topic name:

environment {
        AWS_REGION = 'us-east-1'
        AWS_ACCOUNT_ID = '1234567890'
        SNS_TOPIC_NAME = 'jenkins-notifications'
    }

Define the stages of the pipeline. In this case, there’s only one stage, “Send SNS Notification”:

stages {
        stage('Send SNS Notification') {

Use the withCredentials step to securely provide the AWS access key and secret key to the pipeline script:

steps {
                withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', 
                                   accessKeyVariable: 'AWS_ACCESS_KEY_ID', 
                                   credentialsId: 'aws-sns', 
                                   secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {

Construct the SNS topic ARN using the environment variables defined earlier:

script {
def snsTopicArn = "arn:aws:sns:${AWS_REGION}:${AWS_ACCOUNT_ID}:${SNS_TOPIC_NAME}"

Create the message string using the environment variables env.JOB_NAME and env.BUILD_NUMBER:

def message = "Jenkins job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' has been executed."

Convert the message to a JSON string using the JsonOutput.toJson() method:

def messageJson = JsonOutput.toJson([default: message])

Write the messageJson content to a temporary file named message.json:

writeFile file: 'message.json', text: messageJson

Use the sh step to execute the AWS CLI command to send the SNS notification. The file:// prefix is used to pass the contents of the message.json file as the message:

sh "aws sns publish --region ${AWS_REGION} --topic-arn ${snsTopicArn} --message file://message.json"

Remove the temporary file message.json:

sh "rm -f message.json"

Close the script, withCredentials, steps, stage, stages, and pipeline blocks:

This pipeline script sets up a single-stage pipeline that sends an SNS notification using the AWS CLI. The pipeline uses environment variables and credentials to securely interact with AWS resources.

Configure Jenkins Job

Testing Your AWS SNS Integration & Output

To see the Jenkins output, you would need to run the pipeline script in your own Jenkins environment. After the pipeline completes its execution, the output will be displayed in the console output of the build.

To view the console output of a Jenkins build, follow these steps:

Navigate to your Jenkins instance in your web browser.
Go to the pipeline job that you have configured with the pipeline script.
Click on the build number in the build history section (usually on the left side of the page).
Click on “Console Output” in the build details page.

The console output will display the logs generated during the pipeline execution, including the result of the AWS SNS notification step.

BLUE OCEAN Plugin

Normal/ classic Output

Email Notification:

Conclusion

In conclusion, the “DevOps: Jenkins & AWS Series, Part 4: Jenkins Notifications and AWS SNS Integration — A Step-by-Step Guide” provides a comprehensive walkthrough of integrating Jenkins and AWS Simple Notification Service (SNS) to enhance your DevOps pipeline. By following the guide, you can now configure Jenkins to send notifications using AWS SNS, enabling better communication among your team members and keeping them informed about the status of your builds.

Throughout this guide, we covered setting up the required AWS resources using Terraform, configuring the Jenkins pipeline with the appropriate credentials, and modifying the pipeline script to send SNS notifications. This integration helps streamline your DevOps process, ensuring that stakeholders are informed in real-time about the progress of your projects.

As you continue to explore the world of DevOps and leverage the capabilities of Jenkins and AWS, this guide serves as a valuable resource to enable a more robust and efficient development and deployment process. By integrating Jenkins notifications with AWS SNS, you can take advantage of improved communication, quicker response times, and overall better team coordination.

Keep experimenting and optimizing your workflows to harness the full potential of Jenkins and AWS services in your DevOps journey.