Open in app

Sign in

Write

Sign in

DevOps CICD Pipeline with Ansible, Jenkins, GitHub & AWS

Varun Kumar Manik
12 min readJun 14, 2020

DevOps CI/CD pipeline for LEMP (Linux, Nginx, Mysql and PHP ) installation and Code deployment from DevOps tools.(GitHub + Jenkins + GitHub+ Ansible + AWS)

Hi Folks,

Agenda: Step by step DevOps tutorial, for CI/CD pipeline for AWS EC2 servers. Which includes:

  1. EC2 instance creation in AWS console & instance connectivity via terminal
  2. Ansible installation, Dynamic inventory setup for EC2 servers and connectivity to web servers via ansible.
  3. Git installation, Github repo creation, Github Jenkins webhook integration
  4. Jenkins installation, Plugin setup, Job Creation and Deployment pipeline
  5. Code Deployment → GitHub push → Trigger Jenkins Job → ansible deployment → Web Server code

So without making any delay let’s start:

Step 1: Log in to the AWS Console and Launching an EC2 Instance

Step 2: Launch an EC2 Instance

Step 2: Adding an Elastic IP to the server.

VM is up and running with elastic IP

Note: Repeat Step 2 for spinning up another webserver

Step 3: Connectivity to the EC2 Instance

Change pem file permission

chmod 400  key.pemssh -i key.pem ubuntu@52.224.12.254

Step 4: Installing Ansible

Run the below command to install Ansible

sudo apt install software-properties-common -ysudo apt-add-repository ppa:ansible/ansiblesudo apt update -ysudo apt install python ansible -yansible --version

Ansible is installed now.

Step 5: Installing AWS CLI

Run below command to install AWS CLI

sudo apt install awscli -y

Configuring AWS CLI

Create an IAM role with Jenkins instance, with admin permission (for testing)

and configure the AWS region us-east-1 with “aws configure” command

aws configure

5.1 Testing AWS CLI

Get all instances metadata in region us-east-1

aws ec2 describe-instances

Get Instance Id of all instances in us-east-1 region

aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text

Step 6: Configuring Ansible for Dynamic Inventory AWS

Install python-boto library for version 2 and 3 via given commands

Change to the directory to ansible_testing

Download ec2.ini and ec.py file in you terminal

sudo apt install python-boto python-boto3 -ymkdir ansible_testingcd ansible_testing/wget https://raw.githubusercontent.com/ansible/ansible/stable-2.9/contrib/inventory/ec2.iniwget https://raw.githubusercontent.com/ansible/ansible/stable-1.9/plugins/inventory/ec2.pychmod +x ec2.py

Check if ec2.py is listing our instances properly

python ec2.py

Check ansible connectivity in both servers

Ping-Pong command from Ansible CLI (Jenkins server to the webserver)

ansible -i ec2.py -m ping security_group_web_server_dmz

While running the above command we can get the below error in red color

nano key.pemchmod 400 key.pemansible -i ec2.py -m ping security_group_web_server_dmz --private-key key.pem

To remove the error please change the pem file permission and add parameter — private-key

Now Ansible connectivity between both instance are successful …!!!!

Step 7: Github Settings

Go to the GitHub account and download the repo (medium_blog_cicd) to you your Jenkins server terminal.

Copy ansible_testing directory to medium_blog_cicd directory

Then add all file in git via typing “git add .” command and commit the file with a given message.

Pushing the directory ansible_testing with all files into Github

git clone https://github.com/manikcloud/medium_blog_cicd.gitcp -r ansible_testing/ medium_blog_cicd/cd medium_blog_cicd/git add .git commit -m 'Adding ansible dynamic inventory'git push origin master

Github console with all files

Step 8: Installing Jenkins

Please run the below command one by one in terminal

sudo apt-get install openjdk-8-jdk openjdk-8-jre -ywget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -sudo -isudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'sudo apt update ysudo apt update -ysudo apt install jenkins -ysudo systemctl start jenkinssudo systemctl status jenkins

open your browser and type the IP address with port 8080

http://54.224.12.254:8080

It will ask for a password. You can find the password at below location/ command:

cat /var/lib/jenkins/secrets/initialAdminPassword

Copy and Paste the password and click continue

Click on install suggested plugins

Once plugins are installed.

Click on Save and Finish.

Now Jenkins is installed and ready for a new Job.

Creating a New Job in Jenkins: Testing Ansible connectivity from Jenkins terminal to the Web Server

Enter the name of the job: ansible_testing

Set the source code management to your git hub repository, and provide the URL in the text box.

Choose the build option as: Execute shell and write the below command and save the job.

Click on the Build option. It will run the Job, and you can see the output. Finished: SUCCESS.

Installing LEMP using Jenkins, Ansible and Shell Scripting (Wordpress installation)

#!/bin/bash
# Author - Akshay Gupta
# Version - 2.0.0
# Description - Installs and Configures LEMP Stack and then Wordpress for Port 80 only, in just 1 click.
# In Version 2.0.0 -> Manual/Automatic installation of Wordpress on LEMP Stack is available
# Usage -
#
#	bash Wordpress_Installation_on_LEMP_Stack_in_one_click.sh
#	bash Wordpress_Installation_on_LEMP_Stack_in_one_click.sh 1
#	bash Wordpress_Installation_on_LEMP_Stack_in_one_click.sh 2
#

installations() {
	if [ $(date | awk '{print $2 $3}') != $(ls -al /var/lib/apt/periodic/update-success-stamp | awk '{print $6 $7}') ]
	then
		apt update -y
		apt upgrade -y
		apt dist-upgrade -y
	fi
	apt install -y software-properties-common;
	add-apt-repository -y ppa:ondrej/php;
	apt update -y;
	apt install nginx -y;
	apt install php7.1-mcrypt php7.1-intl php7.1-curl php7.1-xsl php7.1-mbstring php7.1-xsl php7.1-zip php7.1-soap php7.1-gd php7.1-bcmath php7.1-mysql php7.1-fpm -y;
	apt install mysql-server mysql-client -y;
	apt install composer -y;
}

conf_php() {
sed -i 's/memory_limit = -1/memory_limit = 2G/g' /etc/php/7.1/cli/php.ini
sed -i 's/max_execution_time = 30/max_execution_time = 1800/g' /etc/php/7.1/cli/php.ini
sed -i 's/zlib.output_compression = Off/zlib.output_compression = On/g' /etc/php/7.1/cli/php.ini

cat /etc/php/7.1/cli/php.ini | grep memory_limit
cat /etc/php/7.1/cli/php.ini | grep max_execution_time
cat /etc/php/7.1/cli/php.ini | grep 'zlib.output_compression ='

sed -i 's/memory_limit = 128M/memory_limit = 2G/g' /etc/php/7.1/fpm/php.ini
sed -i 's/max_execution_time = 30/max_execution_time = 1800/g' /etc/php/7.1/fpm/php.ini
sed -i 's/zlib.output_compression = Off/zlib.output_compression = On/g' /etc/php/7.1/fpm/php.ini

cat /etc/php/7.1/fpm/php.ini | grep memory_limit
cat /etc/php/7.1/fpm/php.ini | grep max_execution_time
cat /etc/php/7.1/fpm/php.ini | grep 'zlib.output_compression ='
service php7.1-fpm restart
}

conf_nginx() {

mkdir -p $root_dir;
if [[ ! -e /etc/nginx/php_loc.conf ]]; then
cat >> /etc/nginx/php_loc.conf << PHP_BLOCK
#	pass PHP scripts to FastCGI server
	location ~ \.php$ {
	root           $root_dir;
	fastcgi_index  index.php;
	try_files \$uri =404;
#	With php-fpm (or other unix sockets):
	fastcgi_pass fastcgi_backend;
	include        fastcgi_params;
	fastcgi_param  SCRIPT_FILENAME  \$document_root\$fastcgi_script_name;
}
PHP_BLOCK
fi

if [[ ! -e /etc/nginx/sites-available/$nginx_conf ]]; then
	cp /etc/nginx/sites-available/default /etc/nginx/sites-available/$nginx_conf;

	# converting /dir1/dir2/ to \/dir1\/dir2\/, so that it can be added properly in /etc/nginx/sites-available/website conf file
	root_dir2=${root_dir////\\/}

cat >> /etc/nginx/sites-available/$nginx_conf.tmp << UPSTREAM
upstream fastcgi_backend {
	server  unix:/run/php/php7.1-fpm.sock;
}
UPSTREAM
	cat /etc/nginx/sites-available/$nginx_conf.tmp /etc/nginx/sites-available/$nginx_conf > /etc/nginx/sites-available/$nginx_conf.new;
	mv /etc/nginx/sites-available/$nginx_conf.new /etc/nginx/sites-available/$nginx_conf;
	rm -rf /etc/nginx/sites-available/$nginx_conf.tmp;
	sed -i "s/fastcgi_pass 127.0.0.1:9000/fastcgi_pass 127.0.0.1:9000;\
	\ninclude \/etc\/nginx\/php_loc.conf/g" /etc/nginx/sites-available/$nginx_conf;
	sed -i "s/root \/var\/www\/html/root $root_dir2/g" /etc/nginx/sites-available/$nginx_conf;
	sed -i "s/index index.html index.htm index.nginx-debian.html/index index.php index.html index.htm index.nginx-debian.html/g" /etc/nginx/sites-available/$nginx_conf;
	sed -i "s/\include \/etc\/nginx\/sites-enabled\//include \/etc\/nginx\/sites-enabled\/$nginx_conf;\
	\n#/g" /etc/nginx/nginx.conf;
fi
if [[ ! -e /etc/nginx/sites-enabled/$nginx_conf ]]; then
	ln -s /etc/nginx/sites-available/$nginx_conf /etc/nginx/sites-enabled/;
fi
if [[ ! -e $root_dir/test.php ]]; then
	echo '<?php phpinfo(); ?>' > $root_dir/test.php;
fi
service nginx restart
}

conf_db() {

	mysql -h $host -u $rt_user --password=$rt_pw -e "
	CREATE DATABASE $db_name;
	CREATE USER '$user_name'@'%' IDENTIFIED BY '$pw';
	GRANT ALL PRIVILEGES ON $db_name.* TO '$user_name'@'%';
	FLUSH PRIVILEGES;"
}

conf_wp() {
	cd /root/
if [[ ! -e latest.tar.gz ]]; then
	wget http://wordpress.org/latest.tar.gz -O latest.tar.gz
fi
if [[ ! -d wordpress ]]; then
	tar xzf latest.tar.gz
fi

	cd wordpress
if [[ ! -e wp-config.php ]]; then
	cp wp-config-sample.php wp-config.php
	sed -i "s/define( 'DB_NAME', 'database_name_here' )/define( 'DB_NAME', '$db_name' )/g" wp-config.php
	sed -i "s/define( 'DB_USER', 'username_here' )/define( 'DB_USER', '$user_name' )/g" wp-config.php
	sed -i "s/define( 'DB_PASSWORD', 'password_here' )/define( 'DB_PASSWORD', '$pw' )/g" wp-config.php
	sed -i "s/define( 'DB_HOST', 'localhost' )/define( 'DB_PASSWORD', '$host' )/g" wp-config.php
	
	# To give permissions to wp-content to install plugins/themes etc
	echo "define('FS_METHOD', 'direct');" >> wp-config.php
fi
	cd /root/
	cp wordpress/* $root_dir -r
	mkdir -p $root_dir/wp-content/uploads
	chown -R www-data:www-data $root_dir/*
	find $root_dir -type d -exec chmod 755 {} \;
	find $root_dir -type f -exec chmod 644 {} \;
	rm -rf /etc/nginx/sites-enabled/$nginx_conf;
	ln -s /etc/nginx/sites-available/$nginx_conf /etc/nginx/sites-enabled/;
	service nginx restart;
}

echo "1. To perform all steps 01 to 05 with CUSTOM inputs. (without any reboot)."
echo "2. To perform all steps 01 to 05 quietly with DEFAULT inputs and all log in /root/wordpress_lemp_stack_setup.log (without any reboot)."
echo -e "\nDefault Choice is 2."

if [ -z $1 ]; then
	read choice
fi

if [ -z $choice ]; then
	choice="$1"
fi

case $choice in
1)
clear
echo "=====================START=========================="				> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Updates, Upgrades and Distribution Upgrades"					>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Updating, Upgrading and Installing Distribution Upgrades"
installations										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring php.ini for nginx and Wordpress"					>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring php.ini for nginx and Wordpress"
conf_php										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring nginx.conf and sites-available/website for php and Wordpress"		>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring nginx.conf and sites-available/website for php and Wordpress"

echo "Enter location of www-data root directory. . (Default: /var/www/html/website)"
read root_dir
if [ -z $root_dir ]; then
	root_dir='/var/www/html/website'
fi
echo "Enter name (only name not location) of configuration file under /etc/nginx/sites-available/. (Default: website)"
read nginx_conf

if [ -z $nginx_conf ]; then
	nginx_conf='website'
fi

conf_nginx										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Creating DB and User"								>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Creating DB and User"

echo "Enter host IP/Endpoint/DNS: (Default: localhost)"
read host
if [ -z $host ]; then
	host='localhost'
fi
echo "Enter Master username to make connection to MySQL: (Default: root)"
read rt_user
if [ -z $rt_user ]; then
	rt_user='root'
fi
echo "Enter Password of Master username to make connection to MySQL: (Default: NULL/EMPTY)"
read rt_pw
if [ -z $rt_pw ]; then
	rt_pw=''
fi
echo 'Enter name of Database you want to be created: (Default: websiteDB)'
read db_name
if [ -z $db_name ]; then
	db_name='websiteDB'
fi
echo 'Enter name of User you want to be created: (Default: admin)'
read user_name
if [ -z $user_name ]; then
	user_name='admin'
fi
echo "Enter Password for $user_name: (Default: password)"
read pw
if [ -z $pw ]; then
	pw='password'
fi

conf_db											>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Downloading and setting up latest Wordpress version"				>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Downloading and setting up latest Wordpress version"
conf_wp											>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "======================END==========================="				>> /root/wordpress_lemp_stack_setup.log  2>&1

echo "root Directory:					$root_dir"
echo "Website's nginx Configuration File:		$nginx_conf"
echo "Database Name: 					$db_name"
echo "Username: 					$user_name"
echo "Password: 					$pw"
echo "Database Host: 					$host"

exit
;;
2)
clear

root_dir='/var/www/html/website'
nginx_conf='website'
db_name='websiteDB'
user_name='admin'
pw='password'
host='localhost'
rt_user='root'

echo "=====================START=========================="				> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Updates, Upgrades and Distribution Upgrades"					>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Updating, Upgrading and Installing Distribution Upgrades"
installations										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring php.ini for nginx and Wordpress"					>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring php.ini for nginx and Wordpress"
conf_php										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring nginx.conf and sites-available/website for php and Wordpress"		>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Configuring nginx.conf and sites-available/website for php and Wordpress"
conf_nginx										>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Creating DB and User"								>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Creating DB and User"
conf_db											>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Downloading and setting up latest Wordpress version"				>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "Downloading and setting up latest Wordpress version"
conf_wp											>> /root/wordpress_lemp_stack_setup.log  2>&1
echo "======================END==========================="				>> /root/wordpress_lemp_stack_setup.log  2>&1

echo "root Directory:					$root_dir"
echo "Website's nginx Configuration File:		$nginx_conf"
echo "Database Name: 					$db_name"
echo "Username: 					$user_name"
echo "Password: 					$pw"
echo "Database Host: 					$host"


exit
;;
*)
loc="$(readlink -f ${BASH_SOURCE[0]})"
echo "Running $loc with Choice 2"
read -p "Press Enter to continue" </dev/tty
bash $loc 2
;;
esac

Ansible Script used

https://raw.githubusercontent.com/manikcloud/medium_blog_cicd/master/lemp_installation/lemp_installation.yml

Run the newly created job, you can see the output screen as success.

Copy and paste the IP in the browser and install Wordpress

Integrating Github Webhook with Jenkins (No Extra Plugins are required)

Goto your repository and click on webhook section

and add your Jenkins URL ended with /github-webhook/

In Jenkins job go and select the Github hook trigger GITscm polling,

It will trigger the job whenever the user runs a push command on the master branch of your repository.

Now I have made some changes in footer.php for Theme TwentyTwenty in

After making a change in the previous step, I run the git push command. We already had integrated Jenkin to this GitHub repository. So the deployment job runs automatically.

Jenkins output page, showing a successful message.

Now we can refresh the page and see the changes at the bottom.

“Powered by WordPress Varun CI/CD” changed to “Powered by WordPress Varun CI/CD for Medium Story”.

Conclusion:

In this blog I have successfully demonstrated the below points:

  • How to launch AWS EC2 machine and connectivity with the terminal.
  • How to install & Setup Jenkins, Plugin setup, Job Creation and Deployment pipeline
  • How to do Ansible installation, Dynamic inventory setup for EC2 servers and connectivity to web servers via ansible
  • How to install and configure AWS CLI
  • How to do Git installation, Github repo creation, Github Jenkins webhook integration

For more info please connect & Follow me on:

LinkedIn: https://www.linkedin.com/in/vkmanik/

Email: varunmanik1@gmail.com

Facebook: https://www.facebook.com/cloudvirtualization/

References —

  1. https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-18-04
  2. https://www.shellhacks.com/install-jenkins-ubuntu-centos/
  3. https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html
DevOps
Jenkins Pipeline
Ansible
Github
WordPress

--

--

Varun Kumar Manik

Written by Varun Kumar Manik

346 Followers

AWS APN Ambassador | SME of DevOps DevSecOps | Cloud Architect & Trainer | Blogger | Youtuber |Chef

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams