Automated CI/CD Pipeline with Jenkins, Ansible, Apache, and AWS

Project Overview

The goal of this project is to set up an automated CI/CD pipeline for our web application. We'll use Jenkins, Ansible, Apache, and AWS to accomplish this. The pipeline is configured to automatically create, compile, analyse, and deploy new code to an AWS EC2 instance as it is added to the GitHub repository by the developer.

Prerequisites

To begin working on this project, you must meet certain requirements. These include having an active account on GitHub, familiarity with AWS EC2, and a solid understanding of GitHub, Jenkins, Ansible, and Apache webserver.

Steps to cover:

Ensure all four servers (developer, Jenkins, Ansible, and web server) are prepared and configured.
Create a new repository in your GitHub account and set up integration with Jenkins.
Generate an SSH key and establish passwordless connectivity between the servers.
Set the root password for connectivity on all servers and write a playbook on the Ansible server.
Input the web server's public IP address in the inventory file of Ansible.

Install the "Publish Over SSH" plugin in Jenkins and create a job.
Configure both Jenkins and Ansible servers accordingly.

Install the Apache package and start the service on the web server.

Server Preparation and Configuration

Creating EC2 Instances

We will use command scripts to easily and quickly set up Jenkins, Ansible, and a web server on all the necessary servers, which will save us time and effort. However, it's also possible to manually install all the required packages on EC2. We will be using the Ubuntu operating system for this project.

Setting up Jenkins Server

Click on "launch Instances"

Verify the number of instances and the operating system, then rename your EC2 instance to correlate with the name of your associated server.

For now, create a common key pair and one security group for all servers.

Generally, there is only one rule present for port 22 for SSH. More rules must be saved, and more rules must be added.

Port 8080 for Jenkins

Port 80 is used to display our web application.

Once you have saved the rules, navigate to the bottom where you were creating the EC2 instance. Click on "Advanced Details," and at the bottom, you will find a field named "User Data." In this field, add the Jenkins script provided below and then launch the instance.

User data is generally used to run commands or a command script to run when you launch your instance.


#!/bin/bash

#This script sets up the Jenkins server
#Switch to the superuser
sudo su

#Set the hostname for Jenkins
hostnamectl set-hostname jenkins

#Update the package list
apt update

#Install OpenJDK 11
apt install openjdk-11-jdk -y

#Import the Jenkins GPG key
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null

#Add the Jenkins repository to the system's list of sources
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

#Update the package list again
apt update

#Install Jenkins
apt install jenkins -y

#Start Jenkins and enable it to start on boot
systemctl start jenkins
systemctl enable jenkins

#Allow traffic through port 8080 in the firewall
ufw allow 8080

After waiting for a few minutes, you can try accessing your server's public IP in a web browser using port 8080. If everything is set up correctly, Jenkins should be accessible to you, as shown below.

Follow the steps mentioned above to set up Ansible and the webserver. However, this time, make sure to change the script in the "User data" field to reflect the specific setup for Ansible or the webserver.

Additionally, create a Developer server that does not require any script. This VM will serve as a developer system.

Setting up Ansible Server

Below is a command script to set up the Ansible server:

#!/bin/bash

#This script sets up the Ansible server
#Switch to the superuser
sudo su

#Set the hostname for Ansible
hostnamectl set-hostname ansible

#Update the package list
apt update

#Install software-properties-common package
sudo apt install software-properties-common -y

#Add the Ansible repository to the system's list of sources
sudo add-apt-repository --yes --update ppa:ansible/ansible -y

#Update the package list again
apt update

#Install Ansible
apt install ansible -y

Setting up Apache Web Server

Below is a command script to set up the Apache webserver:

#!/bin/bash

#This script sets up the Apache server
#Switch to the superuser
sudo su

#Set the hostname for Apache
hostnamectl set-hostname apache

#Update the package list
apt update

#Install Apache
apt install apache2 -y

#Start Apache and enable it to start on boot
systemctl start apache2
systemctl enable apache2

At this point, your Developer, Jenkins, Ansible, and Webserver instances should be ready and set up according to the configuration steps performed.

Establishing SSH Connectivity

Login to all 3 servers (Jenkins, Ansible, Webserver) with key and set root password to all servers and make changes in /etc/ssh/sshd_config file like below

To change your password use:

passwd root

In AWS, PermitRootLogin is by default turned off. Thus we need it to establish passwordless connections between servers.

The following command must be used to edit the sshd_config file on all 3 servers (Jenkins, Ansible, and Webserver):

vi /etc/ssh/sshd_config

To enable PasswordAuthentication, locate the following line in the sshd_config file, and uncomment PermitRootLogin.

Make the changes shown below to all 3 servers (Jenkins, Ansible, and Webserver).

#PermitRootLogin prohibit-password
PasswordAuthentication no
#PermitEmptyPasswords no

PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yes

After making these adjustments in all 3 (Jenkins, Ansible, and Webserver), restart SSH with the following command:

systemctl restart sshd

Congratulations! You have successfully completed the necessary steps to establish passwordless connections.

Generating SSH Key and Passwordless Connectivity

ssh-keygen

Copy the public key to the remote machine, which will be our webserver

ssh-copy-id root@(webserver-public IP)

Adding Web Server's IP to Ansible Inventory

We must now add the webserver's public IP address to the Ansible server's Host file.

The host file in Ansible is a text file that contains a list of servers or hosts that Ansible manages. It defines the inventory of servers, including their IP addresses or DNS names, and any specific variables or groups that they belong to. It is the primary source of information for Ansible and is typically located in the /etc/ansible/hosts

vi /etc/ansible/hosts

If you want to group hosts together, you can create sections in the host file by enclosing the group name in square brackets. For example:

[webserver]
(webserver IP)

Save the host file and close the text editor. Check the Host IP connection with:

ansible webserver -m ping

The output shows that Ansible was able to successfully ping the host with IP address 54.86.41.158 and received a "pong" response, indicating that the host is reachable and responding to Ansible's requests.

Connect Jenkins and the Ansible server using a passwordless SSH connection.

# Generate an SSH key pair on the Jenkins machine
ssh-keygen

# Copy the public key to the Ansible machine
ssh-copy-id root@ansible-server-IP

# Verify the passwordless connection
ssh root@ansible-server-IP

Creating the GitHub Repository and Adding Code

Create an index.html file and a repository in GitHub, and then add code to the files, respectively.

Add "Hello World!" and commit the changes.

Logging into Jenkins and Creating a Project

The Jenkins interface is now operational on your Jenkins server. Open a web browser and type the following URL to access the Jenkins web interface:

http://your_server_ip_or_hostname:8080

Upon accessing the Jenkins web interface for the first time,

You will be prompted to enter the initial admin password, which can be found in the directory /var/lib/jenkins/secrets/initialAdminPassword

To view the password, use the following command:

cat /var/lib/jenkins/secrets/initialAdminPassword

Setting up Jenkins Project

Once you have unlocked Jenkins, click on "Install suggested plugins". Next, you can proceed to the "Create First Admin User" page and fill in the necessary details. Once all the required steps are completed, you will be directed to the Jenkins dashboard.

Click "New Items" on the Jenkins dashboard, enter a project name, choose "Freestyle project," and then click "OK."

Integrating Jenkins with GitHub

Select git for source code management Add the git repository URL to the repository URL field, and make sure the branch name in the Branch Specifier field matches the branch name in the GitHub repository. My branch will be the main branch in this project.

Configuring SSH Servers in Jenkins

Installing the "Publish Over SSH" Plugin

Install "Public over ssh plugin" to make ssh connections follow this path and to install the plugin

Dashboard -> Manage Plugins -> Available Plugins -> Publish Over SSH

As a post-build activity, the Publish Over SSH plugin in Jenkins allows you to transfer files over SSH to a remote server. Install the plugin, set up the SSH server, specify the source files to transfer, test the configuration, and execute the build.

When the installation is complete and no jobs are running, click on Install without and Restart Jenkins.

Adding Jenkins and Ansible Servers

We need to add two servers for ssh in Jenkins i.e. Jenkins and Ansible server.

To access and configure SSH servers in Jenkins, follow these steps:

Dashboard -> Manage Jenkins -> Configure System -> SSH servers

Add our server name, "Jenkins", hostname , username and password in advance option. and test the configuration below

Repeat these steps and this time add an ansible server and save the modifications.

Adding Webhook in GitHub

First, generate a secret key in Jenkins, then save it to Notepad.

Dashboard -> admin -> Configure -> API token -> create and save token

Understanding Webhooks

A webhook is a mechanism for two applications to communicate with each other in real-time. When a specific event occurs in the source application, it sends a notification to a URL and a payload of data. The receiving application can then take action based on the data in the payload. Webhooks are commonly used in web development to streamline business processes and improve efficiency.

Configuring GitHub Webhook for Jenkins

Github -> project repository -> settings -> webhook -> add webhook

Payload url: http://jenkins-IP:8080/github-webhook 

Add secret key which was save earlier in Notepad

Content type: application/jason

Select individual events: select pushes and pull requests

Add webhook

Creating an Ansible Playbook

Create a playbook at /root/playbook/script.yml on the Ansible server using SSH, then paste the script below.

---
- hosts: webserver
  tasks:
    - name: Copy index.html to web server
      copy:
        src: /opt/index.html
        dest: /var/www/html

This playbook will copy the file /opt/index.html from the Ansible machine to the directory /var/www/html on the hosts in the "webserver" group.

Building the Pipeline in Jenkins

Now let's build a pipeline. Go to the Jenkins dashboard and go to our project in Jenkins

Dashboard -> Demo-project-1 -> Configuration

Make sure to select Build Triggers as GitHub hook trigger for GITScm polling

Now add build steps "Send files or execute commands over SSH"

Add below command in Exec command

rsync -avh /var/lib/jenkins/workspace/Demo-project-1/*.html root@3.89.246.46:/opt/index.html

The command you provided is an rsync command that will copy all .html files from the /var/lib/jenkins/workspace/Demo-project-1/ directory on the Ansible machine to the file /opt/index.html

Add post-build action

Exec command

ansible-playbook /root/playbook/script.yml

This will run an Ansible playbook located at /root/playbook/script.yml and push the code to /var/www/html on the webserver.

Testing the Pipeline

Run the instructions listed below after logging into the developer system to clone the repository and start the pipeline whenever you make changes to index.html and push the code to GitHub.

git clone https://github.com/Pratik1795/demo-project.git
cd demo-project/ 
vi index.html
cd demo-project/
vi index.html
git status
git add .
git commit -m "1st commit" index.html
git push origin main

You can check the job output below to see if it was successful as soon as you push the code.

Your code will be visible when you browse to the webserver IP address on the browser.

Your CI/CD pipeline is now complete. Try pushing different code to the repository and seeing what happens with Jenkins.

Troubleshooting Issues

If the pipeline fails, look at the console output in Jenkins and try to troubleshoot the problem. You can get in touch with me about this project on Linkedin if you're still facing any issues.

Conclusion

The major goal of this project was to create an automated CI/CD pipeline for our online application. To achieve this, we used Jenkins, Ansible, Apache, and AWS. AWS EC2 instances are automatically built, compiled, analyzed, and deployed when new code is added to the GitHub repository after the pipeline has been configured.

DevOps CI/CD Pipeline using Jenkins, Ansible, Webserver and AWS

Table of contents