How to Build a Scalable CI/CD Pipeline on AWS with Jenkins, Git, Docker, and Kubernetes?

Part 1: Setting up Jenkins Master and GitHub Integration

Step 1: Launch Jenkins Master on AWS

1.1 Log in to AWS Console, navigate to EC2, and launch an Amazon Linux instance.

1.2 Connect to your instance via SSH.

Step 2: Install Jenkins and Git

2.1 Update system packages:

sudo yum update -y

2.2 Install Java (required by Jenkins):

sudo yum install java

2.3 Add Jenkins repository and install Jenkins:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm — import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
sudo yum install jenkins -y

2.4 Start Jenkins service:

sudo service jenkins start

2.5 Open Jenkins in your browser, retrieve the initial admin password, and complete the setup wizard.

Step 3: Install Git on Jenkins Master

3.1 Install Git:

sudo yum install git -y

Step 4: Create a GitHub Repository

4.1 Create a new repository on GitHub for your website code and Dockerfile.

4.2 Clone the repository to your local machine:

git clone https://github.com/your-username/your-repo.git

4.3 Create your website code and Dockerfile inside the cloned repository.

Step 5: Configure Webhook for Jenkins

5.1 In your GitHub repository, go to “Settings” > “Webhooks” > “Add webhook.”

5.2 Set Payload URL to http://your-jenkins-ip:8080/github-webhook/.

5.3 Set Content type to application/json.

5.4 Add a Jenkins webhook secret (optional) for additional security.

5.5 Choose “Just the push event” for webhook triggers.

5.6 Save the webhook.

Now, your Jenkins master is set up, Git is installed, and GitHub is configured to trigger Jenkins builds on code pushes.

Part 2: Jenkins Jobs for Docker Build and Push to Docker Hub

Step 6: Create Jenkins Job for Docker Build (Slave Node 1)

6.1 Open Jenkins dashboard and click on “New Item.”

6.2 Enter a job name (e.g., “Docker_Build_Slave1”) and choose “Freestyle project.”

6.3 In the “Source Code Management” section, select Git, and provide your repository URL.

6.4 Under “Build Triggers,” check “GitHub hook trigger for GITScm polling.”

6.5 Add a build step: “Execute shell.”

#!/bin/bash
docker build -t your-dockerhub-username/myweb:$BUILD_ID -t your-dockerhub-username/myweb:latest .

6.6 Add another build step: “Execute shell” for Docker login.

#!/bin/bash
docker login -u your-dockerhub-username -p your-dockerhub-password

6.7 Save the job configuration.

6.7 Save the job configuration.

Step 7: Configure Jenkins Credentials

7.1 In Jenkins dashboard, go to “Credentials” > “System” > “Global credentials (unrestricted)” > “Add Credentials.”

7.2 Add your Docker Hub credentials (username and password).

Step 8: Create Jenkins Job for Kubernetes Deployment (Slave Node 2)

8.1 Create a new job (e.g., “Kubernetes_Deployment_Slave2”) following similar steps to Part 2, Steps 6.2–6.4.

8.2 In the “Build” section, add a build step: “Execute shell” for Kubernetes deployment.

#!/bin/bash
sudo kubectl delete deployment myweb
sudo kubectl delete service mysvc

sudo kubectl apply -f — <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: myweb
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
— name: my-app-container
image: your-dockerhub-username/myweb:latest
imagePullPolicy: Always
ports:
— containerPort: 8080
EOF

sudo kubectl apply -f — <<EOF
apiVersion: v1
kind: Service
metadata:
name: mysvc
spec:
selector:
app: my-app
ports:
— protocol: TCP
port: 80
targetPort: 8080
type: NodePort
EOF

*make the Indentation correct.

8.3 Save the job configuration.

Step 9: Configure Jenkins Credentials for Kubernetes(if needed)

9.1 In Jenkins dashboard, go to “Credentials” > “System” > “Global credentials (unrestricted)” > “Add Credentials.”

9.2 Add your Kubernetes configuration file as a “Secret File.”

Part 3: Jenkins Integration and Completion

Step 10: Configure Jenkins Job Dependencies

10.1 Open Jenkins dashboard and navigate to the “Kubernetes_Deployment_Slave2” job.

10.2 Click on “Configure” and go to the “Build Triggers” section.

10.3 Check the option “Build after other projects are built” and specify “Docker_Build_Slave1” in the “Projects to watch” field.

10.4 Save the configuration.

Step 11: Test the Setup

11.1 Push changes to your GitHub repository to trigger the Jenkins build.

11.2 Verify that “Docker_Build_Slave1” is triggered first, followed by “Kubernetes_Deployment_Slave2” upon successful completion.

Step 12: Monitor Jenkins Build Logs

12.1 Check the console output of each Jenkins job for detailed logs and troubleshooting.

12.2 Inspect the build logs for both jobs to ensure Docker image build and Kubernetes deployment steps are successful.

Step 13: Access Your Deployed Application

13.1 Retrieve the external IP of your Kubernetes cluster or the node’s IP with the NodePort assigned to “mysvc.”

13.2 Open a web browser and access your application using the IP and port (e.g., http://your-node-ip:node-port).

You need to specify which job you want to run in which node by giving and using Labels.In option like restrict where this job can be build while configuring the Job

Congratulations! You’ve successfully set up a comprehensive CI/CD pipeline with Jenkins, Git, Docker, and Kubernetes. This step-by-step guide has walked you through each component, explaining the commands and configurations at each stage. Now, you have a streamlined process for building Docker images, pushing to Docker Hub, and deploying applications on a Kubernetes cluster. Feel free to explore and customize this setup based on your specific project requirements.