Understanding Jenkins CI/CD Pipeline And Its Stages

Last Updated : 21 Feb, 2024

Jenkins is an open-source automation server that enables developers to reliably build, test, and deploy applications. It supports continuous integration and continuous delivery (CI/CD) workflows that allow teams to frequently deliver high-quality software. Jenkins is extremely popular, with over 100,000 installations worldwide.

At its core, Jenkins provides an automation engine with an extensive plugin ecosystem that offers integrations for practically any DevOps toolchain. This allows Jenkins to fit into diverse infrastructure setups and support all types of development processes.

CI/CD

CI/CD means making small code changes often. After each change, Jenkins automatically runs tests. If the tests pass, Jenkins releases the new code. This is better than releasing big code changes. Big changes take longer to test and fix. Small changes can be tested and released faster.

CI/CD pipelines automate the software delivery process. They help developers build, test, and deploy code changes faster and more reliably. Developers make changes to the source code in a version control system like Git. This code repository acts like a shared document. Whenever a change is made, the pipeline kicks off a pre-defined set of steps automatically:

First, an automated build compiles the code and runs tests immediately. If the tests fail, the changes are rejected. This catches errors early. Next is continuous integration. Developers frequently merge their code changes into the main branch. With each merge, builds and tests are run to ensure nothing breaks.

Then comes continuous delivery. The changes are deployed to testing/staging servers. This is like a dress rehearsal before the actual release. Finally, in continuous deployment, the changes are deployed directly to production with little manual intervention. This enables quick delivery of features to users.

In summary, CI/CD pipelines are assembly lines for software delivery. They automate the building, testing, and releasing of code changes rapidly and reliably. Developers get fast feedback if they break things. Companies can release features faster without sacrificing quality.

Key components and concepts

Jobs: These are the repeatable building blocks that Jenkins stitches together to automate tasks in pipelines. Typical jobs include running test scripts, building code, deploying apps, and other steps. Jobs can have configurable kickoff triggers to start them automatically, along with inputs, running environments, and result reports.

Pipelines: Pipelines chain discrete jobs into a coordinated end-to-end workflow that models the entire delivery process from code to production. Instead of separate jobs, pipelines provide visibility and control into the bigger picture process flows that cross DevOps functional silos.

Agents: Agents refer to the distributed network of servers and environments where Jenkins orchestrates executing the jobs and deployment tasks across an application pipeline. Agents can either be servers provisioned internally or external cloud hosts that connect into Jenkins server. Spreading work across agents helps scale up capacity but also introduces challenges of managing all these distributed resources.

Plugins: The key to Jenkins flexibility comes from its over 1,600 plugins which enable it to interoperate with practically every major DevOps, infrastructure, container, automation, and productivity tool. Plugins provide out-of-the-box integrations with hundreds of third party systems – spanning from source code tools like Git to orchestrators like Kubernetes and everything in between. Plugins also equip Jenkins with additional capabilities around notifications, dashboards, analysis, security, governance, and scaling to custom fit organizational needs.

Understanding Jenkins Pipelines

Code: Coders write app source code and manage versions in systems like Git. These code repositories track all changes over time. Webhooks can connect the Git repositories to the Jenkins automation server. These event triggers notify Jenkins when coders push code changes. This automatically starts Jenkins jobs to kick off build pipelines based on code updates without needing manual help.

Build: When code commits happen, Jenkins gets the newest source code files to add updates. The self-hosted agents pool offers environments for builds to run. Trusted build tools like Maven, Ant, Gradle, and Microsoft MSBuild are set up on agents. Jenkins has plug-ins to call these tools. Build tools compile code, bundle libraries, make binaries/artifacts, and package them as JARs, WARs, containers. Unit tests and code checks validate quality and best practices during builds.

Test: Before launch, the built code needs more real testing. Jenkins coordinates testing across the pipeline. Testing includes types like user interface, performance, security, compatibility. Quality assurance teams design test plans, cases and suites to run either manually or using frameworks like Selenium. Jenkins starts all these tests and organizes detailed results reports, logs, metrics.

Signing/Security: Extra controls like code signing can improve quality and security. Automated scanning of builds checks for issues early. Manual approvals from oversight teams enforce governance policies.

Deploy: Jenkins helps deploy vetted code to downstream environments like testing, staging and production. Cloud platforms offer flexible hosts. Containers standardize environments. Tools manage resource pools. Jenkins ties into major platforms like AWS, Azure, Kubernetes, Docker.

Inform: Emails update teams on pipeline progress from code commits to deployments. Dashboards grant visibility through pipeline graphs, logs, testing reports.
Jenkins pipelines let teams build workflows that automatically build, test, and release their apps.

I’ll demonstrate pipeline examples for web apps, mobile apps, and API testing

First, deploying web apps to Kubernetes. When developers push code, Jenkins starts the pipeline. It builds a Docker container image with the code and sends it to a registry. Jenkins then updates Kubernetes config files with the new image details and tells Kubernetes to deploy the web app using this image.
Now delivering mobile apps from developer devices to users. The pipeline starts after code is added to the main codebase. It compiles Android and iOS app versions. Then it runs UI and performance tests on these builds in emulators. If tests pass, Jenkins submits the build for publishing in the Play Store or App Store and tells the mobile team.
Finally, API testing. Jenkins kicks off this pipeline when developers merge code. It runs unit tests on individual modules. Integration tests check modules work together correctly. Load tests hit the API server to measure performance. Jenkins shows reports on test results, code coverage, and issues. Based on results, it promotes the API to higher environments or warns developers of broken builds.

In summary, Jenkins pipelines automate continuously releasing high-quality app updates by coordinating infrastructure. They enable engineer teams to deliver frequently.

Creating Jenkins Jobs

Let’s talk about setting up jobs in Jenkins without tripping any alarms. First, you’ll want to log into your Jenkins dashboard. This is like entering a workshop full of tools to help build your software. Once inside, click “New Item” to start a new project. Give your job a nice name – maybe after your favorite movie character or snack food. The choice is yours!

Now comes the fun part: Picking your job type. There are a few options here that do different things:

Freestyle project – Lets you run custom commands and scripts. Like following a recipe step-by-step.
Pipeline – For stacking tasks together into an automated workflow. Kind of like an assembly line!
Multibranch Pipeline – When you have code in multiple branches, and want to build from each. Like building several models of a toy from different molds.

There are a few more types too, but these cover most use cases. Next, configure your job’s settings. Here you can pick and choose what you want it to do – things like:

Fetch code from version control
Build and compile the code
Run automated tests
Deploy it somewhere after a successful build

The options are endless. Set up your job just as you need it. Finally, save your job and click “Build Now” to test it out. Watch your job execute each step and voila – you’ve successfully automated the build process!

Now whenever new code lands, your job will wake up and do its work automatically. No more manual building or testing. With Jenkins, you can set up an assembly line for your software projects!

Jenkins Pipeline

Stages in Jenkins pipelines

pipeline {
  agent any
  stages {
    stage('Checkout') {
      steps {
        git 'https://github.com/org/repo.git'
      }
    }
    stage('Build') {
      steps {
        // compile code
        // save build artifacts to S3 bucket
      }
    }
    stage('Test') {
      steps {
        // download artifacts from S3
        // load test data into Postgres DB
        // run integration tests
        // continue only if all tests pass
      }
    }
    stage('Deploy') {
      steps {
        // download artifacts from S3
        // deploy to Kubernetes cluster
      }
    }
  }
}

Here is a more detailed explanation of each of the 4 stages in the pipeline:

Get latest code: Uses the git step to check out code from a Git repository. Provides the URL to the repo as a parameter. Clones the latest commit to the local workspace. Ensures all subsequent steps have access to the freshest source code.
Build code: Compiles the application source code into executables/packages. Saves the build artifacts like JARs, containers, etc to cloud storage. Uses S3 or an equivalent object store for storing the outputs. Makes the build outputs available for later stages.
Test code: Fetches the build artifacts from the storage location. Loads test data and schemas into a Postgres database. Executes automated integration tests against the application. Continues only if all test cases pass as expected. Gates deployment on successfully passing tests.
Deploy code: Downloads the vetted build artifacts for deployment. Deploys the application to a Kubernetes cluster. Pushes the containers to the target runtime environment. Makes the built and tested application available to users. This pipeline has discrete steps for each major phase of the CD process – repo checkout, build, test, and deploy. Each stage focuses on one concern and uses standard tools. This modular design enables extensibility and maintainability.

Declarative Pipeline Vs Scripted Pipeline

When I first started using Jenkins for continuous integration and delivery, the concept of pipelines seemed confusing and complex. There were two primary approaches for defining and creating pipelines in Jenkins – declarative pipelines and scripted pipelines. Both approaches could accomplish the core goal of automating software workflows, but in markedly different ways.

Declarative pipelines take a more structured, easy-to-visualize approach. The Jenkinsfile format allows you to lay out pipeline stages, such as build, test, and deploy, in a linear, sequential order. Each stage has a clean block defining what steps should occur within that stage. To me, this maps very cleanly to the progression of taking code from version control, through build, validation, and release processes. I can look at the Jenkinsfile like a flowchart and understand the logical order of events.

In contrast, scripted pipelines offer much more flexibility and customization capability, at the cost of increased complexity. Steps are defined procedurally using Groovy code encapsulated within methods like build(), test(), etc. The logic flow is harder for me to follow, as I have to trace through the method calls to see the overall sequence. The highly customizable nature of scripted pipelines enables much more sophisticated orchestration, but requires more Groovy programming expertise.

Through trial and error, I’ve found that declarative pipelines tend to work best for straightforward, common situations where I need to automate a basic CI/CD workflow. When I want a clean, linear progression from start to finish without a lot of conditional logic, declarative pipelines allow me to define it concisely. However, for more complex scenarios or situations requiring extensive customization, scripted pipelines prove more capable.

The key differences:

Declarative defines pipeline block with nested agent and stages
Scripted uses standard Groovy/Java syntax
Declarative segregates steps into stage blocks
Scripted puts steps inline within stage blocks
Declarative enforces structured syntax
Scripted is free-form code

At the end of the day, both pipeline types enable Jenkins to build, validate and deliver source code automatically. But each comes with different trade-offs in terms of simplicity versus flexibility. Understanding those core differences has been helpful for me in selecting the right approach for my needs and using Jenkins pipelines most productively.

Declarative Pipeline Code Syntax

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        // build steps
      }
    }
    stage('Test') {
      steps { 
        // test steps  
      }
    }
    stage('Deploy') {
      steps {
        // deploy steps
      }
    }
  }
}

Scripted Pipeline Code Syntax

node {

  stage('Build') {
    // build steps
  }

  stage('Test') {
    // test steps
  }

  stage('Deploy') {
    // deploy steps
  }
}

Key best practices of Jenkins

I have been writing about DevOps tools like Jenkins for a while. I have seen how using Jenkins pipelines can really help software teams work better. Jenkins lets you automate a lot of manual work. But many teams just starting with Jenkins struggle. They don’t always use best practices. Then their pipelines become messy, break often, and have security holes. Teams end up wasting time babysitting jobs or fixing issues.

Based on my research and talks with engineers, I want to share tips on how to use Jenkins right. I’ll talk about ways to structure code for reliability and security. I’ll discuss how to define infrastructure as code so it’s reproducible. I’ll cover good plugins to use, scaling your setup, and baking security in from the start. I’ll use real examples from companies like Square, Twilio, and Adobe. These teams use Jenkins well to improve their development.

Jenkins is powerful but you need to be careful. Governance is important – you need a balance between giving developers freedom while maintaining control centrally. I’ll examine how mature teams strike this balance. The goal is to help teams use Jenkins to enhance their workflows, not hinder them. With the right practices, Jenkins can help deliver software faster and more reliably.

Recap of Jenkins CI/CD as the backbone for modern DevOps

Many companies use Jenkins to help teams work together to make software. Jenkins is a tool that lets developers test and release their code changes more quickly. This is called continuous integration and continuous delivery or CI/CD.

Jenkins helps teams:

See problems faster – Tests run after each code change. If tests fail, developers know right away.
Release faster – New code can be released automatically after passing tests. No need to wait for a big release.
Work together – Jenkins shows progress. Everyone knows the status of code changes and tests.

Advantages

Fixes are faster – Problems are found quickly before more coding.
Customers get updates – New features can be released more often.
Less risk – Small changes are easier to test and fix than big changes.
Things to Know About Jenkins
Jenkins takes some work to set up. But once it is running, it automates the process.
Teams should start small then add more as they learn Jenkins.
Jenkins works best when the whole team uses it. Everyone needs to commit code often and fix issues right away.
Jenkins is popular because it is open source and works with many tools. But training and support costs time and money.
Jenkins helps developers work together to test and release code faster. This allows companies to satisfy customers with quick fixes and new features. With some learning and practice, Jenkins can improve many development teams.

Disadvantages

Jenkins can be tricky to learn – It has a lot of settings and plugins that take time to figure out.
Plugins can cause issues – They add features but managing them and keeping them comparatively hard.
Security risks – Jenkins can be vulnerable if not set up properly.
Scaling is tough – Running Jenkins smoothly with many users and jobs requires effort.
Debugging pipelines – When something breaks, finding the issue can be really slow and difficult.
Giving access to different users is complex – It’s easy for users to get too much access on accident.
Integration with other tools isn’t seamless – Often requires custom coding to connect tools together.
Infrastructure changes cause problems – Keeping all environment settings and configs straight is tough.
Takes work to maintain – Upgrades, backups, managing plugins – it’s a lot of overhead.
Hard to switch later – Moving away from Jenkins in the future can be challenging.

Conclusion

If the