Author: Arpit Singh

PART 1 – Deploying MuleSoft Apps to multiple CloudHub environments using Gitlab public server

What is CI/CD

The CI/CD pipeline is one of the best practices for DevOps/development teams to implement, for delivering code changes more frequently and reliably

What is Continuous Integration

  • Consider an application that has its code stored in a Git repository in GitLab.
  • Developers push code changes every day, multiple times a day
  • For every push to the repository, you can create a set of scripts to build and test your application automatically.

What is Continuous Delivery

  • Your application is not only built and tested at every code change pushed to the codebase (Continuous Integration).
  • But, as an additional step, it’s also deployed continuously, though the deployments are triggered manually.
  • This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

What is Continuous Deployment

  • Similar to Continuous Delivery.
  • The difference is that instead of deploying your application manually, you set it to be deployed automatically.
  • It does not require human intervention at all to have your application deployed.

Continuous Delivery OR Continuous Deployment: How to decide the deployment Strategy??

  • Completely depends on the requirements.
  • Generally, companies like Netflix, which deal with 1000+ changes every day have opted for Continuous Deployment
  • While projects that fix a specific date for go live or updates, opt for Continuous Delivery as their deployment Strategy

How GitLab CI/CD works

GitLab CI/CD is configured by a file called .gitlab-ci.yml placed at the repository’s root. The scripts set in this file are executed by the GitLab Runner.

Gitlab Configuration Steps – Best Practices

Below are the steps to configure CI/CD pipeline rules and best practices. Please note that the scope of the below configurations are at the Pipeline level. The code (be it java, mule, node etc.) are independent of the below configurations.

Creating Private Repository in GitLab

Login to https://gitlab.com/ with your credentials. If you do not have a gitlab account, you can create one at https://gitlab.com/users/sign_up

Gitlab has a Community and Enterprise Version. I will be using Community version as it is free but imposes few restrictions which are out of scope for this article

Steps:

  • Browse to the Project page and create a new project as below. Please note that if you do not want your project to be accessible publicly, please select the Visibility Level as Private

Setting up Project Specific Configurations

On clicking Create Project, you will be redirected to the created project repository home page. 

Create default branch 

  • Gitlab by default creates a master branch. 
  • We will be creating a new branch named develop which will be used to release the code to the development environment. 
  • We will be using master branch to release the code to the production environment.

Steps

  • Click on the plus sign and click on New branch
  • Enter develop as the Branch name and Create new branch

Set Default Branch

  • By default, Gitlab assigns master branch as a default branch. We would be using develop as a default branch.
  • By choosing the default branch, we tell Gitlab to use develop branch by default for cloning/downloading the project

Steps

  • On the left side Pane go to Settings-> Repository

Expand the Default Branch Section and select the default branch as develop. Save Changes.

Protect the branches and restrict push and merge access

This configuration is essential to keep our develop and master branch safe of unauthorized  commits. 

Rules for develop branch

  • Only maintainers will be allowed to merge to the develop branch.
  • Only maintainers  are allowed to directly push to the develop branch. 
  • A developer working on the code has to create a new branch, make the code changes and send a merge request to a Maintainer.
  • The Maintainer , since is given permission to merge, can review the code changes and merge the code to develop

Rules for master branch

  • Only maintainers will be allowed to merge to the develop branch.
  • No one is allowed to push directly to master branch 
  • A developer working on the code has to create a new branch, make the code changes and send a merge request to a Maintainer.
  • The Maintainer , since is given permission to merge, can review the code changes and merge the code to develop

Steps:

  • Expand the Protected Branches Section and Configure the below:
    • Branch: develop
    • Allowed to merge: Maintainers
    • Allowed to push: No one
    • Branch: master
    • Allowed to merge: No One
    • Allowed to push: No one

Define CI Variables

Go to Settings-> CI/CD -> Variables

We will be using the below Variables that will be used for deployment to cloudhub

  • ANYPOINT_USERNAME_DEV
  • ANYPOINT_PASSWORD_DEV
  • CLOUDHUB_ENV_SANDBOX
  • ANYPOINT_USERNAME_PROD
  • ANYPOINT_PASSWORD_PROD
  • CLOUDHUB_ENV_PRODUCTION

Keep in mind that since we have protected the develop and master branches, it is mandatory that we protect all the above variables as well. Only protected variables are accessible to protected branches.

Gitlab Runners

Gitlab uses runners to execute the Gitlab pipelines. We will be using gitlab public shared runner to execute the pipelines,  but ideally, enterprise level pipelines use their own private runners hosted on private server, the scope of which is out of the current article

Go to Settings.> CI/CD->Runners

As you can see, the default configurations uses shared runners, and we can set up specific Runners as well

Setting up Slack notifications for build/deployment failures

Gitlab has amazing integrations with a lot of third party services including slack, Github, Jira, Hangout chats etc. 

Steps:

Go to Settings-> Integrations

You can see the list of applications available for integrations with Gitlab 

Select Slack Notifications

Click on Add an incoming Webhook. This is a deep link to slack which will forward you to your slack workspace app directory. Click on Add to Slack. 

Choose a channel where you want the CI/CD pipeline status notified.

Click on Add Incoming Webhooks Integration

Copy the created webhook url. We will need this url when integrating slack with gitlab

Paste the above webhook url under Settings->Integrations->Slack Notifications->WebHook

Test settings and save changes

If everything goes well, you will see the below message

Project Specific Configurations

Create a Simple Mule App

As mentioned in pre requisites, I have created a very simple mule App with default configurations in pom.xml.

The final project structure looks as below

Environment Specific Configurations

Mule4 documentation has a detailed description of deploying mule apps to multiple environments dev, qa, stage, prod etc. [ Configuring Environment Properties ]. I would be using this to make by pipeline deploy the mule app on either develop or prod environment

Below is how I have made the configurations:

Assuming that we have a run time variable as env which can accept values dev/prod,

In my Global configurations, I have set the property file name as {env}.yml. Based on the env=dev or env=prod , my corresponding config file will be picked i.e dev.yml or prod.yml

The property files just has a simple http port configurations

Also, I have created a global Property env with value as dev. This will act as a default env value in case we do not provide the env variable value during the run time

pom.xml configurations

Maven build profiles

CI/CD best general requirements also include having different build profiles for different environments. When we talk in terms of Mule Apps, it can be divided into two parts:

  1. Where will our app be deployed i.e. CloudHub, On-Premise, AWS EC2 
  2. Once the above the achieved, which environment will the app be deployed i.e dev or prod etc.

Talking about the first part, I would be using below build profiles:

  1. CloudHub
  1. Standalone
  2. arm

This article will only deal with CloudHub deployment.

Mule Maven Plugin

This plugin takes care of packaging and deployment of Mule app to CloudHub. This is a required plugin if you want to automate your deployment via CI/CD (This is not just Gitlab specific and applied to other CI tools as well like Jenkis, Teamcity etc.)

If you look at the <cloudHubDeployment> section, Anypoint username and password, CloudHub env(Sandbox/production), all these properties are fetched from the Gitlab CI/CD variables which we configured under Define CI Variables.

This way the only place where our credentials will be exposed is at the Gitlab Repository level, and only accessible by the Project Admin and the users the admin provides access to. 

Pretty Secure!

Now if you look at the properties section

We are getting the env  from run time [But not from GitLab CI Variables!]. So, the other place where we can pass the run time values is by executing maven command on the project. I will show that in the next section

.gitlab-ci.yml configurations

This is the place where the magic of Gitlab CI/CD lies. This file takes care of all the CI/CD

  • Maven build.
  • Maven package
  • Maven deploy
  • Passing run time parameters
    Accessing Gitlab CI Variables
  • Setting up rules for deployment (CD)
  • setting up different stages i.e dev/prod

I would strongly recommend to refer this link and check out the file properties

Configurations done in the file for CI/CD

Docker Image used

As mentioned earlier, Since we are using gitlab server for the deployment, gitlab provides several pre-built images for different project builds like maven, nodeJs, serverless.

We can also create our own docker images, but that would require creating our private runner which is out of the scope of this article

Docker Image used: maven:3.6.1-jdk-8

Stages

Since we have defined two environment config files, we have corresponding deployment stages as well

Deployment to Dev

We have set three variables. These are the variables that are mapped to the Gitlab CI Variables

In the script section, we define the maven command which takes care of the below:

  1. Packaging the project to an executable mule jar
  2. Defining the profile to be used for deployment [ -pcloudhub ]
  3. Passing the run time variables to be used by the mule app 

-Denv=dev -DCLOUDHUB_ENV=$CLOUDHUB_ENV -DANYPOINT_USERNAME=$ANYPOINT_USERNAME_DEV -DANYPOINT_PASSWORD=$ANYPOINT_PASSWORD_DEV

  1. Deploying the mule app using mule maven plugin [-dmuleDeploy]

Finally we define the CD rules

This configuration tells Gitlab to only initiate the pipeline if there is a change in the develop branch.

If you try to link the Gitlab Setting up Project Configurations with the above rule, you can link the rules very easily!

Deployment to Production

The only difference between dev and production configurations is below:

  1. Variables: We will use production specific values
  2. CD rule for deployment: This tells that the master pipeline can only be triggered manually via WEB i.e using protected tags and initiating the run manually. Extra layer of safety, since it’s production right!

Pushing the code to gitlab

And that’s it. You are ready to push your code and test whether your Mule app gets deployed to cloudhub.

Clone your gitlab project to your local

Go to gitlab, and browse to the project you created earlier

Copy the Https URL under Clone with Https

Open your local terminal. I am using windows, so I will be using command prompt.

Go to the directory you want to clone your project.

Enter the below command

Git clone <https link> 

You will get a cloned copy of your develop branch (Why develop?).

Browse to your mule app root directory, copy the contents, and paste in your cloned directory.

Next comes the part of pushing these changes to gitlab repo.

Run the below commands at the root directory of your cloned repo

 git add .

 git commit -m “Initial commit”

 git push

And that’s it.

Checking your pipeline status and real time logs

Go to Gitlab CI/CD->Jobs

You will see a running job

Click on Running button to check out the runner logs

As you can see, the Job is successful, and the artifact is successfully deployed to CloudHub

Checking the application in Runtime Manager 

If you now go to Cloudhub->Runtime Manager, you can see you app is deployed

And if you go to settings-> properties, you can see the value of env as dev

3 Comments

  1. Eric Ouellette on July 6, 2020 at 3:06 pm

    This article was exactly what I wanted to see. Unfortunately I cannot see the images/resources that are hosted in the ‘google user content’. For example I get a 403 error from this resource:
    https://lh5.googleusercontent.com/ZjQEy0wGE1d4wE6sr2_bFdL9UXVBgZs3HoNMFzqes_jKCLvHVUvIywBb19JJenIlin5tcDe6Ngu74vUaZEXATv9ckioZTriIU-Bg7YeN_n98_ivSowXQALQTqMhvTUxMp1TFIEqu

    • Arpit Singh on July 12, 2020 at 10:38 am

      Hey Eric.. Your Google User Content need to be whitelisted. I’m not sure how exactly your architecture is, so cannot comment on it in details.

  2. Uday Sankar on July 18, 2020 at 5:35 am

    Apisero is giving us what exactly we want and we are searching for.

Leave a Comment