Initial Setup

Github

Tenants are initially provided two Github repositories

1. <tenant_short_code>-jenkins-config: Repository for Pipleine configuration

2. <tenant_short_code>-flux-config: Repository for Helm release configurations

Jenkins Config Repo

This repository will be used to configure your build pipeline, specifically the build triggers and integrated SDP libraries. This repository will contain two files:

pipeline_config.groovy - This file defines a naming convention as well as SDP libraries that are used by the Jenkins Templating Engine (JTE). Let’s take a look at a sample file (content omitted for brevity).

application_environments{
  development{
    short_name = "dev"
    long_name = "Development"
    k8s_context = "development"
  }
  [...]
}

keywords{
    prod        =  /^[Pp]roduction$/
    develop     =  /^[Dd]evelop(ment|er|)$/
    stagingenv  =  /^[Ss]taging$/
    testing     =  /^[Tt]est$/
    hotfix      =  /^[Hh]ot[Ff]ix-/
    release     =  /^[Rr]elease-(\d+.)*\d$/
    feature     =  /^[Ff]eature.*/
}

libraries{
  @override mcaas_docker {
    build_strategy = "dockerfile"
    registry_type = "ecr"
    registry = "<aws_account>.dkr.ecr.us-east-1.amazonaws.com"
    [...]
  }
  git {
    github_enterprise
  }
  mcaas_anchore {
      [...]
  }
  sdp {
      images{
        [...]
      }
  }
}

The application_environments section define a naming convention used by Kubernetes to tag build artifacts for specific environments. This section is the same for all tenants and should not be changed.

The keywords section define regular expressions for matching branch names. These are used by the Jenkinsfile to determine if a build action needs to be triggered. This is discussed in more depth below.

The libraries section define supporting SDP libraries which provide the implementation for the build tools used by the Pipeline. the Docker library is used to build the images, the Github library is used to check out the workspace and the Twistlock library is used for container scanning. These initial libraries should remain unchanged but additional libraries may be added if more build functionalty is needed.

Jenkinsfile - This file defines the Pipeline steps to be executed for specific Github actions. Lets take a look at a sample configuration.

on_pull_request from: feature, to: develop, {
        mcaas_build_and_push()
        mcaas_scan_container_image()
}

on_merge to: develop, from: feature, {
        mcaas_build_and_push()
        mcaas_retag(env.GIT_SHA, "dev-"+ env.GIT_SHA)
}

The on_pull_request action (Github library) will monitor Pull Requests in Github and will trigger build steps when the source and target branch names match one of the configurations. The example above will trigger the build and push steps (Docker library) and the scan container image step (Twistlock library) when a Pull Request has been opened from a feature branch to the develop branch.

The on_merge action (Github library) will monitor Merge Requests in Github and will trigger build steps when the source and target branch names match one of the configurations. The example above will trigger the build, push and retag steps (Docker library) when a Pull Request has been merged from a feature branch to the develop branch.

Flux Config Repo

This repository will define the HelmRelease files for each deployment. Each application will have its own release file. The folder structure should follow the hierarchy below:

module/ <-- this is a sample module folder for the tenant
module/base <-- resources here will be deployed to all environments' clusters
module/development <-- resources here will be deployed to development environment cluster

MCaaS provides a repository with sample Helm charts and release files at https://github.helix.gsa.gov/MCaaS/mcaas-tenant-charts

For an in depth look at how to use these charts, see the Continuous Deployment section.

Jenkins Email Extension

Email Extension is a email notification plugin used by Jenkins. It allows Jenkins jobs to send email notifications during jobs or after they complete. Among other use-cases, it can be leveraged to notify users upon a build's failure.

Usage

Basic integration

MCaaS has installed and configured this plugin onto the MCaaS Tenant Jenkins box. It is set up to send emails via the smtp.gsa.gov server, with the from address set to mcaas-support@gsa.gov.

Email notifications can be added into new or existing Jenkins pipelines via the following snippet:

Jenkinsfile

emailext body: '<message here>',
  subject: '<subject here>',
  to: '<destination address>'

Email on Job Failure

The following snippet can be used by wrapping the existing Jenkinsfile. It will then generate email notifications when a build fails:

Jenkinsfile

try {
  <build trigger definitions>

} catch (err) {
	// todo: figure out how to check for build error
	echo "sending email to <destination address> for error ${err}"
	mail to: development.email_to,
		subject: "Failed MCaaS Jenkins Pipeline Job: ${currentBuild.fullDisplayName}",
		body: "Hello,\n \nSomething is wrong with build ${currentBuild.fullDisplayName}}\n \nFor more information, please see the build link and error below.\n \nFor additional help, please open a Jira Service Ticket via https://cm-jira.usa.gov/servicedesk/customer/portal/12/create/174 and ask for it to be routed to MCaaS.\n \nThank you,\nMCaaS\n \nBuild: ${env.BUILD_URL}\n \nError: ${err}"
}

Additional Configuration Options

Notification upon additional conditions or events can be implmented in the Jenkinsfile. Some helpful resources to get started include the Email plugin documentation, and this Jenkins pipeline tour email page (click Toggle Scripted Pipeline, for applicable syntax).

you to specify recipients for each type of trigger as well as a pre-send script that can notify the user of a change the pipeline shows. A email generated by Jenkins will be sent with a link to the job that he failed to show, which will also show brief information about the error.