This project tries to solve the following problems:

  • Creation of a common deployment pipeline.

  • Propagation of good testing and deployment practices.

  • Reducing the time required to deploy a feature to production.

A common way of running, configuring, and deploying applications lowers support costs and time needed by new developers to blend in when they change projects.

It contains bash scripts that allow you to build, test and deploy your applications in various languages and frameworks.

1. Introduction

This section describes the rationale behind the opinionated pipeline. We go through each deployment step and describe it in detail.

You do not need to use all the pieces of Cloud Pipelines. You can (and should) gradually migrate your applications to use those pieces of Cloud Pipelines that you think best suit your needs.

1.1. Five-second Introduction

Cloud Pipelines provides scripts, configuration, and convention for automated deployment pipeline creation with Cloud Foundry or Kubernetes. We support various languages and frameworks. Since this project uses bash scripts, you can use it with whatever automation server you have.

1.2. Five-minute Introduction

Cloud Pipelines comes with bash scripts (available under src//main/bash) that represent the logic of all steps in our opinionated deployment pipeline. Since we believe in convention over configuration, for the supported framework and languages, we assume that the projects follow certain conventions of task naming, profile setting, and so on. That way, if you create a new application, your application can follow those conventions and the deployment pipeline works. Since no one pipeline can serve the purposes of all teams in a company, we believe that minor deployment pipeline tweaking should take place. That is why we allow the usage of that cloud-pipelines.yml descriptor, which allows for provide some customization.

1.2.1. How to Use It

This repository can be treated as a template for your pipeline. We provide some opinionated implementation that you can alter to suit your needs. To use it, we recommend downloading the Cloud Pipelines repository as a zip file, unzipping it in a directory, initializing a Git project in that directory, and then modifying the project to suit your needs. The following bash script shows how to do so:

$ # pass the branch (e.g. master) or a particular tag (e.g. v1.0.0.RELEASE)
$ CLOUD_PIPELINES_RELEASE=...
$ curl -LOk https://github.com/CloudPipelines/scripts/archive/${CLOUD_PIPELINES_RELEASE}.zip
$ unzip ${CLOUD_PIPELINES_RELEASE}.zip
$ cd spring-cloud-pipelines-${CLOUD_PIPELINES_RELEASE}
$ git init
$ # modify the pipelines to suit your needs
$ git add .
$ git commit -m "Initial commit"
$ git remote add origin ${YOUR_REPOSITORY_URL}
$ git push origin master

To keep your repository aligned with the changes in the upstream repository, you can also clone the repository. To not have many merge conflicts, we recommend using the custom folder hooks to override functions.

Cloud Pipelines Scripts contains bash scripts that are required at runtime of execution of a pipeline. If you want to read their documentation, it’s available under src/main/bash/README.adoc file of Cloud Pipelines repository.

1.2.2. How It Works

As the following image shows, Cloud Pipelines contains logic to generate a pipeline and the runtime to execute pipeline steps.

how
Figure 1. How Cloud Pipelines works

Once a pipeline is created (for example, by using the Jenkins Job DSL or from a Concourse templated pipeline), when the jobs are ran, they clone or download Cloud Pipelines code to run each step. Those steps run functions that are defined in the commons module of Cloud Pipelines.

Cloud Pipelines performs steps to guess what kind of a project your repository is (for example, JVM or PHP) and what framework it uses (Maven or Gradle), and it can deploy your application to a cloud (Cloud Foundry or Kubernetes). You can read about how it works by reading the How the Scripts Work section.

All of that happens automatically if your application follows the conventions. You can read about them in the Project Opinions section.

1.2.3. Deployment & languages compatibility matrix

In the following table we present which language is supported by which deployment mechanism.

Table 1. Deployment & languages compatibility matrix
Language CF K8S Ansible

JVM with Gradle

JVM with Maven

PHP with Composer

NodeJS with NPM

Dotnet core

For K8S, a deployment unit is a docker image so any language and framework can be used.

1.2.4. Centralized Pipeline Creation

You can use Cloud Pipelines to generate pipelines for all the projects in your system. You can scan all your repositories (for example, you can call the Stash or Github API to retrieve the list of repositories) and then:

  • For Jenkins, call the seed job and pass the REPOS parameter, which contains the list of repositories.

  • For Concourse, call fly and set the pipeline for every repository.

To achieve this you can use the Project Crawler library.

We recommend using Cloud Pipelines this way.

1.2.5. A Pipeline for Each Repository

You can use Cloud Pipelines in such a way that each project contains its own pipeline definition in its code. Cloud Pipelines clones the code with the pipeline definitions (the bash scripts), so the only piece of logic that needs to be in your application’s repository is the pipeline definition.

1.3. The Flow

The following images show the flow of the opinionated pipeline:

flow concourse
Figure 2. Flow in Concourse
flow
Figure 3. Flow in Jenkins

We first describe the overall concept behind the flow and then split it into pieces and describe each piece independently.

1.4. Vocabulary

This section defines some common vocabulary. We describe four typical environments in terms of running the pipeline.

1.4.1. Environments

We typically encounter the following environments:

  • build environment is a machine where the building of the application takes place. It is a continuous integration or continuous delivery tool worker.

  • test is an environment where you can deploy an application to test it. It does not resemble production, because we cannot be sure of its state (which application is deployed there and in which version). It can be used by multiple teams at the same time.

  • stage is an environment that does resemble production. Most likely, applications are deployed there in versions that correspond to those deployed to production. Typically, staging databases hold (often obfuscated) production data. Most often, this environment is a single environment shared between many teams. In other words, in order to run some performance and user acceptance tests, you have to block and wait until the environment is free.

  • prod is the production environment where we want our tested applications to be deployed for our customers.

1.4.2. Tests

We typically encounter the following kinds of tests:

  • Unit tests: Tests that run on the application during the build phase. No integrations with databases or HTTP server stubs or other resources take place. Generally speaking, your application should have plenty of these tests to provide fast feedback about whether your features work.

  • Integration tests: Tests that run on the built application during the build phase. Integrations with in-memory databases and HTTP server stubs take place. According to the test pyramid, in most cases, you should not have many of these kind of tests.

  • Smoke tests: Tests that run on a deployed application. The concept of these tests is to check that the crucial parts of your application are working properly. If you have 100 features in your application but you gain the most money from five features, you could write smoke tests for those five features. We are talking about smoke tests of an application, not of the whole system. In our understanding inside the opinionated pipeline, these tests are executed against an application that is surrounded with stubs.

  • End-to-end tests: Tests that run on a system composed of multiple applications. These tests ensure that the tested feature works when the whole system is set up. Due to the fact that it takes a lot of time, effort, and resources to maintain such an environment and that these tests are often unreliable (due to many different moving pieces, such as network, database, and others), you should have a handful of those tests. They should be only for critical parts of your business. Since only production is the key verifier of whether your feature works, some companies do not even want to have these tests and move directly to deployment to production. When your system contains KPI monitoring and alerting, you can quickly react when your deployed application does not behave properly.

  • Performance testing: Tests run on an application or set of applications to check if your system can handle a big load. In the case of our opinionated pipeline, these tests can run either on test (against a stubbed environment) or on staging (against the whole system).

1.4.3. Testing against Stubs

Before we go into the details of the flow, consider the example described by the following image:

monolith
Figure 4. Two monolithic applications deployed for end to end testing

When you have only a handful of applications, end-to-end testing is beneficial. From the operations perspective, it is maintainable for a finite number of deployed instances. From the developers perspective, it is nice to verify the whole flow in the system for a feature.

In the case of microservices, the scale starts to be a problem, as the following image shows:

many microservices
Figure 5. Many microservices deployed in different versions

The following questions arise:

  • Should I queue deployments of microservices on one testing environment or should I have an environment per microservice?

    • If I queue deployments, people have to wait for hours to have their tests run. That can be a problem

  • To remove that issue, I can have an environment for each microservice.

    • Who will pay the bills? (Imagine 100 microservices, each having each own environment).

    • Who will support each of those environments?

    • Should we spawn a new environment each time we execute a new pipeline and then wrap it up or should we have them up and running for the whole day?

  • In which versions should I deploy the dependent microservices - development or production versions?

    • If I have development versions, I can test my application against a feature that is not yet on production. That can lead to exceptions in production.

    • If I test against production versions, I can never test against a feature under development anytime before deployment to production.

One of the possibilities of tackling these problems is to not do end-to-end tests.

The following image shows one solution to the problem, in the form of stubbed dependencies:

stubbed dependencies
Figure 6. Execute tests on a deployed microservice on stubbed dependencies

If we stub out all the dependencies of our application, most of the problems presented earlier disappear. There is no need to start and setup the infrastructure required by the dependent microservices. That way, the testing setup looks like the following image:

stubbed dependencies
Figure 7. We’re testing microservices in isolation

Such an approach to testing and deployment gives the following benefits (thanks to the usage of Spring Cloud Contract):

  • No need to deploy dependent services.

  • The stubs used for the tests run on a deployed microservice are the same as those used during integration tests.

  • Those stubs have been tested against the application that produces them (see Spring Cloud Contract for more information).

  • We do not have many slow tests running on a deployed application, so the pipeline gets executed much faster.

  • We do not have to queue deployments. We test in isolation so that pipelines do not interfere with each other.

  • We do not have to spawn virtual machines each time for deployment purposes.

However, this approach brings the following challenges:

  • No end-to-end tests before production. You do not have full certainty that a feature is working.

  • The first time the applications interact in a real way is on production.

As with every solution, it has its benefits and drawbacks. The opinionated pipeline lets you configure whether you want to follow this flow or not.

1.4.4. General View

The general view behind this deployment pipeline is to:

  • Test the application in isolation.

  • Test the backwards compatibility of the application, in order to roll it back if necessary.

  • Allow testing of the packaged application in a deployed environment.

  • Allow user acceptance tests and performance tests in a deployed environment.

  • Allow deployment to production.

The pipeline could have been split to more steps, but it seems that all of the aforementioned actions fit nicely in our opinionated proposal.

1.5. Pipeline Descriptor

Each application can contain a file (called cloud-pipelines.yml) with the following structure:

language_type: jvm
pipeline:
	# used for multi module projects
	main_module: things/thing
	# used for multi projects
	project_names:
		- monoRepoA
		- monoRepoB
	# should deploy to stage automatically and run e2e tests
	auto_stage: true
	# should deploy to production automatically
	auto_prod: true
	# should the api compatibility check be there
	api_compatibility_step: true
	# should the test rollback step be there
	rollback_step: true
	# should the stage step be there
	stage_step: true
	# should the test step (including rollback) be there
	test_step: true
lowercaseEnvironmentName1:
	# used by spinnaker
	deployment_strategy: HIGHlANDER
	# list of services to be deployed
	services:
		- type: service1Type
		  name: service1Name
		  coordinates: value
		- type: service2Type
		  name: service2Name
		  key: value
lowercaseEnvironmentName2:
	# used by spinnaker
	deployment_strategy: HIGHlANDER
	# list of services to be deployed
	services:
		- type: service3Type
		  name: service3Name
		  coordinates: value
		- type: service4Type
		  name: service4Name
		  key: value

If you have a multi-module project, you should point to the folder that contains the module that produces the fat jar. In the preceding example, that module would be present under the things/thing folder. If you have a single module project, you need not create this section.

For a given environment, we declare a list of infrastructure services that we want to have deployed. Services have:

  • type (examples: eureka, mysql, rabbitmq, and stubrunner): This value gets then applied to the deployService Bash function

  • [KUBERNETES]: For mysql, you can pass the database name in the database property.

  • name: The name of the service to get deployed.

  • coordinates: The coordinates that let you fetch the binary of the service. It can be a Maven coordinate (groupid:artifactid:version), a docker image (organization/nameOfImage), and so on.

  • Arbitrary key value pairs, which let you customize the services as you wish.

1.5.1. Pipeline Descriptor for Cloud Foundry

When deploying to Cloud Foundry you can provide services of the following types:

  • type: broker

    • broker: The name of the CF broker

    • plan: The name of the plan

    • params: Additional parameters are converted to JSON

    • useExisting: Whether to use an existing one or create a new one (defaults to false)

  • type: app

    • coordinates: The Maven coordinates of the stub runner jar

    • manifestPath: The path to the manifest for the stub runner jar

  • type: cups

    • params: Additional parameters are converted to JSON

  • type: cupsSyslog

    • url: The URL to the syslog drain

  • type: cupsRoute

    • url: The URL to the route service

  • type: stubrunner

    • coordinates: The Maven coordinates of the stub runner jar

    • manifestPath: The path to the manifest for the stub runner jar

The following example shows the contents of a YAML file that defines the preceding values:

# This file describes which services are required by this application
# in order for the smoke tests on the TEST environment and end to end tests
# on the STAGE environment to pass

# lowercase name of the environment
test:
  # list of required services
  services:
    - name: config-server
      type: broker
      broker: p-config-server
      plan: standard
      params:
        git:
          uri: https://github.com/ciberkleid/app-config
      useExisting: true
    - name: cloud-bus
      type: broker
      broker: cloudamqp
      plan: lemur
      useExisting: true
    - name: service-registry
      type: broker
      broker: p-service-registry
      plan: standard
      useExisting: true
    - name: circuit-breaker-dashboard
      type: broker
      broker: p-circuit-breaker-dashboard
      plan: standard
      useExisting: true
    - name: stubrunner
      type: stubrunner
      coordinates: io.pivotal:cloudfoundry-stub-runner-boot:0.0.1.M1
      manifestPath: sc-pipelines/manifest-stubrunner.yml

stage:
  services:
    - name: config-server
      type: broker
      broker: p-config-server
      plan: standard
      params:
        git:
          uri: https://github.com/ciberkleid/app-config
    - name: cloud-bus
      type: broker
      broker: cloudamqp
      plan: lemur
    - name: service-registry
      type: broker
      broker: p-service-registry
      plan: standard
    - name: circuit-breaker-dashboard
      type: broker
      broker: p-circuit-breaker-dashboard
      plan: standard

Another CF specific property is artifact_type. Its value can be either binary or source. Certain languages (such as Java) require a binary to be uploaded, but others (such as PHP) require you to push the sources. The default value is binary.

1.6. Project Setup

Cloud Pipelines supports three main types of project setup:

  • Single Project

  • Multi Module

  • Multi Project (also known as mono repo)

A Single Project is a project that contains a single module that gets built and packaged into a single, executable artifact.

A Multi Module project is a project that contains multiple modules. After building all modules, one gets packaged into a single, executable artifact. You have to point to that module in your pipeline descriptor.

A Multi Project is a project that contains multiple projects. Each of those projects can in turn be a Single Project or a Multi Module project. Spring Cloud Pipelines assume that, if a PROJECT_NAME environment variable corresponds to a folder with the same name in the root of the repository, this is the project it should build. For example, for PROJECT_NAME=something, if there’s a folder named something, then Cloud Pipelines treats the something directory as the root of the something project.

2. How the Scripts Work

This section describes how the scripts and jobs correspond to each other. If you need to see detailed documentation of the bash scripts, go to the code repository and read src//main/bash/README.adoc.

2.1. Build and Deployment

The following text image shows a high-level overview:

build and deployment

Before we run the script, we need to answer a few questions related to your repository:

  • What is your language (for example, jvm,php, or something else)?

  • what framework do you use (for example, maven or gradle)?

  • what PAAS do you use (for example, cf or k8s)?

The following sequence diagram describes how the sourcing of bash scripts takes place.

sourcing

The process works as follows:

  1. A script (for example, build_and_upload.sh) is called.

  2. It sources the pipeline.sh script that contains all the essential function “interfaces” and environment variables.

  3. pipeline.sh needs information about the project type. It sources projectType/pipeline-projectType.sh.

  4. projectType/pipeline-projectType.sh contains logic to determine the language.

    1. Verify whether a repository contains files that correspond to the given languages (for example, mvnw or composer.json).

    2. Verify whether a concrete framework that we support (for example, maven or gradle) is present.

  5. Once we know what the project type is, we can deal with PAAS. Depending on the value of the PAAS_TYPE environment variable, we can source proper PAAS functions (for example, pipeline-cf.sh for Cloud Foundry).

  6. Determine whether we can do some further customization.

    1. If ADDITIONAL_SCRIPTS_TARBALL_URL env variable is set, we can download a tarball with additional scripts and copy it to ${cloud-pipelines-root}/src/main/bash/

    2. Search for a file called ${cloud-pipelines-root}/src/main/bash/custom/build_and_upload.sh to override any functions you want.

  7. Run the build function from build_and_upload.sh

3. Opinionated Implementation

This section describes a full flow of deploying an app via the deployment pipeline.

We will use:

  • A Paas instance. For example:

  • The infrastructure applications deployed to a binary storage (for the demo, we provide Artifactory).

3.1. Build

The following image shows the results of building the demo pipeline (which the rest of this chapter describes):

build
Figure 8. Build and upload artifacts

In this step, the first thing we do is we generate a version of the pipeline.

If there has been a previous production deployment, we first run API compatibility check, as follows:

  • We search for the latest production deployment

  • The search is done via picking the latest prod/${appName}/${version} tag

  • We call the api compatibility task for the given language and framework

Next, we run unit, integration, and contract tests. We analyze the application code, and we call the appropriate build functions, depending on the used language and framework. Finally, we:

  • Publish a binary / image / artifact of the application. In some cases there’s no need to upload anything since sources get published

  • Publish a package containing stubs of the application.

We also tag the repository with dev/${appName}/${version}. That way, in each subsequent step of the pipeline, we can retrieve the tagged version. Also, we know exactly which version of the pipeline corresponds to which Git hash.

3.2. Test

The following image shows the result of doing smoke tests and rolling back:

test
Figure 9. Smoke test and rollback test on test environment

Here, we:

  • Start the services defined in the pipeline descriptor

  • Upload the binary / sources / image to the PaaS

  • If the application uses a database, it should get upgraded by a migration tool upon application startup

  • If requested, the deployment pipeline can surround the deployed application with stubs of the applications it communicates with

  • From the checked-out code, we run the smoke tests

  • Once the tests pass, we search for the last production release. Once the application is deployed to production, we tag it with prod/${appName}/${version}. If there is no such tag (there was no production release), no rollback tests are run. If there was a production release, the tests get executed.

  • Assuming that there was a production release, we check out the code that corresponds to that release (we check out the tag), download the appropriate artifact and upload it to the PaaS (or just push the sources).

  • The old artifact runs against the NEW version of the database.

  • We run the old smoke tests against the freshly deployed application, surrounded by stubs. If those tests pass, we have a high probability that the application is backwards compatible.

  • The default behavior is that, after all of those steps, the user can manually click to deploy the application to a stage environment.

3.3. Stage

The following image shows the result of deploying to a stage environment:

stage
Figure 10. End to end tests on stage environment

Here, we:

  • Start the services defined in the pipeline descriptor

  • Upload the binary / sources / image to the PaaS

  • From the checked-out code, we run the end to end tests

3.4. Prod

The following image shows the result of deploying to a production environment:

prod
Figure 11. Deployment to production
This step does deployment to production. On production, we assume that you have the infrastructure running. That is why, before you run this step, you must run a script that provisions the services on the production environment.

Here, we:

  • Tag the Git repo with prod/${appName}/${version}.

  • Upload the binary / sources / image to the PaaS

  • We do Blue Green deployment

    • For Cloud Foundry:

      • We rename the current instance of the application (for example, myService to myService-venerable).

      • We deploy the new instance of the app under the fooService name

      • Now, two instances of the same application are running on production.

    • For Kubernetes:

      • We deploy a service with the name of the application (for example, myService)

      • We do a deployment with the name of the application with version suffix,with the name escaped to fulfill the DNS name requirements (for example, fooService-1-0-0-M1-123-456-VERSION).

      • All deployments of the same application have the same label name, which is equal to the application name (for example, myService).

      • The service routes the traffic by basing on the name label selector.

      • Now two instances of the same application are running in production.

  • In the Complete switch over, which is a manual step, we stop the old instance.

    Remember to run this step only after you have confirmed that both instances work.
  • In the Rollback, which is a manual step,

    • We route all the traffic to the old instance.

      • In CF, we do that by ensuring that blue is running and removing green.

      • In K8S, we do that by scaling the number of instances of green to 0.

    • We remov the latest prod Git tag.

4. Project Opinions

This section goes through the assumptions we made in the project structure and project properties.

4.1. Cloud Foundry Project Opinions

We take the following opinionated decisions for a Cloud Foundry based project:

  • The application is built by using the Maven or Gradle wrapper.

  • The application is deployed to Cloud Foundry.

  • Your application needs a manifest.yml Cloud Foundry descriptor.

  • For Maven (example project), we assume:

    • Usage of the Maven Wrapper.

    • settings.xml is parametrized to pass the credentials to push code to Artifactory:

      • M2_SETTINGS_REPO_ID contains the server ID for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_USERNAME contains the username for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_PASSWORD contains the password for Artifactory or Nexus deployment.

    • Artifacts are deployed by ./mvnw clean deploy.

    • We use the stubrunner.ids property to retrieve list of collaborators for which stubs should be downloaded.

    • repo.with.binaries property (injected by the pipeline): Contains the URL to the repo containing binaries (for example, Artifactory).

    • distribution.management.release.id property (injected by the pipeline): Contains the ID of the distribution management. It corresponds to server ID in settings.xml.

    • distribution.management.release.url property (injected by the pipeline): Contains the URL of the repository that contains binaries (for example, Artifactory).

    • Running API compatibility tests with the apicompatibility Maven profile.

    • latest.production.version property (injected by the pipeline): Contains the latest production version for the repo (retrieved from Git tags).

    • Running smoke tests on a deployed app with the smoke Maven profile.

    • Running end to end tests on a deployed app with the e2e Maven profile.

  • For Gradle (example project check the gradle/pipeline.gradle file), we assume:

    • Usage of the Gradlew Wrapper.

    • A deploy task for artifact deployment.

    • The REPO_WITH_BINARIES_FOR_UPLOAD environment variable (Injected by the pipeline) contains the URL to the repository that contains binaries (for example, Artifactory).

    • The M2_SETTINGS_REPO_USERNAME environment variable contains the user name used to send the binary to the repository that contains binaries (for exampl,e Artifactory).

    • The M2_SETTINGS_REPO_PASSWORD environment variable contains the password used to send the binary to the repository that contains binaries (for example, Artifactory).

    • Running API compatibility tests with the apiCompatibility task.

    • latestProductionVersion property (injected by the pipeline): Contains the latest production version for the repository (retrieved from Git tags).

    • Running smoke tests on a deployed app with the smoke task.

    • Running end to end tests on a deployed app with the e2e task.

    • groupId task to retrieve the group ID.

    • artifactId task to retrieve the artifact ID.

    • currentVersion task to retrieve the current version.

    • stubIds task to retrieve the list of collaborators for which stubs should be downloaded.

  • For PHP (example project), we asssume:

    • Usage of Composer.

    • composer install is called to fetch libraries.

    • The whole application is compressed to tar.gz and uploaded to binary storage.

      • REPO_WITH_BINARIES_FOR_UPLOAD environment variable (injected by the pipeline): Contains the URL of the repository that contains binaries (for example, Artifactory)

      • The M2_SETTINGS_REPO_USERNAME environment variable contains the user name used to send the binary to the repo containing binaries (for example, Artifactory).

      • The M2_SETTINGS_REPO_PASSWORD environment variable contains the password used to send the binary to the repo containing binaries (for example, Artifactory).

    • group-id: Composer task that echoes the group ID.

    • app-name: Composer task that echoes application name.

    • stub-ids: Composer task that echoes stub runner ids.

    • test-apicompatibility: Composer task that is executed for api compatibility tests.

    • test-smoke: Composer task that is executed for smoke testing (the APPLICATION_URL and STUBRUNNER_URL environment variables are available here to be used).

    • test-e2e: Composer task that is executed for end-to-end testing (APPLICATION_URL env vars is available here to be used)

    • target is assumed to be the output folder. Put it in .gitignore

  • For NodeJS (example project), we assume:

    • Usage of npm

    • npm install is called to fetch libraries.

    • npm test is called to run tests.

    • npm run group-id: npm task that echoes the group ID.

    • npm run app-name: npm task that echoes application name.

    • npm run stub-ids: npm task that echoes stub runner IDs.

    • npm run test-apicompatibility: npm task that is executed for api compatibility tests.

    • npm run test-smoke: npm task that is executed for smoke testing.

    • npm run test-e2e: npm task that is executed for end-to-end testing.

    • target is assumed to be the output folder. Put it in .gitignore

  • For .Net (example project):

    • Usage of ASP.NET core

    • dotnet build is called to build the project.

    • dotnet msbuild /nologo /t:UnitTests is called to run unit tests.

    • dotnet msbuild /nologo /t:IntegrationTests is called to run integration tests.

    • dotnet msbuild /nologo /t:Publish /p:Configuration=Release is called to publish a ZIP with a self-contained DLL, together with all manifests and deployment files.

    • dotnet msbuild /nologo /t:GroupId is the npm task that echos the group ID.

    • dotnet msbuild /nologo /t:AppName is the npm task that echos application name.

    • dotnet msbuild /nologo /t:StubIds is the npm task that echos stub runner IDs.

    • dotnet msbuild /nologo /t:ApiCompatibilityTest is run for API compatibility tests.

    • dotnet msbuild /nologo /t:SmokeTests is executed for smoke testing.

    • dotnet msbuild /nologo /t:E2eTests is executed for end-to-end testing.

    • target is assumed to be the output folder. Add it to .gitignore.

4.2. Kubernetes Project Opinions

We use the following opinionated decisions for a Cloud Foundry based project:

  • The application is built by using the Maven or Gradle wrappers.

  • The application is deployed to Kubernetes.

  • The Java Docker image needs to allow passing of system properties through the SYSTEM_PROPS environment variable.

  • For Maven (example project), we assume:

    • Usage of the Maven Wrapper.

    • settings.xml is parametrized to pass the credentials to push code to Artifactory and Docker repositories:

      • M2_SETTINGS_REPO_ID: Server ID for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_USERNAME: User name for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_PASSWORD: Password for Artifactory or Nexus deployment.

      • DOCKER_SERVER_ID: Server ID for Docker image pushing.

      • DOCKER_USERNAME: User name for Docker image pushing.

      • DOCKER_PASSWORD: Password for Docker image pushing.

      • DOCKER_EMAIL: Email for Artifactory or Nexus deployment

    • DOCKER_REGISTRY_URL environment variable: Contains (Overridable - defaults to DockerHub) URL of the Docker registry.

    • DOCKER_REGISTRY_ORGANIZATION environment variable: Contains the organization where your Docker repository resides.

    • Artifacts and Docker image deployment is done by using ./mvnw clean deploy.

    • stubrunner.ids property: To retrieve list of collaborators for which stubs should be downloaded.

    • repo.with.binaries property (injected by the pipeline): Contains the URL to the repo containing binaries (for example, Artifactory).

    • distribution.management.release.id property (injected by the pipeline): Contains the ID of the distribution management. Corresponds to the server ID in settings.xml

    • distribution.management.release.url property (injected by the pipeline): Contains the URL or the repository that contains binaries (for example, Artifactory).

    • deployment.yml contains the Kubernetes deployment descriptor.

    • service.yml contains the Kubernetes service descriptor.

    • running API compatibility tests with the apicompatibility Maven profile.

    • latest.production.version property (injected by the pipeline): Contains the latest production version for the repository (retrieved from Git tags).

    • Running smoke tests on a deployed app with the smoke Maven profile.

    • Running end to end tests on a deployed app with the e2e Maven profile.

  • For Gradle (example project check the gradle/pipeline.gradle file), we assume:

    • Usage of the Gradlew Wrapper.

    • deploy task for artifact deployment.

    • REPO_WITH_BINARIES_FOR_UPLOAD env var (injected by the pipeline): Contains the URL to the repository that contains binaries (for example, Artifactory).

    • M2_SETTINGS_REPO_USERNAME environment variable: User name used to send the binary to the repository that contains binaries (for example, Artifactory).

    • M2_SETTINGS_REPO_PASSWORD environment variable: Password used to send the binary to the repository that contains binaries (for example, Artifactory).

    • DOCKER_REGISTRY_URL environment variable: (Overridable - defaults to DockerHub) URL of the Docker registry.

    • DOCKER_USERNAME environment variable: User name used to send the the Docker image.

    • DOCKER_PASSWORD environment variable: Password used to send the the Docker image.

    • DOCKER_EMAIL environment variable: Email used to send the the Docker image.

    • DOCKER_REGISTRY_ORGANIZATION environment variable: Contains the organization where your Docker repo resides.

    • deployment.yml contains the Kubernetes deployment descriptor.

    • service.yml contains the Kubernetes service descriptor.

    • Running API compatibility tests with the apiCompatibility task.

    • latestProductionVersion property (injected by the pipeline): Contains the latest production version for the repositoryi (retrieved from Git tags).

    • Running smoke tests on a deployed application with the smoke task.

    • Running end to end tests on a deployed application with the e2e task.

    • groupId task to retrieve group ID.

    • artifactId task to retrieve artifact ID.

    • currentVersion task to retrieve the current version.

    • stubIds task to retrieve the list of collaborators for which stubs should be downloaded.

4.3. Ansible Project Opinions

We use the following opinionated decisions for deployment using Ansible:

  • The application is built by using the Maven or Gradle wrappers.

  • The application is deployed via Ansible.

  • Each environment has its own Ansible inventory file with the name of the environment. E.g. for environment test the inventory file would be ${ANSIBLE_INVENTORY_DIR}/test.

  • Given the presence of the Ansible inventory will first try to load a custom playbook from ${ANSIBLE_CUSTOM_PLAYBOOKS_DIR}/${playbook_name}. E.g. for playbook with name [deploy-jvm-service.yml] will search by default for ansible/custom/deploy-jvm-service.yml to apply first. If there’s no such file will continue applying the default playbook /ansible/deploy-jvm-service.yml

  • Uses Docker images for services required by tests

  • Deploys Java applications as JARs

  • For Maven (example project), we assume:

    • Usage of the Maven Wrapper.

    • settings.xml is parametrized to pass the credentials to push code to Artifactory:

      • M2_SETTINGS_REPO_ID contains the server ID for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_USERNAME contains the username for Artifactory or Nexus deployment.

      • M2_SETTINGS_REPO_PASSWORD contains the password for Artifactory or Nexus deployment.

    • Artifacts are deployed by ./mvnw clean deploy.

    • We use the stubrunner.ids property to retrieve list of collaborators for which stubs should be downloaded.

    • repo.with.binaries property (injected by the pipeline): Contains the URL to the repo containing binaries (for example, Artifactory).

    • distribution.management.release.id property (injected by the pipeline): Contains the ID of the distribution management. It corresponds to server ID in settings.xml.

    • distribution.management.release.url property (injected by the pipeline): Contains the URL of the repository that contains binaries (for example, Artifactory).

    • Running API compatibility tests with the apicompatibility Maven profile.

    • latest.production.version property (injected by the pipeline): Contains the latest production version for the repo (retrieved from Git tags).

    • Running smoke tests on a deployed app with the smoke Maven profile.

    • Running end to end tests on a deployed app with the e2e Maven profile.

  • For Gradle (example project check the gradle/pipeline.gradle file), we assume:

    • Usage of the Gradlew Wrapper.

    • A deploy task for artifact deployment.

    • The REPO_WITH_BINARIES_FOR_UPLOAD environment variable (Injected by the pipeline) contains the URL to the repository that contains binaries (for example, Artifactory).

    • The M2_SETTINGS_REPO_USERNAME environment variable contains the user name used to send the binary to the repository that contains binaries (for exampl,e Artifactory).

    • The M2_SETTINGS_REPO_PASSWORD environment variable contains the password used to send the binary to the repository that contains binaries (for example, Artifactory).

    • Running API compatibility tests with the apiCompatibility task.

    • latestProductionVersion property (injected by the pipeline): Contains the latest production version for the repository (retrieved from Git tags).

    • Running smoke tests on a deployed app with the smoke task.

    • Running end to end tests on a deployed app with the e2e task.

    • groupId task to retrieve the group ID.

    • artifactId task to retrieve the artifact ID.

    • currentVersion task to retrieve the current version.

    • stubIds task to retrieve the list of collaborators for which stubs should be downloaded.

5. Customizing the Project

Cloud Pipelines offers a number of ways to customize a Pipelines project:

5.1. Overriding Scripts

Since Cloud Pipelines evolves, you may want to pull the most recent changes to your Cloud Pipelines fork. To not have merge conflicts, the best approach to extending the functionality is to use a separate script with customizations.

When we execute a script that represents a step (for example, a script named build_and_upload.sh), after we source all the deployment and build-specific scripts (such as pipeline-cf.sh and projectType/pipeline-jvm.sh with projectType/pipeline-gradle.sh), we set a hook that lets you customize the behavior. If the script that we run is src//main/bash/build_and_upload.sh, we search for a script in the Cloud Pipelines repository under src/main/bash/custom/build_and_upload.sh, and we source that script just before running any functions.

The following example shows such a customization:

Example 1. custom/build_and_upload.sh
#!/bin/bash

function build() {
    echo "I am executing a custom build function"
}

export -f build

when the build function is called for our Gradle project, instead of calling the Gradle build process, we echo the following text: I am executing a custom build function.

Another customization option is the convention that under src/main/bash/custom, if we find the file equal to the name of the sourced PAAS script (e.g. pipeline-cf.sh) then we will source it after all the other scripts got sourced. That way you can override the way platform related functionality is executed. Example:

Example 2. custom/pipeline-cf.sh
#!/bin/bash

function logInToPaas() {
    echo "I am executing a custom login function"
}

export -f logInToPaas

In order to improve the extensibility, we allow fetching of a tarball with additional files / scripts that should be applied at runtime. That means that if you have a custom implementation of a platform that you would like to apply, instead of maintaining your fork of Cloud Pipeline Scripts, you can create your own repository, containing the necessary files, and produce a tarball with those files. Then you can use ADDITIONAL_SCRIPTS_TARBALL_URL ADDITIONAL_SCRIPTS_REPO_USERNAME, ADDITIONAL_SCRIPTS_REPO_PASSWORD environment variables, to provide the URL of the tarball, together with username and password for basic authentication if necessary (we default the credentials to M2_SETTINGS_REPO_USERNAME:M2_SETTINGS_REPO_PASSWORD env vars). If the ADDITIONAL_SCRIPTS_TARBALL_URL is present, then we will fetch the tarball and unpack it in the src/main/bash directory of Cloud Pipelines Scripts.

If you want to customize the Cloud Pipelines build, you can update the contents of the gradle/custom.gradle build script. That way your customizations will not interfere with the changes in the main part of the code, thus there should be no merge conflicts when pulling the changes from Cloud Pipeline repositories.

6. Building the Project

This section covers how to build the project. It covers:

6.1. Project Setup

In the src/main/bash folder, you can find all the Bash scripts that contain the pipeline logic. These scripts are reused by both the Concourse and Jenkins pipelines.

In the dist folder, you can find the packaged sources of the project. Since the package contains no tests or documentation, it is extremely small and can be used in the pipelines.

In the docs folder, you can find the whole generated documentation of the project.

In the docs-source folder, you can find the sources required to generate the documentation.

6.2. Prerequisites

As prerequisites, you need to have shellcheck, bats, jq and ruby installed. If you use a Linux machine, bats and shellcheck are installed for you.

To install the required software on Linux, type the following command:

$ sudo apt-get install -y ruby jq

If you use a Mac, run the following commands to install the missing software:

$ brew install jq
$ brew install ruby
$ brew install bats
$ brew install shellcheck

6.3. Bats Submodules

To make bats work properly, we needed to attach Git submodules. To have them initialized, either clone the project or (if you have already cloned the project) pull to update it. The following command clones the project:

$ git clone --recursive https://github.com/CloudPipelines/scripts.git

The following commands pull the project:

$ git submodule init
$ git submodule update

If you forget about this step, Gradle runs these steps for you.

6.4. Build and test

Once you have installed all the prerequisites, you can run the following command to build and test the project:

$ ./gradlew clean build

6.5. Generate Documentation

To generate the documentation, run the following command:

$ ./gradlew generateDocs

6.6. Distributions

Cloud Pipelines has a lot of tests, including Git repositories. Those and the documentation “weigh” a lot. That is why, under the dist folder, we publish zip and tar.gz distributions of the sources without tests and documentation. Whenever we release a distribution, we attach a VERSION file to it that contains build and SCM information (build time, revision number, and other details). To skip the distribution generation pass the skipDist property on the command line, as follows:

$ ./gradlew build -PskipDist

6.7. Making a Release

You can run the release task to automatically test the project, build the distributions, change the versions, build the docs, upload the docs to Spring Cloud Static, tag the repo, and then revert the changed versions back to default. To do so, run the following command:

$ ./gradlew release -PnewVersion=1.0.0.RELEASE

7. Releasing the Project

This section covers how to release the project by publishing a Docker image.

8. CI Server Worker Prerequisites

Cloud Pipelines uses Bash scripts extensively. The following list shows the software that needs to be installed on a CI server worker for the build to pass:

 apt-get -y install \
    bash \
    git \
    tar \
    zip \
    curl \
    ruby \
    wget \
    unzip \
    python \
    jq