Pivotal Engineering Journal

Technical articles from Pivotal engineers.

Faster Pipelines With Compiled BOSH Releases

Compile once, deploy many times.


The Problem

The Pivotal Core Services team is responsible for the cf-mysql release. Deploying this BOSH release requires compiling MariaDB, Galera, and XtraBackup. These packages depend on each other and must be compiled serially, making overall compilation take longer than 40 minutes.

In our Continuous Integration (CI) pipeline, we have a traditional fan-out/fan-in structure where we deploy our release to many environments before running a test against each one.

We treat these environments as disposable clones of one another; we pull them from a pool and clean them before and after each test to make builds reproducible and reduce test pollution. The problem with this strategy is that deploying to a clean environment requires recompilation of all packages, meaning our CI pipeline would often take between 2.5 to 3 hours from start to finish, even when none of our packages were changed.

How Compiled Releases Can Help

BOSH’s compiled releases allow you to compile all of your packages once, export a compressed archive of your release, and then deploy that release to other environments.

We knew that compiled releases would make deployments faster, but we had trouble utilizing them in our test pipeline. Creating the compiled release as part of the pipeline does not make the pipeline faster since the compilation still has to take place. It only becomes valuable if you can use it later to avoid recompiling.

We would save time if we could find a way to compile the release up front and only compile the packages that had changed.

Solution

We created a shared BOSH director where compiled releases are created before deployments in our pipelines. Because the BOSH director is never cleaned up, it’s able to reuse compiled packages that have not changed.

There is now a job at the beginning of our pipeline that compiles the release and downloads that release as a Concourse resource before immediately passing it along and uploading it to another director. This step means that the second bosh director, the one that is always cleaned up, no longer needs to compile anything.

When a package does change, now only that package needs to be recompiled. This means that in the average case, compilation takes around 5 minutes, and only in the worst case will compilation take up to 40 minutes, e.g. when we change the version of MariaDB, which is rare.

Caveats

When BOSH is exporting a compiled release, it creates a lock on the release name. This means that you can not have multiple jobs uploading and exporting releases for the same BOSH release simultaneously. To solve this problem we created a concourse pool resource that acts as a mutex for our BOSH director doing the compilation.

Exporting a release does take a bit of time, so depending on the compilation time of your BOSH release this may not be a win.

Resources