Pivotal Engineering Journal

Technical articles from Pivotal engineers.

How to Customize a BOSH Stemcell

BOSH Stemcells are Linux-based bootable disk images upon which BOSH applications may be deployed. This blog post describes a process to customize a stemcell (most often used to troubleshoot stemcell boot problems).

Posted on by
Categories:   BOSH   
Edit this post on GitHub.

In this blog post, we describe the procedure we followed in order to create a custom Google Compute Engine (GCE) stemcell with a user cunnie whose ~/.ssh/authorized_keys is pre-populated with a specific public key.

0. Download the Stemcell to Customize

We need to make our changes from a Linux machine (macOS can not mount Linux filesystems as a loopback device). [macOS mount]

We repurpose our BOSH Lite Director as a stemcell-building VM (BOSH Lite is already installed on our macOS workstation, it has mounted our macOS’s filesystem under /vagrant (simplifies stemcell transfer and eliminates running-out-of-space issues), and using it to build our stemcell in no way impedes its ability to perform as a BOSH Director).

 # we use the BOSH Lite directory as a staging point
cd ~/workspace/bosh-lite
 # download the stemcell:
curl -L https://bosh.io/d/stemcells/bosh-google-kvm-ubuntu-trusty-go_agent?v=3263.3 -o tmp/custom_stemcell_3263.3.tgz

1. Use BOSH Lite to Modify stemcell

 # bring up BOSH Lite (if it isn't already running):
vagrant up
vagrant ssh
 # we are now in a Linux box; we will customize our stemcell here:
 # create our mountpoint
sudo mkdir /mnt/stemcell
cd /vagrant/tmp
mkdir stemcell image
cd stemcell
 # unpack the stemcell:
tar xvf ../custom_stemcell_3263.3.tgz
cd ../image
 # unpack the bootable Linux disk image
tar xvf ../stemcell/image
 # connect the bootable Linux disk image to a loopback device
sudo losetup /dev/loop0 disk.raw
 # probe for the new partitions
sudo partprobe /dev/loop0
 # mount the disk image
sudo mount /dev/loop0p1 /mnt/stemcell
 # we like to use chroot to avoid accidentally polluting the BOSH Lite filesystem
sudo chroot /mnt/stemcell /bin/bash
 # update the stemcell to 3263.3.1
echo -n 3263.3.1 > /var/vcap/bosh/etc/stemcell_version
 # we create user 'cunnie' in the 'admin' group (sudo), passwd `c1oudc0w`
useradd -m -G admin -p $(openssl passwd -1 -salt xyz c1oudc0w) cunnie
 # change to the new user's homedir
cd ~cunnie
 # we create the directory
mkdir .ssh
 # install the public key
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9An3FOF/vUnEA2VkaYHoACjbmk3G4yAHE3lXnGpIhz3EV5k4B5RzEFKZnAIFcX18eBjYQIN9xQO0L9xkhlCyrQHrnXBjCDwt/BuQSiRvp3tlx9g0tGyuuJRI5n656Shc7w/g4UbrQWUBdLKjxTT4kTgAdK+1pgDbhAXdPtMwt4D/sz5OEFdf5O5Cp+0spxC+Ctdb94taZhScqB4xt6dRl7bwI28vZdq6Sjg/hbMBbTXzSJ17+ql8LJtXiUHO5W7MwNtKdZmlglOUy3CEIwDz3FdI9zKEfnfpfosp/hu+07/8Y02+U/fsjQyJy8ZCSsGY2e2XpvNNVj/3mnj8fP5cX cunnie@nono.io' > .ssh/authorized_keys
 # set ownerships and permissions
chmod 700 .ssh
chmod 600 .ssh/authorized_keys
chown -R cunnie:cunnie .ssh
 # we're done, hop back to our "normal" BOSH Lite root filesystem
 # unmount our filesystem
sudo umount /mnt/stemcell
 # delete our loop device
sudo losetup -d /dev/loop0
 # tar up our modified Linux bootable disk image
tar czvf ../stemcell/image *

We find it good practice to modify the stemcell number when customizing a stemcell — it prevents many kinds of errors, and well worth the additional time spent re-compiling BOSH releases. In this example, we bump the stemcell number to 3263.3.1:

 # change back to our stemcell directory
cd ../stemcell
 # update the manifest to use a new stemcell number
vi stemcell.MF
-version: '3263.3'
+version: '3263.3.1'
bosh_protocol: 1
sha1: 6dec63560e5ee516e8495eeb39553e81049e19b8
operating_system: ubuntu-trusty
  name: bosh-google-kvm-ubuntu-trusty-go_agent
-  version: '3263.3'
+  version: '3263.3.1'

We create our new, custom stemcell; we put the new version number in the name:

 # create the stemcell
tar czvf ../custom_stemcell_3263.3.1.tgz *
 # exit the Vagrant ssh session

2. Upload the New Stemcell

We upload our new stemcell to our director, which takes ~6 minutes with our internet connection.

 # we upload the stemcell to our BOSH Director
bosh upload-stemcell ~/workspace/bosh-lite/tmp/custom_stemcell_3263.3.1.tgz

3. Deploy

We use the new BOSH Golang CLI to deploy: [Golang CLI]

bosh deploy -d concourse concourse-ntp-pdns-gce.yml \
  -l <(lpass show --note deployments) \
  -l <(curl -L https://raw.githubusercontent.com/cunnie/sslip.io/master/conf/sslip.io%2Bnono.io.yml) \

4. Test

We test our custom stemcell by ssh’ing into our newly-deployed VM (ns-gce.nono.io) using the special user account we created, “cunnie”, with our private key:

ssh -i ~/.ssh/google cunnie@ns-gce.nono.io


[macOS mount] We would be interested if someone has mounted a Linux filesystem to a macOS machine using a loopback device. If the procedure is not too burdensome, we may include it in our blog post.

[Golang CLI] We are using an experimental Golang-based BOSH command line interface (CLI), and the arguments are slightly different than those of canonical Ruby-based BOSH CLI; however, the arguments are similar enough to be readily adapted to the Ruby CLI (e.g. the Golang CLI’s bosh upload-stemcell equivalent to the Ruby CLI’s bosh upload stemcell (no dash)).

The new CLI also allows variable interpolation, with the value of the variables to interpolate passed in via YAML file on the command line. This feature allows the redaction of sensitive values (e.g. SSL keys) from the manifest. The format is similar to Concourse’s interpolation, except that interpolated values are bracketed by double parentheses “((key))”, whereas Concourse uses double curly braces “{{key}}”.

Similar to Concourse, the experimental BOSH CLI allows the YAML file containing the secrets to be passed via the command line, e.g. -l ~/secrets.yml or -l <(lpass show --note secrets)

The Golang CLI is in alpha and should not be used on production systems.