HAProxy is an optional load balancer included in the canonical open source Cloud Foundry deployment. Its intended use is on IaaSes (Infrastructures as a Service) that do not offer built-in load balancers . On vSphere, this means without the optional network virtualization solutions, NSX-T and NSX-V. This blog post describes how to assign an IPv6 address to an HAProxy load balancer in a Cloud Foundry deployment.
Users following this blog post should be familiar with BOSH, BOSH’s manifest operations files, IPv6, and deploying Cloud Foundry using cf-deployment.
We set up our DNS (Domain Name System) records as shown in the table below
(we’re using the same domain,
cf.nono.io, for both system and app domains, and
this is probably not a good
|Hostname||IPv4 Address||IPv6 Address|
Our DNS server uses a BIND-format (Berkeley Internet Name Domain)  file, and here are the raw entries (tweaked for readability):
cf.nono.io. A 10.0.250.10 AAAA 2601:646:100:69f5::10 *.cf.nono.io. A 10.0.250.10 AAAA 2601:646:100:69f5::10
Note that the IPv4 address (10.0.250.10) is in a private
network and not reachable from
the internet, but the IPv6 address (2601:646:100:69f5::10) is in a public one.
Indeed, the IPv6 address is in a
/64 subnet, one of 8 subnets that Comcast has
Only IPv6-enabled clients can reach our Cloud Foundry; IPv4-only clients can’t.
Let’s download what we need and log in to our BOSH Director:
mkdir -p ~/workspace cd ~/workspace git clone https://github.com/cloudfoundry/cf-deployment git clone https://github.com/cunnie/deployments git clone https://github.com/cloudfoundry/cf-acceptance-tests.git export BOSH_ENVIRONMENT=vsphere # our vSphere BOSH Director's environment alias bosh login cd deployments
cf-deploymentcontains the BOSH manifest and manifest operations files to deploy Cloud Foundry.
deploymentscontains additional scripts and manifest operations files to deploy Cloud Foundry with an IPv6 HAProxy.
cf-acceptance-testscontains a sample CF app that we can push to test IPv6 connectivity.
Normally we’d add our IPv6 Network to our Cloud Config’s YAML file and then type
bosh update-cloud-config, but that’s not what we’re going to do in this
case. Instead, we’re going to use the Cloud Config included cf-deployment and
use a custom manifest operations files to add our IPv6 network to it.
Here are the relevant lines from our script:
# We don't use the primary cloud config (we already have one); instead, # we set up a secondary config bosh update-config \ --non-interactive \ --type cloud \ --name cf \ <(bosh int \ -o $DEPLOYMENTS_DIR/cf/cloud-config-operations.yml \ -l $DEPLOYMENTS_DIR/cf/cloud-config-vars.yml \ $DEPLOYMENTS_DIR/../cf-deployment/iaas-support/vsphere/cloud-config.yml)
The environment variable
$DEPLOYMENTS_DIR refers to the directory which
contains our customizations (
Let’s talk about our customizations; We customize as follows:
We set up our variables in
The file is uninteresting—it sets up the IPv4 subnets for the three AZs, two
of which (
az3) will be immediately removed as part of our
We apply our manifest operations (
We remove two of the AZs, leaving one,
az1. We don’t want a sprawling
deployment; Our vSphere cluster is too small. We can only fit one AZ.
az1's’ subnet to include a static IP address, 10.0.250.10, which
will be assigned to the HAproxy, and which is the DNS
A record for
We introduce the IPv6 subnet, 2601:646:100:69f5::/64, including a static
IPv6 address, 2601:646:100:69f5::10, which will be assigned to the HAproxy,
and which is the DNS
AAAA record for
As long as BOSH DNS is colocated on all deployed VMs (which is the default BOSH Runtime Config), you shouldn’t need to make changes.
Deploying the VMs requires one command,
bosh deploy, but with many options.
Here are the relevant lines from our deploy shell
bosh \ -e vsphere \ -d cf \ deploy \ --no-redact \ $DEPLOYMENTS_DIR/../cf-deployment/cf-deployment.yml \ -l <(lpass show --note cf.yml) \ -v system_domain=cf.nono.io \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/scale-to-one-az.yml \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-haproxy.yml \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-latest-stemcell.yml \ -o $DEPLOYMENTS_DIR/cf/letsencrypt.yml \ -o $DEPLOYMENTS_DIR/cf/haproxy-on-ipv6.yml \ -v haproxy_private_ip=10.0.250.10 \ --var-file=star_cf_nono_io_crt=$HOME/.acme.sh/\*.cf.nono.io/fullchain.cer \ --var-file=star_cf_nono_io_key=$HOME/.acme.sh/\*.cf.nono.io/\*.cf.nono.io.key \
-o $DEPLOYMENTS_DIR/cf/haproxy-on-ipv6.yml is the most important line for
deploying HAProxy with IPv6. It’s a BOSH manifest operations
file, and will be covered in the
-v haproxy_private_ip=10.0.250.10 sets the HAProxy to the IPv4 address
pointed to by
-v system_domain=cf.nono.io sets the system domain, which should point to
the IPv4 address above.
<(lpass show --note cf.yml) is a YAML file that sets only one property,
cf_admin_password. The author prefers to have a easy-to-remember admin
password rather than needing to extract the password from a BOSH-generated
CredHub secret. This line, this YAML file, is unnecessary; you may safely skip
Any parameter beginning with
$DEPLOYMENTS_DIR/../cf-deployment refers to a
file in cf-deployment GitHub
repo, which means it’s a canonical manifest or manifest operations file.
-o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-haproxy.yml. Enable
HAProxy, otherwise there’s no VM to which we can assign an IPv6 address.
-o $DEPLOYMENTS_DIR/cf/letsencrypt.yml is a custom manifest operations file
the deploys HAProxy with a valid, commercial TLS certificate issued by Let’s
Encrypt. Its usage outside the scope of this
document, but if enough are interested the author may write a blog post to
describe how to deploy Cloud Foundry with a free Let’s Encrypt wildcard
certificate. You may safely skip this file.
--var-file=star_cf_nono_io_key are also
related to the Let’s Encrypt TLS certificate. You may skip them.
The 23-line manifest operations file is too long to inline in this blog post, but let’s review key points:
# haproxy has an IPv6 address - type: replace path: /instance_groups/name=haproxy/networks/name=PAS-IPv6? value: name: PAS-IPv6 static_ips: - 2601:0646:0100:69f5:0000:0000:0000:0010
# configure haproxy to bind to the IPv6 in6addr_any address "::" - type: replace path: /instance_groups/name=haproxy/jobs/name=haproxy/properties/ha_proxy/binding_ip? value: "::" # configure haproxy to bind to both IPv4 & IPv6 interfaces - type: replace path: /instance_groups/name=haproxy/jobs/name=haproxy/properties/ha_proxy/v4v6? value: true
We push an application, then
curl the application over IPv6.
cf api api.cf.nono.io --skip-ssl-validation # replace your CF API endpoint here cf login cf target -o system -s system # or whichever org & space you want to use cf push dora -p ../cf-acceptance-tests/assets/dora curl -6 -k https://dora.cf.nono.io # replace your application domain here # "Hi, I'm Dora!"
A reasonable question is, “do browsers have trouble consistently navigating to
HAProxy’s IPv6 address given that
*.cf.nono.io resolve to both IPv4 and IPv6
We haven’t experienced trouble with either the Google Chrome or Firefox browsers. Similarly, we haven’t experienced problems with the Cloud Foundry CLI (Command Line Interface); however, we suspect that if we had a machine on our internal network whose IPv4 addressed matched the IPv4 address of our HAProxy (i.e. 10.0.250.10), then connectivity would be erratic.
Assigning the HAProxy VM’s default gateway to the IPv4 or IPv6 interface is a nuanced decision. If the BOSH Director is in a different IPv4 subnet, you must assign the default gateway to the IPv4 interface so that the VM can reach the BOSH director and retrieve its configuration and releases (as is the case in our setup).
If you have IPv6 router advertisements on your IPv6 subnet, your IPv6 interface will pick up its default gateway for free. If you do not have router advertisements, you will need a more complex manifest operations file; check here for inspiration.
For best results, use Ubuntu Xenial stemcells 621.51 or greater; IPv6 router advertisement are enabled by default.
Most IaaSes (Amazon Web Services (AWS), Google Cloud, and Microsoft Azure) offer load balancers, and they typically charge $200+/yr for the service.
The author is of the opinion is that load balancers are often an unnecessary expense, and many users are better off with a carefully-monitored single webserver rather than a fleet of webservers fronted by a load balancer. To quote Andrew Carnegie:
put all your eggs in one basket, and then watch that basket
BIND is an old-school DNS server. Nowadays most users are better off using DNS-as-a-service (e.g. Amazon’s Route 53, Google Cloud DNS), but for the hardcore, deploying your own DNS servers is a wonderfully addictive option. The author, for example, has been running his own DNS servers since 1996, and currently maintains 4 DNS servers on 4 different IaaSes (Azure, AWS, Google, Hetzner) on 3 different continents (America, Asia, Europe).
The author is an unapologetic DNS fanboy, and had Paul Mockapetris, the author of the original DNS RFC, autograph his laptop with a permanent marker.