Pivotal Engineering Journal

Technical articles from Pivotal engineers.

Deploy To vSphere NSX-T Opaque Networks Using BOSH

BOSH now allows attaching vSphere deployed VMs to NSX-T’s Opaque Networks

Posted on by
Categories:   BOSH    BOSH CPI    vSphere    NSX-T   
Edit this post on GitHub.

VMware’s vSphere is an Infrastructure as a Service (IaaS) which runs Virtual Machines (VMs). BOSH is a VM orchestrator which automates the creation of VMs. NSX-T is a pluggable Network backend for vSphere (and other hypervisors). NSX-T allows the creation of opaque networks in vSphere, networks whose detail and configuration of the network is unknown to vSphere and which is managed outside vSphere.

With the release of BOSH vSphere CPI v40, users can attach their BOSH-deployed VMs to an NSX-T opaque network.


This blog post describes how to attach a deployed VM to an opaque network. We first deploy a BOSH director attached to an opaque network via the BOSH CLI, then we use the BOSH director to deploy two VMs to the opaque network.

0. Quick Start

Like other vSphere networks, the name of the opaque network as it appears in vSphere should match the name of the network as it appears under the cloud_properties section of the manifest.

In the following screenshot from the vSphere web interface, we can see the opaque network, opaque. The unique icon (a network interface sprouting from a cloud) identifies it as an opaque network.

In our corresponding BOSH Cloud Config we specify that VMs placed in the BOSH western_us network should be attached to the vSphere opaque network. Note that under cloud_properties we make sure to use the opaque network name as it appears in vSphere, opaque:

networks:
- name: western_us
  type: manual
  subnets:
  - range: 192.168.0.0/24
    cloud_properties:
      name: opaque # must match name of network in vSphere

At this point, the seasoned BOSH manifest writer will have enough information to deploy to NSX-T networks, and may find the remainder of this post uninteresting. The remainder of the blog post is directed towards those interested in detailed examples.

1. Double Deployment: BOSH Director First, VMs Second

We will deploy twice to demonstrate BOSH’s opaque network feature: first to demonstrate its effectiveness with the BOSH CLI, second to demonstrate its effectiveness with the BOSH director:

  • Our first deployment is the BOSH Director itself, using the new BOSH Command Line Interface (CLI) (bosh2). Due to an artifact of our environment [artifact] , The BOSH director cannot attach solely to the opaque network; to work around this restriction, we will attach it to two networks: a distributed virtual port group, scarlet, and the NSX-T opaque network, opaque.
  • Our second deployment is two VMs which reside exclusively on the opaque network. A successful BOSH deploy indicates that the opaque network is functioning properly — a BOSH deploy won’t succeed unless the director is able to communicate with the VMs over the network.

1.0 Deploy BOSH Director

We use the this manifest to deploy our BOSH director.

We type the following commands (note the vcenter_ip and vcenter_password have been obscured) to deploy our BOSH director, log into it, and upload a stemcell:

bosh2 create-env bosh-vsphere.yml -v vcenter_ip=vcenter.XXXX -v vcenter_password=XXXX -l vsphere-creds.yml
bosh2 -e 10.85.46.6 --ca-cert <(bosh2 int vsphere-creds.yml --path /director_ssl/ca) alias-env nsx-t

We check our vSphere Web Client to make sure our BOSH director is attached to both networks:

We authenticate against our BOSH director and upload a stemcell:

export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh2 int ./vsphere-creds.yml --path /admin_password`
bosh2 -e nsx-t upload-stemcell https://s3.amazonaws.com/bosh-core-stemcells/vsphere/bosh-stemcell-3363.15-vsphere-esxi-centos-7-go_agent.tgz

Our director is up and running; we are now ready to use our director to deploy two VMs.

1.1. Use BOSH Director to Deploy Two VMs

We upload our Cloud Config:

bosh2 -e nsx-t -n update-cloud-config cloud-config-vsphere.yml

Now we create a minimal deployment consisting of two VMs (BOSH manifest).

bosh2 -e nsx-t -n deploy -d minimal minimal.yml

We make sure the deploy succeeds — a successful deploy indicates the BOSH director and its two VMs are able to communicate over the opaque network:

...
Task 56 done

Succeeded

Addendum: Technical Requirements

  • BOSH vSphere CPI v40+ (v40 tested)
  • vSphere 5.5 (VMware vCenter Server Appliance 6.0.0.20000 tested)
  • NSX-T 1.0+ (1.0.1.0.0.4191070 tested)

Notes

We use bosh-deployment to generate our BOSH director’s manifest. We use this custom script which uses this custom configuration. Much of the complexity derives from the need to create a dual-homed director.

All hosts in a cluster should have at least one physical interface allocated to NSX-T. If not, VMs deployed to hosts with no physical cards allocated to NSX-T will not be able to communicate.

If the operators of a vSphere environment have made the unfortunate decision to identically name multiple networks (e.g. VM Network), the distributed virtual port group will be selected first, followed by the opaque network, followed by the standard switch port group.

Technical details: The BOSH vSphere CPI introduces a new code path which examines the type of network to which the VM is being attached, and, if it’s an opaque network, uses the vSphere API to apply an opaque-specific backing to the VM’s network interface card.

Footnotes

[artifact]

The astute reader may ask, “Multi-homed? Is there something about NSX-T’s opaque networks that require a multi-homed BOSH director?”

[A computer which is attached to more than one network is referred to as a “multi-homed” computer. Routers and gateways are the canonical multi-homed computers.]

The short answer is no, opaque networks do not require a multi-homed BOSH director. The reason we’re deploying a multi-homed director is a side-effect of the manner in which the opaque network was created for our environment: it was created without routable IP addressess and without a router. Had it been created with routable IP addresses and had a default gateway attached, a multi-homed director would not have been necessary.