Pivotal Engineering Journal

Technical articles from Pivotal engineers.

dep Coming to Unify Package Management in Go

Golang’s future standard package manager is becoming more usable everyday. Here’s why it’s necessary and how you can try it out today.

Posted on by
Categories:   Golang    Package Management    dep   
Edit this post on GitHub.

Warrior standing atop a mountain, arm raised in victory

dep has just reached an important milestone and there’s enough reason for Golang practitioners to kick the tires on it. I’m going to provide a brief primer on dep, followed by an example, and finally leave you with some expectations for the future.

For those unfamiliar with dep, it’s the community led effort to create the one true standard package manager for the entire Go ecosystem. This is in contrasts with the myriad of tools that exist in Go land today, where there’s no single convention on how to handle dependency management.

How we Got Here

When Golang was first released, its maintainers decided not to prescribe a package manager for the language and thought the community would come up with their own tooling to solve package management and dependency vendoring concerns. And while indeed many tools were created, application developers were never able to standardize on one implementation, and the ecosystem ended up relying on an ever-growing list of different solutions. This created confusion among beginners learning the language, and fuelled never-ending debates among large Go teams regarding dependency best practices. You can read more about the recent history of Golang dependency management if you want to know the gritty details.

Unification

After much discussion, the Golang maintainers and the community decided to finally resolve the debate by creating a spec for a new tool that will become the defacto package manager of the future. dep is the implementation of that spec and the group’s plan is to one day create a proposal to incorporate it into the official Golang toolchain.

dep works in similar way as Bundler for ruby. It uses a manifest file and a lock file, similar to Bundler’s Gemfile and Gemfile.lock. The manifest file can be human edited, and contains a list of the app’s immediate dependencies and where to fetch them. The lock file is autogenerated from the manifest and parsing app code, and contains a list of all of an app’s immediate and transitive dependencies. For each dependency in the lock file, there’s a specific SHA to denote exactly which revision should be used.

An Example

Here’s a sample Go app with a single file main.go:

package main

import "github.com/fatih/color"

func main() {
    color.Red("Hello, world!")
}

You can easily install dep and run the init command:

$ go get -u github.com/golang/dep/cmd/dep
$ dep init

The tool helps create the initial manifest file Gopkg.toml, populating it with the dependency color which dep found in the sample code.

[[constraint]]
  name = "github.com/fatih/color"
  branch = "master"

You can see that it’s following the master branch for the lone package, but if we wanted to we can specify a semver instead (if the package supports it). We can edit the manifest file manually later if we want to add more packages, or use dep ensure <package import path> to help us. For now, lets fetch the dependency:

$ dep ensure

This will fetch the dependency listed in the manifest file using the latest revision in the specified branch. Furthermore, it’ll also fetch the app’s transitive dependencies. All the downloaded packages are placed in the vendor directory, and a lock file detailing all the revisions is autogenerated in Gopkg.lock:

[[projects]]
  branch = "master"
  name = "github.com/fatih/color"
  packages = ["."]
  revision = "62e9147c64a1ed519147b62a56a14e83e2be02c1"

[[projects]]
  name = "github.com/mattn/go-colorable"
  packages = ["."]
  revision = "d228849504861217f796da67fae4f6e347643f15"
  version = "v0.0.7"

[[projects]]
  name = "github.com/mattn/go-isatty"
  packages = ["."]
  revision = "fc9e8d8ef48496124e79ae0df75490096eccf6fe"
  version = "v0.0.2"

[[projects]]
  branch = "master"
  name = "golang.org/x/sys"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  inputs-digest = "801a26ab69076db10a9cdea5b1fe83b567e63bc4754b1c857a5cd094a89e6fab"
  solver-name = "gps-cdcl"
  solver-version = 1

You can see that because of color, dep has pulled in transitive dependencies go-colorable and go-isatty, and sys, which color relies on. If you examine the vendor directory, you’ll see all the packages there:

$ find ./vendor -type d -print
./vendor
./vendor/github.com
./vendor/github.com/fatih
./vendor/github.com/fatih/color
./vendor/github.com/mattn
./vendor/github.com/mattn/go-colorable
./vendor/github.com/mattn/go-colorable/_example
./vendor/github.com/mattn/go-colorable/_example2
./vendor/github.com/mattn/go-isatty
./vendor/golang.org
./vendor/golang.org/x
./vendor/golang.org/x/sys
./vendor/golang.org/x/sys/plan9
./vendor/golang.org/x/sys/unix
./vendor/golang.org/x/sys/unix/linux
./vendor/golang.org/x/sys/windows
./vendor/golang.org/x/sys/windows/registry
./vendor/golang.org/x/sys/windows/svc
./vendor/golang.org/x/sys/windows/svc/debug
./vendor/golang.org/x/sys/windows/svc/eventlog
./vendor/golang.org/x/sys/windows/svc/example
./vendor/golang.org/x/sys/windows/svc/mgr

Great, now you can commit the the manifest file, the lock file, and vendor, and the app would then be properly package-managed and its dependencies vendored.

You can also examined the state of your dependencies:

$ dep status
PROJECT                        CONSTRAINT  VERSION        REVISION  LATEST   PKGS USED
github.com/fatih/color         ^1.5.0      v1.5.0         570b54c   570b54c  1
github.com/mattn/go-colorable  *           v0.0.7         d228849   d228849  1
github.com/mattn/go-isatty     *           v0.0.2         fc9e8d8   fc9e8d8  1
golang.org/x/sys               *           branch master  b90f89a   b90f89a  1

Pretty useful huh? (Especially compared to past tooling)

You (and other developers) can now build and run the app:

$ go build -o example
$ ./example
Hello, world!

Should I use dep today?

The milestone I mentioned at the top of the article explains that from now on the structure of the manifest and lock files are considered stable. This means that future version of dep will be backwards compatible with currently generated files.

dep will very likely become the standard in the near future. If you’re working on a hobbyist project, I suggest that you try out dep today. It might also be educational to convert one of your old projects that isn’t deployed in production. For large projects, I would first take a look at the spec to read up on some important implementation details, including how dependency resolution works when there are multiple versions of the same dependency.

If you’re writing a production app, however, I don’t think you should rely on dep just yet. Its maintainers still warn of missing/changing features. Furthermore, in the roadmap you’ll see there’re still important implementation concerns that the dep developers have yet to working on.

The Future

When it’s done, dep will definitely be a step forward from Golang’s current dependency management workflow. Even if it doesn’t end up being accepted into the official toolchain, it’ll probably become better than what developers are using today. Furthermore, the greatest value of dep might be its potential to push consensus among developers using different management processes today. This will make learning Go easier for beginners, and allow creators to spend less time disagreeing about process, and more time being productive.