The client libraries that Kubernetes ships are meant to be imported, and you definitely don’t need this post explaining how to import them in your Golang based project. A simple go get ... should do the trick. But, what about the packages that are not meant to be imported? Or the ones that cannot be imported because of “technical reasons” ? Could you simply add them to your import statements in the .go file, and the go binary will do the right thing when you build the code? Well, let’s find that out!

Usecase

I have a project, and in it, the imported packages look like this:

import (
 "crypto/x509"
 "fmt"
 "io/ioutil"
 "log"
 "path/filepath"

 "github.com/spf13/cobra"
 "k8s.io/client-go/util/cert"
 "k8s.io/client-go/util/keyutil"
 "k8s.io/kubernetes/test/utils"
)

source

So it consists of a couple of inbuilt packages, and then the CLI library Cobra, a couple of client-go packages. The only out of the ordinary thing is the k8s.io/kubernetes/test/utils package.

Importing

Let us tidy up the dependencies before building the code:

$ go get github.com/spf13/cobra \
>        k8s.io/client-go/util/cert \
>        k8s.io/client-go/util/keyutil \
>        k8s.io/kubernetes/test/utils
go: downloading k8s.io/client-go v1.5.2
go: downloading github.com/spf13/cobra v1.1.3
go: downloading k8s.io/kubernetes v1.21.1
go: downloading k8s.io/client-go v0.21.1
go get: k8s.io/kubernetes@v1.15.0-alpha.0 updating to
        k8s.io/kubernetes@v1.21.1 requires
        k8s.io/api@v0.0.0: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0

Now, this is a weird error:

k8s.io/api@v0.0.0: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0

tl;dr Give me the solution

NOTE: Please read the “Caveats” section before you venture into using this solution.

Save the following bash script in download-deps.sh:

#!/bin/bash

VERSION=${1#"v"}
if [ -z "$VERSION" ]; then
  echo "Please specify the Kubernetes version: e.g."
  echo "./download-deps.sh v1.21.0"
  exit 1
fi

set -euo pipefail

# Find out all the replaced imports, make a list of them.
MODS=($(
  curl -sS "https://raw.githubusercontent.com/kubernetes/kubernetes/v${VERSION}/go.mod" |
    sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
))

# Now add those similar replace statements in the local go.mod file, but first find the version that
# the Kubernetes is using for them.
for MOD in "${MODS[@]}"; do
  V=$(
    go mod download -json "${MOD}@kubernetes-${VERSION}" |
      sed -n 's|.*"Version": "\(.*\)".*|\1|p'
  )

  go mod edit "-replace=${MOD}=${MOD}@${V}"
done

go get "k8s.io/kubernetes@v${VERSION}"
go mod download

Make the script usable:

chmod u+x download-deps.sh

Use the following script. It will update the go.mod file with the required dependencies:

$ ./download-deps.sh v1.21.0
go: downloading k8s.io/kubernetes v1.21.0
go get: added k8s.io/kubernetes v1.21.0

Now you can build your code! Keep reading further to understand why this happens.

Kudos 👏 to Andy Bursavich for writing the aforementioned script. I took the script from this comment and made minor modifications.

Reasoning

The error k8s.io/api@v0.0.0: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0 is caused because, in Kubernetes’s go.mod, you will find the following snippet:

require (
...
 k8s.io/api v0.0.0
 k8s.io/apiextensions-apiserver v0.0.0
 k8s.io/apimachinery v0.0.0
 k8s.io/apiserver v0.0.0
...
)

replace (
...
 k8s.io/api => ./staging/src/k8s.io/api
 k8s.io/apiextensions-apiserver => ./staging/src/k8s.io/apiextensions-apiserver
 k8s.io/apimachinery => ./staging/src/k8s.io/apimachinery
 k8s.io/apiserver => ./staging/src/k8s.io/apiserver
...
)

In the Kubernetes repository, the packages listed under require directive are tagged with a pseudo-version v0.0.0 but then are replaced with a local code in the staging directory. Now replace directive works fine when building Kubernetes itself but won’t work when someone is importing it. The Golang documentation says the following about replace directive:

replace directives only apply in the main module’s go.mod file and are ignored in other modules.

source

Generally, the pseudo-version has a format v0.0.0-Time-commit_id, but in this case, it practically is a dead end. Why is Kubernetes doing such a thing in their project? Why not tag with an actual pseudo-version format? Well, because the code is available locally. And the directories in the staging folder are published as standalone projects for folks to use later on.

So the net effect is that I am trying to import code using go get, but it is tagged with a pseudo-version v0.0.0 and hence go get fails. The solution to this problem is to find the correct versions of those dependencies yourself and add them to your repo’s go.mod under replace directive.

This is precisely what the script is doing.

Final Go Mod

So in our project, the go.mod looks like this from the successful imports before running the script:

module foobar

go 1.16

require (
        github.com/spf13/cobra v1.1.3 // indirect
        k8s.io/client-go v0.21.1 // indirect
)

After running the script, this is how it looks like:

module foobar

go 1.16

require (
        github.com/spf13/cobra v1.1.3 // indirect
        k8s.io/client-go v0.21.1 // indirect
        k8s.io/kubernetes v1.21.0 // indirect
)

replace k8s.io/api => k8s.io/api v0.21.0
replace k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.21.0
replace k8s.io/apimachinery => k8s.io/apimachinery v0.21.1-rc.0
replace k8s.io/apiserver => k8s.io/apiserver v0.21.0
...

Caveats

Kubernetes upstream cautions against using the packages as we have done in this blog (Thanks, Dims, for bringing this to my notice):

To use Kubernetes code as a library in other applications, see the list of published components. Use of the k8s.io/kubernetes module or k8s.io/kubernetes/... packages as libraries is not supported.

source

If you are importing a package that is not supported as a library by the upstream community, you might be up for a catch-up going forward. The unsupported, non-library packages are subject to API definition changes without any public warning. And if your production code relies on such a package, it might impact you. At that point, you have no right to go and accuse upstream of not being public about the changes.

That being said, if there are any libraries you identify that are widely used and should be published, bring them to the notice of the upstream community. They will surely help in publishing them. This helps the broader community by consuming the libraries from a widely published place and saving anyone and everyone from catching up. But unless they are not published as libraries consuming anything else is a call for a lot of code churn.