Image Vector

The Gardener components are deploying several different container images into the garden, seed, and the shoot clusters. The image repositories and tags are defined in a central image vector file. Obviously, the image versions defined there must fit together with the deployment manifests (e.g., some command-line flags do only exist in certain versions).

Example

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.4"
  targetVersion: "1.20.x"
  architectures:
  - amd64
  - arm64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  ref: registry.k8s.io/pause:3.5
  targetVersion: ">= 1.21"
  architectures:
  - amd64
  - arm64

That means that Gardener will use the pause-container with tag 3.4 for all clusters with Kubernetes version 1.20.x, and the image with ref registry.k8s.io/pause:3.5 for all clusters with Kubernetes >= 1.21.

NOTE

As you can see, it is possible to provide the full image reference via the ref field. Another option is to use the repository and tag fields. tag may also be a digest only (starting with sha256:...), or it can contain both tag and digest (v1.2.3@sha256:...).

Architectures

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: registry.k8s.io/pause
  tag: "3.5"
  architectures:
  - amd64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  ref: registry.k8s.io/pause:3.5
  architectures:
  - arm64
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  ref: registry.k8s.io/pause:3.5
  architectures:
  - amd64
  - arm64

architectures is an optional field of image. It is a list of strings specifying CPU architecture of machines on which this image can be used. The valid options for the architectures field are as follows:

• amd64 : This specifies that the image can run only on machines having CPU architecture amd64.
• arm64 : This specifies that the image can run only on machines having CPU architecture arm64.

If an image doesn't specify any architectures, then by default it is considered to support both amd64 and arm64 architectures.

Overwriting Image Vector

In some environments it is not possible to use these "pre-defined" images that come with a Gardener release. A prominent example for that is Alicloud in China, which does not allow access to Google's GCR. In these cases, you might want to overwrite certain images, e.g., point the pause-container to a different registry.

⚠️ If you specify an image that does not fit to the resource manifest, then the reconciliations might fail.

In order to overwrite the images, you must provide a similar file to the Gardener component. If your registry requires a custom CA, you can also supply a caBundle in the same file (see CA Bundle for Private Registries):

images:
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  repository: my-custom-image-registry/pause
  tag: "3.4"
  version: "1.20.x"
- name: pause-container
  sourceRepository: github.com/kubernetes/kubernetes/blob/master/build/pause/Dockerfile
  ref: my-custom-image-registry/pause:3.5
  version: ">= 1.21"

IMPORTANT

When the overwriting file contains ref for an image but the source file doesn't, then this invalidates both repository and tag of the source. When it contains repository for an image but the source file uses ref, then this invalidates ref of the source.

For gardenlet, you can create a ConfigMap containing the above content and mount it as a volume into the gardenlet pod. Next, specify the environment variable IMAGEVECTOR_OVERWRITE, whose value must be the path to the file you just mounted. The approach works similarly for gardener-operator.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gardenlet-images-overwrite
  namespace: garden
data:
  images_overwrite.yaml: |
    images:
    - ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gardenlet
  namespace: garden
spec:
  template:
    spec:
      containers:
      - name: gardenlet
        env:
        - name: IMAGEVECTOR_OVERWRITE
          value: /imagevector-overwrite/images_overwrite.yaml
        volumeMounts:
        - name: gardenlet-images-overwrite
          mountPath: /imagevector-overwrite
      volumes:
      - name: gardenlet-images-overwrite
        configMap:
          name: gardenlet-images-overwrite

CA Bundle for Private Registries

If the registry used in the image vector overwrite is backed by a custom CA (i.e., it presents a TLS certificate that is not trusted by the default system roots), you can supply the CA bundle alongside the image overrides in the same file:

images:
- name: pause-container
  repository: my-private-registry.example.com/pause
  tag: "3.5"
caBundle:
  inline: |
    -----BEGIN CERTIFICATE-----
    <certificate data>
    -----END CERTIFICATE-----

The caBundle.inline field accepts a PEM-encoded CA certificate bundle. If set for gardenlet, it will distribute the CA certificate bundle to the worker nodes of every shoot managed by this seed, so that containerd on those nodes trusts the registry when pulling images.

IMPORTANT

Only the gardenlet's image vector caBundle is distributed to shoot worker nodes. Setting caBundle in the gardener-operator's image vector override does not install the CA on any nodes. For the runtime/soil cluster itself, custom CAs must be installed on nodes by other means (e.g., via the cloud provider's node configuration API).

If extensions deployed by gardenlet use container images from a private registry with a custom CA, that CA must also be included in the gardenlet's image vector caBundle. This is the only mechanism for installing custom CAs on shoot worker nodes — the caBundleSecretRef on an Extension or ControllerDeployment only covers TLS for pulling the Helm chart archive and does not affect containerd's trust store on the nodes. In practice this means operators should consolidate all private registry CAs (both their own and those required by extensions) into the gardenlet's IMAGEVECTOR_OVERWRITE caBundle.

NOTE

The certificate must be a CA certificate and must not be expired. Gardener validates both conditions when loading the image vector overwrite file.

Image Vectors for Dependent Components

Gardener is deploying a lot of different components that might deploy other images themselves. These components might use an image vector as well. Operators might want to customize the image locations for these transitive images as well, hence, they might need to specify an image vector overwrite for the components directly deployed by Gardener.

It is possible to specify the IMAGEVECTOR_OVERWRITE_COMPONENTS environment variable to Gardener that points to a file with the following content:

components:
- name: etcd-druid
  imageVectorOverwrite: |
    images:
    - name: etcd
      tag: v1.2.3
      repository: etcd/etcd

Gardener will, if supported by the directly deployed component (etcd-druid in this example), inject the given imageVectorOverwrite into the Deployment manifest. The respective component is responsible for using the overwritten images instead of its defaults.

Helm Chart Image Vector

Some Gardener components might also deploy packaged Helm charts which are pulled from an OCI repository. The concepts are the very same as for the container images. The only difference is that the environment variable for overwriting this chart image vector is called IMAGEVECTOR_OVERWRITE_CHARTS.

If the chart OCI registry uses a custom CA, you can supply it via caBundle.inline in the same override file. Gardener will automatically include this CA in the TLS trust store whenever it pulls an OCI Helm chart.

IMPORTANT

The caBundle here (like caBundleSecretRef on an Extension or ControllerDeployment) only secures the TLS connection used to pull the chart archive from the OCI registry. It does not install that CA on shoot worker nodes. Container images referenced inside those Helm charts will only be trusted by containerd if the CA is also present in the gardenlet's IMAGEVECTOR_OVERWRITE caBundle (see CA Bundle for Private Registries).

images:
- name: my-chart
  repository: my-private-registry.example.com/charts/my-chart
  tag: "1.2.3"
caBundle:
  inline: |
    -----BEGIN CERTIFICATE-----
    <certificate data>
    -----END CERTIFICATE-----