Skip to content

image-sources.md

Latest commit

 

History

History
222 lines (179 loc) · 11.2 KB

image-sources.md

File metadata and controls

222 lines (179 loc) · 11.2 KB
Raw

Image sources

An image source is the concrete image a deployment builds from or pulls, and this document is about declaring those. It is what you need when the images have to come from somewhere other than their default public location, typically a closed artifact registry in an on-premise installation.

If instead you want to define or customize the base images your applications inherit from, see base and common images.

A CloudHarness deployment takes its images from two places:

  • the images CloudHarness builds from a Dockerfile, whose FROM is resolved through the source_images build arguments;
  • everything it does not build but pulls: the images of the vendored sub-charts (Argo, Kafka UI, Elasticsearch), the ones an application declares in its own values (JupyterHub, Kafka), the ones in its harness configuration (databases, gatekeepers, extra containers), and the ones the generated resources themselves run (database backup, volume migration).

Both kinds are listed together in the generated values-overrides.yaml, described in Discover every image the deployment uses. For mirroring a whole deployment into your own registry, start there.

In both cases the change is made in the deployment configuration of your own project, but the file and the YAML path depend on which of the two kinds the image belongs to.

Change the base image for applications with a Dockerfile

Changing the base image for your application or applications which have a Dockerfile is done through the source_images entry of your values-template.yaml file. This entry defines a mapping between the ARG of your Dockerfile and the value you want to inject. Here is an example of a declared mapping to change the base image for Python based apps and Node based apps:

# deployment-configuration/values-template.yaml
source_images:
  NODE: "mybaseimg:14.5"
  PYTHON: "myotherimage:15.1"

The list of the base image variables and the value they resolve to is automatically generated by the harness-deployment command under the source_images key of both the generated helm/values.yaml and helm/values-overrides.yaml files. You can then copy/paste the entries you need in your values-template.yaml file and tweak the values there.

Note that this goes in values-template.yaml (plural values), which is merged at the root of the generated configuration. A source_images entry placed in value-template.yaml (singular) is silently ignored, because that file holds per-application defaults and the per-application source_images are overwritten by the ARG values read from the application's own Dockerfile.

These variables are passed to the builds as --build-arg, so they only affect the images that CloudHarness builds itself. Images that are merely pulled at runtime are listed separately, as described below.

Change the image of applications which inject their image in the helm chart

Those applications do not provide a Dockerfile, but directly an image which is injected in the helm chart from the helm template, either by a vendored sub-chart or by the application's own values. Rather than looking such a path up by hand, read it off the generated file described next, then redefine it as explained in Make an override permanent.

Discover every image the deployment uses

You do not need to look up any path by hand. Every time harness-deployment runs it writes deployment/helm/values-overrides.yaml next to the generated values.yaml, listing every image the chart pulls, at the exact path Helm reads it from:

# deployment/helm/values-overrides.yaml  (generated)
source_images:                      # base images injected as Dockerfile build arguments
  NODE: 'node:22-alpine'
  PYTHON: 'python:3.12-slim-trixie'
argo-workflows:                     # vendored sub-chart, keyed by its own chart name
  controller:
    image: {registry: quay.io, repository: argoproj/workflow-controller, tag: v3.1.15}
kafka-ui:
  image: {registry: docker.io, repository: provectuslabs/kafka-ui, tag: v0.7.2}
elasticsearch: {image: docker.elastic.co/elasticsearch/elasticsearch, imageTag: 8.17.0}
apps:                               # images declared inline by an application
  events:
    kafka: {image: 'docker.io/apache/kafka:4.0.2'}
  jupyterhub:
    hub:
      image: {name: quay.io/jupyterhub/k8s-hub, tag: '3.2.1'}

The file is a reference, not part of any deployment: no pipeline applies it, since it is regenerated on every run and would only ever restate the values already in effect. It is a real Helm values file though, so you can pass it explicitly to try an image out:

harness-deployment cloudharness . -i events -i argo
# edit deployment/helm/values-overrides.yaml, then
helm template deployment/helm -f deployment/helm/values-overrides.yaml

Because the file is regenerated on every run, use it to discover paths and to try an image out. Make the change permanent by redefining the same path in your own solution, as described below.

The only images left out are the ones CloudHarness builds itself, which are not an image source to redirect: an application's own image and harness.deployment.image. An application that declares a prebuilt image instead of being built (build: false) does not build anything, so its harness.deployment.image is listed like any other pulled image. An application running an image of the build through harness.deployment.image_ref does not build anything either, but its image is still one CloudHarness builds, so it is left out as well.

Every listed path is one that genuinely takes effect: a value shadowed by something of higher precedence is not reported, so editing any entry in the file changes what gets deployed. An application declaring harness.database.image_ref, for instance, runs the task image built under that reference, so its harness.database.<type>.image is left out rather than shown as an override that would do nothing. This is what makes the file usable for repointing a deployment wholesale, for instance with yq. An image shared by many applications, such as the gatekeeper, is listed once rather than repeated on each of them.

Make an override permanent

Which file to edit depends on where the image sits in the generated configuration. The two cases are distinct, and using the wrong one fails silently.

Vendored sub-chart images go at the root of deployment-configuration/values-template.yaml, under the sub-chart key, using the same paths as the generated file. Only the fields you list are overridden, the rest keep the vendored defaults:

# deployment-configuration/values-template.yaml
argo-workflows:
  controller:
    image:
      tag: v3.7.2
kafka-ui:
  image:
    registry: myregistry.io
    repository: mirror/kafka-ui

Images an application declares inline (everything the generated file lists under apps.<app>) are overridden by overriding the application: create the same values file in your own solution and redefine the path, without the apps.<app> prefix.

# applications/jupyterhub/deploy/values.yaml, in your own solution
singleuser:
  image:
    name: myregistry.io/my/singleuser
    tag: "9.9.9"

Do not put an apps: block in values-template.yaml to this end: the generator resets that key before loading the applications, so such an override is silently discarded even though the generated file displays the image under apps.<app>.

A vendored sub-chart is addressed by the name in its own Chart.yaml, which is not always the CloudHarness application name. This is how Helm passes parent values down to a sub-chart, so using the application directory name instead (argo: rather than argo-workflows:) silently does nothing. The current mapping is:

Application Sub-chart key
argo argo-workflows
events kafka-ui
elasticsearch elasticsearch

Here are the identified images inside of CloudHarness which do not come from a Dockerfile, grouped by how each one is overridden. The paths are the ones shown by values-overrides.yaml, i.e. root-relative in the generated values.yaml.

Vendored sub-charts, overridden at the root of values-template.yaml:

Application Image path
Argo controller argo-workflows.controller.image.{registry, repository, tag}
Argo executor argo-workflows.executor.image.{registry, repository, tag}
Argo server argo-workflows.server.image.{registry, repository, tag}
Events UI kafka-ui.image.{registry, repository, tag}
Elasticsearch elasticsearch.image and elasticsearch.imageTag

Images declared inline by an application, overridden by redefining the path in that application's deploy/values.yaml in your own solution (drop the apps.<app> prefix):

Application Image path
Events kafka apps.events.kafka.image
JupyterHub hub apps.jupyterhub.hub.image.{name, tag}
JupyterHub singleuser apps.jupyterhub.singleuser.image.{name, tag}
JupyterHub proxy apps.jupyterhub.proxy.chp.image.{name, tag}
JupyterHub scheduling apps.jupyterhub.scheduling.userScheduler.image.{name, tag}
JupyterHub prepuller apps.jupyterhub.prePuller.hook.image.{name, tag}
Neo4J reverseProxy apps.neo4j.reverseProxy.image
Sentry redis apps.sentry.redis.image

Images an application pulls as part of its harness configuration. These are set per application the same way, or for every application at once through value-template.yaml:

Image Image path
MongoDB apps.<app>.harness.database.mongo.image
Neo4J apps.<app>.harness.database.neo4j.image
Postgres apps.<app>.harness.database.postgres.image
Database built by CloudHarness not an image source: see harness.database.image_ref
extra containers apps.<app>.harness.deployment.extraContainers.<name>.image
prebuilt application image (build: false only) apps.<app>.harness.deployment.image
Application image built by CloudHarness not an image source: see harness.deployment.image_ref

Images the generated resources run themselves, overridden at the root of values-template.yaml:

Image Image path
Gatekeeper source_images.GATEKEEPER
Database backup cronjob backup.image
Volume migration job volumeMigration.image
Volume migration wait init container volumeMigration.wait.image

A gatekeeper is generated for every secured application, so its image is configured once, at source_images.GATEKEEPER, instead of being repeated on each application. A single application can still opt out through apps.<app>.harness.proxy.gatekeeper.image, which is unset by default and so does not show up in values-overrides.yaml until you set it.

The root proxy.gatekeeper.image used to override the gatekeeper of every application. It is no longer read: set source_images.GATEKEEPER instead. harness-deployment warns if it is still set.

The JupyterHub hub image is a special case: JupyterHub's own template renders the hub pod from the image CloudHarness builds for the application, so apps.jupyterhub.hub.image is listed and can be set but does not change the container that runs.