Download PDF

Deploying VCF Addons in an Air-Gapped Environment

Julius M. Nicolescu

May 2026

Overview

This document describes how to make VMware Cloud Foundation (VCF) CLI plugins and VKS Standard Packages available in a network-isolated (air-gapped) environment. In an air-gapped deployment, the nodes and management tools have no direct access to Broadcom’s public package registries. All software must be mirrored to a local container registry and then configured as the source for addon installation. This guide uses Harbor — a CNCF-graduated, OCI-compliant container registry — as the local mirror. Harbor stores both the VCF CLI plugin inventory artifacts and the full VKS Standard Packages bundle (~409 OCI objects), and serves them to the Supervisor’s addon controller over HTTPS using its own TLS certificate.

The procedure is divided into three parts:

Part Summary
Part I Download VCF CLI and its plugins on an internet-connected machine, then upload them to Harbor
Part II Download the VKS Standard Packages bundle, mirror it to Harbor
Part III Register Harbor with the vSphere Supervisor and redirect the addon repository to the local mirror

A validation step at the end confirms that addon images are being pulled from Harbor and not from the internet.


Reference Environment

The procedures and configurations in this document were validated against the following platform versions:

Component Version
VMware Cloud Foundation 9.0.2.0
Supervisor v1.32.9+vmware.2-fips.vsc9.0.2.0100-25262241
vSphere Kubernetes Service 3.6.0+v1.35
Kubernetes Release v1.35.2+vmware.1-vkr.3

Architecture

┌────────────────────────────────────┐       ┌──────────────────────────────────────────────┐
│  Internet-Connected Machine        │       │  Air-Gapped Environment                      │
│                                    │       │                                              │
│  1. Download VCF CLI               │       │  ┌──────────────────┐                        │
│  2. vcf plugin download-bundle ────┼──tar──┼─►│  Harbor Registry │                        │
│  3. imgpkg copy -b ────────────────┼──tar──┼─►│  (local mirror)  │                        │
│                                    │       │  └──────┬───────────┘                        │
└────────────────────────────────────┘       │         │                                    │
                                             │  ┌──────▼───────────┐                        │
                                             │  │ vSphere          │                        │
                                             │  │ Supervisor       │◄── kubectl / vcf CLI   │
                                             │  └──────┬───────────┘                        │
                                             │         │ AddonRepository (override)         │
                                             │  ┌──────▼───────────┐                        │
                                             │  │ VKS Guest        │                        │
                                             │  │ Cluster          │                        │
                                             │  └──────────────────┘                        │
                                             └──────────────────────────────────────────────┘

Harbor Registry

Harbor is an open-source, CNCF-graduated container registry that provides OCI-compliant artifact storage, role-based access control, vulnerability scanning, and content trust over a standard HTTPS endpoint. In this deployment, Harbor is the central artifact store for all VCF and VKS software components inside the air-gapped environment.

Why Harbor for VKS Air-Gapped Deployments

Harbor is well-suited to this role for several reasons:

Harbor Projects Used in This Guide

Two Harbor projects are created during this procedure:

Project Name Visibility Contents Used by
VCF CLI Plugins vcf-cli Public VCF CLI plugin inventory OCI artifacts VCF CLI on any workstation inside the air-gapped network
VKS Standard Packages vcf-packages Public Full VKS Standard Packages bundle (~409 OCI artifacts) Supervisor addon controller; VKS cluster nodes

Both projects are configured as public in this guide, allowing anonymous pulls without authentication. This simplifies the Supervisor AddonRepository configuration — no pull secret is required. If your security policy mandates private projects, create a Kubernetes Secret with Harbor credentials and reference it in the AddonRepository spec under spec.fetch.imgpkgBundle.secretRef.

Repository Layout in Harbor

After completing both mirror operations, the Harbor instance has the following repository structure:

harbor.example.com/
├── vcf-cli/
│   └── plugins/
│       ├── plugin-inventory:latest          ← VCF CLI plugin index artifact
│       ├── addon/                           ← addon plugin OCI artifact
│       ├── cluster/                         ← cluster plugin OCI artifact
│       └── ...                             ← remaining plugins
│
└── vcf-packages/
    └── 3.6.0-20260521/
        └── vks-standard-packages:3.6.0-20260521   ← full bundle (~409 artifacts)

VCF CLI reads the plugin-inventory:latest artifact at vcf-cli/plugins/ to enumerate available plugins. The Supervisor addon controller reads the bundle root at vcf-packages/3.6.0-20260521/vks-standard-packages to enumerate available addons.

Robot Accounts

For production environments, replace the $HARBOR_USER/$HARBOR_PASS admin credentials used in this guide with robot accounts. Robot accounts are scoped to specific Harbor projects, can be granted push-only or pull-only access, and issue token-based credentials that can be rotated independently of the admin account.

Create a robot account with push and pull access to both VCF projects via the Harbor REST API:

curl -k -u "$HARBOR_USER:$HARBOR_PASS" \
  -H "Content-Type: application/json" \
  -X POST "https://$HARBOR/api/v2.0/robots" \
  -d '{
    "name": "vcf-mirror-robot",
    "duration": -1,
    "description": "Push/pull access for VCF air-gapped mirroring",
    "permissions": [
      {
        "kind": "project",
        "namespace": "vcf-packages",
        "access": [
          {"resource": "repository", "action": "push"},
          {"resource": "repository", "action": "pull"}
        ]
      },
      {
        "kind": "project",
        "namespace": "vcf-cli",
        "access": [
          {"resource": "repository", "action": "push"},
          {"resource": "repository", "action": "pull"}
        ]
      }
    ]
  }' | jq '{name: .name, secret: .secret}'

The response includes a secret field — this is the robot account password and is shown only once. Use robot$vcf-mirror-robot as the username (Harbor prefixes all robot account names with robot$). Store the secret in a vault or a chmod 600 secrets file, then substitute it for $HARBOR_PASS throughout this guide.

Harbor CA Certificate Trust Chain

Harbor’s TLS certificate is signed by a private CA in most on-premises deployments. The CA certificate must be trusted by four distinct consumers, each using a different trust mechanism:

Consumer Trust mechanism Where covered
imgpkg on the mirroring machine OS trust store + --registry-ca-cert-path flag Part II, Step 3
docker on the mirroring machine /etc/docker/certs.d/$HARBOR/ca.crt Part II, Step 3
vSphere Supervisor Container Registry registration in the vSphere Client UI Part III, Step 1
VKS cluster nodes (at boot) osConfiguration.trust.additionalTrustedCAs in the Cluster manifest Cluster creation guide

All four trust steps are required. Missing any one of them produces a TLS handshake failure at the point that consumer first contacts Harbor — a pod ImagePullBackOff on a VKS cluster node, for example, almost always means the node CA trust step was skipped.


Prerequisites

Before starting, ensure the following are available:

Set the following environment variables before running any commands. Replace the placeholder values with your actual environment details:

export HARBOR='harbor.example.com'
export HARBOR_USER='admin'
export HARBOR_PASS='<your-harbor-password>'
export SUPERVISOR_IP='<supervisor-api-endpoint-ip>'
export CLUSTER_NAMESPACE='<vsphere-namespace>'
export VCF_CLI_VSPHERE_USERNAME='administrator@vsphere.local'
export VCF_CLI_VSPHERE_PASSWORD='<your-vsphere-password>'

Security note: Avoid embedding credentials directly in scripts or YAML files. Use a secrets manager or a .env file with restricted permissions (chmod 600) when working in a shared environment.


Part I — VCF CLI Plugins

VCF CLI is the command-line interface for managing VCF workloads. It uses a plugin architecture: core capabilities are provided by separately installed plugins (such as addon, cluster, kubernetes-release). In an air-gapped environment, the plugin inventory must be served from your local Harbor instance rather than from Broadcom’s public OCI registry.

Step 1 — Download on an Internet-Connected Machine

1.1 Download and Install VCF CLI

Download the VCF CLI binary from Broadcom’s package repository and install it system-wide:

curl -L -o vcf-cli.tar.gz \
  https://packages.broadcom.com/artifactory/vcf-distro/vcf-cli/linux/amd64/v9.0.2/vcf-cli.tar.gz

tar -xzvf vcf-cli.tar.gz
sudo install -m 0755 vcf-cli-linux_amd64 /usr/local/bin/vcf

# Verify the installation
/usr/local/bin/vcf version

1.2 Inspect the Default Plugin Source

VCF CLI resolves plugins from an OCI artifact called the plugin inventory. Verify which registry it currently points to:

vcf plugin source list

Expected output:

NAME     IMAGE
default  projects.packages.broadcom.com/vcf-cli/plugins/plugin-inventory:latest

This confirms that VCF CLI currently fetches plugins from Broadcom’s public registry. The goal of this procedure is to redirect this to your local Harbor.

1.3 List Available Plugins

Before downloading, inspect the available plugins and their versions:

vcf plugin search

Expected output:

NAME                DESCRIPTION                                                                       LATEST
addon               Add-on lifecycle management                                                       v3.6.2
cluster             Kubernetes cluster operations                                                     v3.6.2
imgpkg              package, distribute, and relocate your configuration and dependent oci images     v9.0.2
kubernetes-release  Kubernetes release operations                                                     v3.6.2
namespaces          discover vsphere supervisor namespaces you have access to                         v9.0.0
package             vcf package management                                                            v3.6.2
pais                Welcome to the Private AI Services platform.                                      v2.1.0
registry-secret     registry-secret management                                                        v3.6.2
secret              Secret Store Plugin for VCF CLI                                                   v9.0.2
telemetry           Telemetry for VCF Workload CLI                                                    v9.0.2
vm                  VM Plugin for VCF Workload CLI                                                    v9.0.2

1.4 Download the Plugin Bundle

Download all plugins as a single archive. This file will be transferred to the air-gapped environment:

# Download all plugins (~6 GB)
vcf plugin download-bundle --to-tar /data/vcf-9.0.2.0/vcf-plugins-v9.0.2.tar.gz

du -h /data/vcf-9.0.2.0/vcf-plugins-v9.0.2.tar.gz
# Expected: ~6.2G

Alternatively, download only specific plugins to reduce the transfer size:

vcf plugin download-bundle --plugin addon:v3.6.2 \
  --to-tar /data/vcf-9.0.2.0/vcf-plugins-addon-v3.6.2.tar.gz

vcf plugin download-bundle --plugin cluster:v3.6.2 \
  --to-tar /data/vcf-9.0.2.0/vcf-plugins-cluster-v3.6.2.tar.gz

vcf plugin download-bundle --plugin imgpkg:v9.0.2 \
  --to-tar /data/vcf-9.0.2.0/vcf-plugins-imgpkg-v9.0.2.tar.gz

vcf plugin download-bundle --plugin kubernetes-release:v3.6.2 \
  --to-tar /data/vcf-9.0.2.0/vcf-plugins-kubernetes-release-v3.6.2.tar.gz

Transfer the resulting .tar.gz file(s) to the air-gapped environment using whatever mechanism is available (USB, secure file transfer, etc.).


Step 2 — Upload to Harbor in the Air-Gapped Environment

2.1 Create the Harbor Project for VCF CLI Plugins

Harbor organizes images into projects (similar to namespaces). Create a dedicated project for VCF CLI plugins:

HARBOR_PROJECT='vcf-cli'

curl -k -u "$HARBOR_USER:$HARBOR_PASS" \
  -H "Content-Type: application/json" \
  -X POST "https://$HARBOR/api/v2.0/projects" \
  -d "{\"project_name\":\"$HARBOR_PROJECT\",\"public\":true}"

# Verify the project was created
curl -k -u "$HARBOR_USER:$HARBOR_PASS" \
  "https://$HARBOR/api/v2.0/projects?name=$HARBOR_PROJECT"

Setting the project as "public": true allows VCF CLI to pull the plugin inventory without authenticating. Set it to false and provide credentials in vcf plugin source update if your security policy requires it.

2.2 Authenticate Docker to Harbor

The vcf plugin upload-bundle command uses Docker’s credential store internally. Log in before uploading:

echo "$HARBOR_PASS" | docker login "$HARBOR" -u "$HARBOR_USER" --password-stdin

Production note: In production, replace $HARBOR_USER/$HARBOR_PASS with a Harbor robot account (see the Robot Accounts section). The robot username format is robot$<account-name> — for example, robot$vcf-mirror-robot. Using a robot account limits the blast radius of a credential leak: it cannot access other Harbor projects or perform administrative operations.

2.3 Upload the Plugin Bundle

Push the bundle archive to the Harbor project:

vcf plugin upload-bundle \
  --tar /data/vcf-9.0.2.0/vcf-plugins-v9.0.2.tar.gz \
  --to-repo $HARBOR/$HARBOR_PROJECT/plugins

This command unpacks the OCI artifacts from the tar file and pushes them to $HARBOR/vcf-cli/plugins/, including the plugin inventory index artifact (plugin-inventory:latest).

2.4 Redirect VCF CLI to the Local Plugin Source

Update the VCF CLI configuration to resolve plugins from your local Harbor instead of Broadcom’s public registry:

vcf plugin source update default \
  --uri $HARBOR/$HARBOR_PROJECT/plugins/plugin-inventory:latest

# Confirm the source was updated
vcf plugin source list

Expected output (the URI now points to your Harbor):

NAME     IMAGE
default  harbor.example.com/vcf-cli/plugins/plugin-inventory:latest

2.5 Install Plugins from the Local Source

Remove any cached plugin metadata, then install the required plugins:

# Clear cached plugin state
vcf plugin clean

# Verify the plugin list resolves from Harbor
vcf plugin search

Expected output (note the refresh message referencing your Harbor):

[i] Refreshing plugin inventory cache for "harbor.example.com/vcf-cli/plugins/plugin-inventory:latest"...
NAME                DESCRIPTION                                                 LATEST
addon               Add-on lifecycle management                                 v3.6.2
cluster             Kubernetes cluster operations                               v3.6.2
...

Install the core plugins needed for VKS cluster and addon management:

vcf plugin install addon
vcf plugin install cluster
vcf plugin install kubernetes-release

# Verify all plugins are installed
vcf plugin list

Expected output:

NAME                DESCRIPTION                              INSTALLED  RECOMMENDED  STATUS
addon               Add-on lifecycle management              v3.6.0     v3.6.0       installed
cluster             Kubernetes cluster operations            v3.6.0     v3.6.0       installed
kubernetes-release  Kubernetes release operations            v3.6.0     v3.6.0       installed
namespaces          discover vsphere supervisor namespaces   v9.0.0     v9.0.0       installed
package             VCF Package management                   v3.6.0     v3.6.0       installed
registry-secret     Registry secret management               v3.6.0     v3.6.0       installed
...

Part II — VKS Standard Packages

The VKS Standard Packages repository is a large OCI bundle (~13 GB) that contains all container images required by VKS add-ons: cert-manager, Cilium, Contour, Prometheus, Velero, and others. In an air-gapped environment, this bundle must be mirrored to Harbor so that the Supervisor’s addon controller can pull images locally.

The tool used to copy OCI bundles is imgpkg, part of the Carvel toolchain.

Step 1 — Install imgpkg / Carvel Tools

imgpkg is part of the Carvel suite of Kubernetes packaging tools. Install it on the machine where you will perform the mirroring:

# Download the Carvel installation script
wget -O /tmp/install.sh https://carvel.dev/install.sh

# Make it executable and run as root
chmod +x /tmp/install.sh
sudo /tmp/install.sh

Verify the installation:

imgpkg version

Step 2 — Mirror the Standard Packages Bundle

2.1 Determine the Package Version

Consult the VKS Standard Packages Release Notes to find the correct bundle version for your VCF release.

This guide uses version 3.6.0+20260521. Set the version as a variable:

VKS_VERSION="3.6.0-20260521"
VKS_BUNDLE="projects.packages.broadcom.com/vsphere/supervisor/vks-standard-packages/${VKS_VERSION}/vks-standard-packages:${VKS_VERSION}"

2.2 Download the Bundle to a Tar File

Run imgpkg copy with --to-tar on an internet-connected machine. This downloads the entire bundle — all images and OCI metadata — into a single portable tar archive:

imgpkg copy \
  -b $VKS_BUNDLE \
  --to-tar /data/vcf-9.0.2.0/vcf-standard-packages-${VKS_VERSION}.tar

du -h /data/vcf-9.0.2.0/vcf-standard-packages-${VKS_VERSION}.tar
# Expected: ~13 GB

Transfer this file to the air-gapped environment.

Step 3 — Trust the Harbor CA Certificate

Before imgpkg can push to Harbor over HTTPS, the Harbor CA certificate must be trusted by both the OS and Docker. This is required whenever Harbor uses a self-signed or private CA.

3.1 Retrieve the Harbor CA Certificate

mkdir -p /tmp/harbor-ca

openssl s_client -showcerts \
  -connect $HARBOR:443 \
  -servername $HARBOR </dev/null 2>/dev/null \
  | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/ {print}' \
  > /tmp/harbor-ca/$HARBOR.crt

3.2 Verify the Certificate

openssl x509 -in /tmp/harbor-ca/$HARBOR.crt -noout -subject -issuer -dates

Review the output to confirm the certificate belongs to your Harbor instance and has not expired.

3.3 Trust the Certificate in Docker

Docker uses per-registry certificate directories under /etc/docker/certs.d/. Create the directory and place the CA certificate there:

sudo mkdir -p /etc/docker/certs.d/$HARBOR
sudo cp /tmp/harbor-ca/$HARBOR.crt /etc/docker/certs.d/$HARBOR/ca.crt
sudo systemctl restart docker

3.4 Trust the Certificate at the OS Level (RHEL / Rocky / Oracle Linux)

The OS trust store is used by tools like curl and imgpkg (which uses Go’s TLS stack). Add the certificate to the system trust store:

sudo cp /tmp/harbor-ca/$HARBOR.crt /etc/pki/ca-trust/source/anchors/$HARBOR.crt
sudo update-ca-trust

On Debian/Ubuntu systems, use:

sudo cp /tmp/harbor-ca/$HARBOR.crt /usr/local/share/ca-certificates/$HARBOR.crt
sudo update-ca-certificates

Step 4 — Upload the Bundle to Harbor

4.1 Create the Harbor Project for Packages

The vcf-packages project will hold the VKS Standard Packages bundle under a versioned path. The project is created as public so the Supervisor addon controller can pull images without embedding credentials in the AddonRepository resource.

HARBOR_PROJECT='vcf-packages'

curl -k -u "$HARBOR_USER:$HARBOR_PASS" \
  -H "Content-Type: application/json" \
  -X POST "https://$HARBOR/api/v2.0/projects" \
  -d "{\"project_name\":\"$HARBOR_PROJECT\",\"public\":true}"

# Verify project creation
curl -k -u "$HARBOR_USER:$HARBOR_PASS" \
  "https://$HARBOR/api/v2.0/projects?name=$HARBOR_PROJECT"

4.2 Push the Bundle to Harbor

Use imgpkg copy with --from-tar to push the bundle from the tar archive to your Harbor repository. The --registry-ca-cert-path flag explicitly passes the Harbor CA so imgpkg trusts the HTTPS connection:

imgpkg copy \
  --tar /data/vcf-9.0.2.0/vcf-standard-packages-${VKS_VERSION}.tar \
  --to-repo $HARBOR/$HARBOR_PROJECT/${VKS_VERSION}/vks-standard-packages \
  --registry-ca-cert-path /tmp/harbor-ca/$HARBOR.crt

Note: This push operation may take 30–60 minutes depending on Harbor’s disk speed and available CPU, as it processes ~409 OCI artifacts.

4.3 Verify the Repository in Harbor

Confirm that the repository and artifacts were created successfully:

curl -ksu "$HARBOR_USER:$HARBOR_PASS" \
  "https://$HARBOR/api/v2.0/projects/$HARBOR_PROJECT/repositories" | jq

Expected output (example with two versions present):

[
  {
    "artifact_count": 409,
    "creation_time": "2026-05-30T21:17:52.937Z",
    "id": 79,
    "name": "vcf-packages/3.6.0-20260521/vks-standard-packages",
    "project_id": 8,
    "pull_count": 0,
    "update_time": "2026-05-30T21:17:52.937Z"
  }
]

An artifact_count of ~409 confirms the full bundle was transferred successfully.


Part III — Register Harbor with the vSphere Supervisor

The vSphere Supervisor must be configured to trust Harbor before the addon controller can pull images from it. This is done in two sub-steps:

  1. Register the Harbor registry and its CA certificate in the vSphere Client UI
  2. Create a custom AddonRepository resource that points to the Harbor mirror

Step 1 — Register Harbor in the vSphere Client (UI)

Navigate to the vSphere Client and add Harbor as a trusted container registry for the Supervisor:

Supervisor Management
  → Supervisors
  → [Select your Supervisor]
  → Configure
  → Container Registries
  → Add

Fill in the form with the following values:

Field Value
Name harbor-<your-org-name> (e.g., harbor-vcf-example)
Registry Host URL harbor.example.com
CA Certificate Full PEM contents of /tmp/harbor-ca/$HARBOR.crt
Username / Password Harbor admin credentials (only required if the project is private)

Click Save. The Supervisor will now trust TLS certificates signed by the Harbor CA.

Step 2 — Connect to the Supervisor Cluster

Use VCF CLI to establish a Kubernetes context pointing to the Supervisor’s API server. This allows kubectl commands to manage Supervisor-level resources such as AddonRepository and AddonRepositoryInstall.

export CLUSTER_NAMESPACE='<your-vsphere-namespace>'
export SUPERVISOR_IP='<supervisor-api-ip>'
export VCF_CLI_VSPHERE_USERNAME='administrator@vsphere.local'
export VCF_CLI_VSPHERE_PASSWORD='<your-vsphere-password>'

# Remove any existing supervisor context and create a fresh one
vcf context delete supervisor
vcf context create supervisor \
  --endpoint ${SUPERVISOR_IP} \
  --username ${VCF_CLI_VSPHERE_USERNAME} \
  --insecure-skip-tls-verify \
  --type k8s \
  --auth-type basic

vcf context use supervisor

Note: The --insecure-skip-tls-verify flag is used here because many Supervisor endpoints use a self-signed certificate. In production, provide the Supervisor CA certificate instead.

Step 3 — Create the AddonRepository Override

The Supervisor uses an AddonRepository custom resource to determine where addon packages are fetched from. By default, it points to Broadcom’s public registry. You will create a new AddonRepository that points to Harbor, then patch the existing AddonRepositoryInstall to use it.

3.1 Set Variables

REGISTRY="$HARBOR"
HARBOR_PROJECT="vcf-packages"
VKS_VERSION="3.6.0-20260521"
VKS_STANDARD_REPO="$HARBOR_PROJECT/${VKS_VERSION}/vks-standard-packages"

IMAGE_URL="${REGISTRY}/${VKS_STANDARD_REPO}:${VKS_VERSION}"
echo "Using image: $IMAGE_URL"

Expected output:

Using image: harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages:3.6.0-20260521

3.2 Create the AddonRepository Manifest

The AddonRepository resource tells the Supervisor’s addon controller where to fetch the package bundle. The annotations field includes a JSON payload that records which version and packages are available — this is read by the addon controller to populate the available addon list.

cat > /tmp/default-addonrepository-override.yaml <<EOF
apiVersion: addons.kubernetes.vmware.com/v1alpha1
kind: AddonRepository
metadata:
  name: default-addonrepository-override
  namespace: vmware-system-vks-public
  annotations:
    addons.kubernetes.vmware.com/package-offerings: |
      {
        "repositoryVersion": "${VKS_VERSION}",
        "packages": {}
      }
spec:
  fetch:
    imgpkgBundle:
      imageURL: ${IMAGE_URL}
  targetRepositoryName: standard-packages
  version: ${VKS_VERSION}
EOF

kubectl apply -f /tmp/default-addonrepository-override.yaml

3.3 Redirect the AddonRepositoryInstall to the Override

The AddonRepositoryInstall resource (default-addon-repo-install) controls which AddonRepository is active. Patch it to point to the override and add an annotation that prevents VKS-managed automation from reverting your change:

NEW_REPO="default-addonrepository-override"
NAMESPACE="vmware-system-vks-public"

kubectl patch addonrepositoryinstall default-addon-repo-install \
  -n "$NAMESPACE" \
  --type merge \
  -p "{
    \"metadata\": {
      \"annotations\": {
        \"addons.kubernetes.vmware.com/skip-vks-managed-addon-repository-update\": \"\"
      }
    },
    \"spec\": {
      \"addonRepositoryRef\": {
        \"name\": \"$NEW_REPO\",
        \"namespace\": \"$NAMESPACE\"
      }
    }
  }"

Important: The annotation skip-vks-managed-addon-repository-update is required. Without it, the VKS platform automation will periodically overwrite your patch and revert the addonRepositoryRef back to the default Broadcom-hosted repository.

Step 4 — Validate the AddonRepository Configuration

4.1 Verify the Override Repository

kubectl get addonrepository default-addonrepository-override \
  -n vmware-system-vks-public

Expected output:

NAME                               TARGETREPOSITORY    VERSION          AGE
default-addonrepository-override   standard-packages   3.6.0-20260521   19h

4.2 Verify the Image URL Points to Harbor

kubectl get addonrepository default-addonrepository-override \
  -n vmware-system-vks-public \
  -o jsonpath='{.spec.fetch.imgpkgBundle.imageURL}{"\n"}'

Expected output:

harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages:3.6.0-20260521

4.3 Verify the AddonRepositoryInstall Uses the Override

kubectl get addonrepositoryinstall default-addon-repo-install \
  -n vmware-system-vks-public

Expected output:

NAME                         ADDONREPOSITORY                    AGE
default-addon-repo-install   default-addonrepository-override   31d

4.4 List All Available Addon Repositories

vcf addon repository list

Expected output (Harbor-hosted repository is now listed):

NAME                                    NAMESPACE                 SOURCE
default-addonrepository-3.5.0-20251218  vmware-system-vks-public  projects.packages.broadcom.com/...
default-addonrepository-3.6.0-20260416  vmware-system-vks-public  projects.packages.broadcom.com/...
default-addonrepository-override        vmware-system-vks-public  harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages:3.6.0-20260521

4.5 List Available Addons

Confirm the Supervisor can enumerate available addons from the local repository:

vcf addon available list
kubectl get addons -A

Expected output (showing addons sourced from the override repository):

NAMESPACE                  NAME                          AGE
vmware-system-vks-public   ako                           31d
vmware-system-vks-public   antrea                        31d
vmware-system-vks-public   cert-manager                  31d
vmware-system-vks-public   cilium                        18h
vmware-system-vks-public   contour                       31d
vmware-system-vks-public   gatekeeper                    18h
vmware-system-vks-public   helm-controller               18h
vmware-system-vks-public   prometheus                    31d
vmware-system-vks-public   velero                        31d
...

Validation — Install cert-manager from the Local Repository

Install the cert-manager addon on a test cluster and verify that all container images are pulled from Harbor.

List Available cert-manager Versions

vcf addon available list cert-manager

Expected output:

NAMESPACE                 ADDONNAME     VERSION                ADDON-RELEASE-NAME                                        PACKAGE
vmware-system-vks-public  cert-manager  1.18.2+vmware.2-vks.2  cert-manager.kubernetes.vmware.com.1.18.2-vmware.2-vks.2  ...
vmware-system-vks-public  cert-manager  1.19.4+vmware.1-vks.1  cert-manager.kubernetes.vmware.com.1.19.4-vmware.1-vks.1  ...

Install cert-manager

export CLUSTER_NAME='<your-test-cluster-name>'

vcf addon install create cert-manager \
  --cluster-name $CLUSTER_NAME \
  --namespace $CLUSTER_NAMESPACE \
  --verbose 9 -y

Confirm Installation Status

vcf addon install list \
  --cluster-name $CLUSTER_NAME \
  --namespace $CLUSTER_NAMESPACE

Expected output:

ADDONNAME     NAMESPACE              ADDONRELEASENAME                                          PAUSED  READY  DELETE/UPGRADE
cert-manager  <your-namespace>       cert-manager.kubernetes.vmware.com.1.19.4-vmware.1-vks.1  false   True   Allowed

Verify Images Are Pulled from Harbor

Switch your kubeconfig to the guest cluster, then inspect the image references for the cert-manager pods:

kubectl get pods -n cert-manager \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' \
  | column -t -N POD,IMAGE

Expected output — all images reference your Harbor registry:

POD                                      IMAGE
cert-manager-85c5d5d754-2fsp8            harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages@sha256:92bc...
cert-manager-cainjector-94cf76d75-vnwjr  harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages@sha256:1581...
cert-manager-webhook-6f85f6c47b-tccfx    harbor.example.com/vcf-packages/3.6.0-20260521/vks-standard-packages@sha256:72ad...

If all image references point to harbor.example.com, the air-gapped configuration is complete and working correctly.

Clean Up the Test Addon

vcf addon install delete cert-manager \
  --cluster-name $CLUSTER_NAME \
  --namespace $CLUSTER_NAMESPACE -y

Troubleshooting

Symptom Likely Cause Resolution
vcf plugin search shows no plugins Plugin source still points to Broadcom; local Harbor unreachable Run vcf plugin source list and verify the URI points to Harbor
imgpkg copy fails with TLS error Harbor CA not trusted Re-run Step 3.3 and 3.4 and verify update-ca-trust completed
kubectl apply returns “image pull backoff” Harbor project is private or Docker login expired Re-run docker login $HARBOR
addonrepositoryinstall reverts to Broadcom source Missing skip-vks-managed-addon-repository-update annotation Re-apply the patch in Step 3.3
artifact_count in Harbor API is 0 imgpkg copy failed partway through Re-run the imgpkg copy --from-tar command
Harbor returns 401 Unauthorized when pushing Robot account lacks push access to the project, or credentials are wrong Verify the robot account has repository:push permission on the target project; confirm the username format is robot$<name>
Harbor returns 403 Forbidden when creating a project The authenticated user is not a Harbor admin Use admin credentials (or a robot account with system-level create permission) for project creation; a project-scoped robot cannot create new projects
VKS cluster nodes fail with ImagePullBackOff after cluster creation Harbor CA not injected into cluster nodes Verify osConfiguration.trust.additionalTrustedCAs is set in the Cluster manifest and the referenced Secret exists; the CA trust step in the cluster manifest is separate from the Supervisor CA registration
imgpkg copy succeeds but artifact_count is lower than expected (< 409) Harbor storage quota exceeded, or push interrupted Check Harbor project storage quota (Harbor UI → Projects → vcf-packages → Configuration); increase the quota or set it to unlimited, then re-run the push
vSphere Supervisor cannot pull from Harbor despite Container Registry being registered Supervisor node CA trust not propagated After registering the Harbor CA in the vSphere Client, wait 2–3 minutes for the Supervisor to sync the CA; check the Container Registry status in the UI for any error messages

Reference

Name URL
VCF CLI Documentation https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-service-administration-and-development/9-0.html
VKS Standard Packages Release Notes https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-service-administration-and-development/9-0/release-notes/vks-standard-packages-release-notes.html
Harbor Documentation https://goharbor.io/docs/
vCenter Namespace Management APIs https://developer.broadcom.com/xapis/vsphere-automation-api/latest/vcenter-namespace-management/