Overview

This document describes how to configure log forwarding for a VMware vSphere Supervisor cluster to an external log platform — VCF Operations for Logs — using the Fluent Bit DaemonSet that Broadcom ships and manages in the vmware-system-logging namespace.

The Supervisor cluster is fundamentally different from a VKS guest cluster in this context. A VKS workload cluster allows full control over its logging stack: you deploy and configure Fluent Bit (or any other log shipper) freely. The Supervisor, by contrast, is a platform-managed control plane. You cannot deploy arbitrary workloads into it. Instead, Broadcom provides a pre-installed Fluent Bit DaemonSet and exposes a single editable ConfigMap — fluentbit-config-custom — where operators define their log sources, filters, and forwarding destinations.

The most operationally valuable Supervisor logs to forward are:

• API audit logs — a record of every API request processed by the Supervisor: who performed the action, on which resource, and what the outcome was. Essential for access auditing, compliance, and incident investigation.
• API server container logs — runtime log output from the kube-apiserver process itself: startup messages, admission webhook errors, authentication failures, and plugin activity. Complementary to the audit log; captures what the audit trail does not.

This guide configures Fluent Bit to collect both log streams, enrich them with structured metadata fields, and forward them to VCF Operations for Logs over HTTPS.

Prerequisite: A VCF CLI context pointing to the Supervisor must be available and functional before proceeding.

Reference Environment

The procedures and configurations in this document were validated against the following platform versions:

Architecture

The Supervisor log pipeline runs entirely within the Supervisor control plane. Fluent Bit runs as a DaemonSet — one pod per Supervisor control plane node — and reads log files directly from the node’s filesystem via hostPath mounts.

┌──────────────────────────────────────────────────────────────────────┐
│  vSphere Supervisor (Photon OS 5.0 control plane nodes)              │
│                                                                      │
│  /var/log/vmware/audit/kube-apiserver.log                            │
│  /var/log/containers/*kube-apiserver*.log                            │
│             │                                                        │
│             ▼                                                        │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  vmware-system-logging                                       │    │
│  │                                                              │    │
│  │  fluentbit (DaemonSet, one pod per control plane node)       │    │
│  │    │                                                         │    │
│  │    ├─ INPUT  tail  audit.apiserver.supervisor                │    │
│  │    ├─ INPUT  tail  container.kube-apiserver.supervisor       │    │
│  │    │                                                         │    │
│  │    ├─ FILTER record_modifier  (add metadata fields)          │    │
│  │    ├─ FILTER modify           (add text label)               │    │
│  │    │                                                         │    │
│  │    └─ OUTPUT http             (TLS, JSON)                    │    │
│  └──────────────────────────────────────────────────────────────┘    │
│             │                                                        │
└─────────────┼────────────────────────────────────────────────────────┘
              │  HTTPS POST  :9543/api/v2/events
              ▼
┌─────────────────────────────────────────┐
│  VCF Operations for Logs                │
│  192.168.200.40:9543                    │
└─────────────────────────────────────────┘

ConfigMap layer:

Only fluentbit-config-custom should be modified. The system ConfigMaps are owned by the platform and will be overwritten during Supervisor upgrades.

Prerequisites

• VCF CLI installed with an active context pointing to the Supervisor API endpoint
• kubectl available and able to reach the Supervisor
• VCF Operations for Logs deployed and reachable from the Supervisor control plane nodes on port 9543
• Sufficient privileges to edit ConfigMaps in vmware-system-logging (requires Supervisor administrator role)

Set the following environment variables before running any commands:

export CLUSTER_NAMESPACE='lab-poc-namespace-dev'
export SUPERVISOR_IP='10.77.68.3'
export VCF_CLI_VSPHERE_USERNAME='administrator@vsphere.local'
export VCF_CLI_VSPHERE_PASSWORD='<password>'

Step 1 — Connect to the Supervisor Cluster

Establish a VCF CLI context for the Supervisor. If a context named supervisor already exists from a previous session, delete it first to avoid stale credentials.

vcf context delete supervisor -y
vcf context list

vcf context create supervisor \
  --endpoint ${SUPERVISOR_IP} \
  --username ${VCF_CLI_VSPHERE_USERNAME} \
  --insecure-skip-tls-verify \
  --type k8s \
  --auth-type basic

vcf context use supervisor

Verify connectivity:

kubectl get nodes

Step 2 — Verify the Fluent Bit Installation

Before making any changes, confirm that the vmware-system-logging namespace and all Fluent Bit pods are healthy.

2.1 Check the Logging Namespace

kubectl get ns vmware-system-logging

Expected output:

NAME                    STATUS   AGE
vmware-system-logging   Active   37d

2.2 Check Fluent Bit Pod Status

One Fluent Bit pod runs on each Supervisor control plane node:

kubectl get pods -n vmware-system-logging

Expected output:

NAME              READY   STATUS    RESTARTS       AGE
fluentbit-4s9n7   1/1     Running   3 (2d5h ago)   22d
fluentbit-bk5vj   1/1     Running   2 (2d5h ago)   22d
fluentbit-vpfh8   1/1     Running   3 (2d5h ago)   22d

A three-node Supervisor control plane produces three pods. Restarts in single digits are normal — Fluent Bit is restarted when the ConfigMap is updated or during node maintenance.

2.3 Inspect the ConfigMaps

kubectl get cm -n vmware-system-logging

Expected output:

NAME                      DATA   AGE
fluentbit-config          1      37d
fluentbit-config-custom   4      37d
fluentbit-config-system   4      37d
kube-root-ca.crt          1      37d

Inspect the current contents of the editable ConfigMap:

kubectl get cm fluentbit-config-custom -n vmware-system-logging -o yaml

By default this ConfigMap contains empty or minimal placeholder entries. All customisation is applied by replacing its content in Step 3.

Step 3 — Prepare the Custom ConfigMap

The fluentbit-config-custom ConfigMap has four keys, each corresponding to a Fluent Bit configuration file section. All four keys must be present in the ConfigMap, even if a section requires no content — some Supervisor Fluent Bit versions expect every file to exist at mount time.

3.1 Parsers

No custom parsers are required for this configuration. The json parser (used for audit logs) and the cri parser (used for container logs) are both built into the Supervisor’s Fluent Bit version.

The parsers-custom.conf key must still be present:

[parsers-custom.conf]
# No custom parsers required.

3.2 Inputs

Two log streams are collected.

Stream 1 — Kubernetes API audit logs

Audit logs record every request processed by the Supervisor’s kube-apiserver. Each line is a JSON object conforming to the audit.k8s.io/v1 schema, containing: the requesting user, the target resource, the verb (get, create, delete, etc.), source IPs, and the response code.

Important: The correct audit log path on Photon OS 5.0 Supervisor nodes is /var/log/vmware/audit/kube-apiserver.log. The path /var/log/apiserver-audit/kube-apiserver-audit.log does not exist on this Supervisor version and will cause the input to silently collect nothing.

Stream 2 — Kubernetes API server container logs

Container logs capture the kube-apiserver process’s own stdout/stderr output: startup and shutdown messages, admission webhook decisions, certificate rotation activity, and runtime errors. These are not audit events — they are unstructured (or lightly structured) text log lines from the Go runtime.

3.3 Filters

Two filter stages are applied to each stream.

record_modifier — adds structured metadata fields to every log event. These fields make logs searchable and filterable in VCF Operations for Logs without parsing the raw log content:

modify — adds a human-readable text field to every event. VCF Operations for Logs full-text search operates on the text field, so setting it to a descriptive string enables simple keyword searches to locate all events from a given stream.

3.4 Outputs

Both streams are forwarded to VCF Operations for Logs over HTTPS using the Fluent Bit HTTP output plugin, which posts log batches as JSON arrays to the /api/v2/events endpoint.

Security note — TLS without certificate verification: The Supervisor login does not permit creating Secret resources in vmware-system-logging, which means the VCF Operations for Logs CA certificate cannot be mounted into the Fluent Bit pods. As a result, tls.verify Off is required: traffic is encrypted but the server’s identity is not validated. In a production environment, if the CA certificate can be made available (for example, by pre-staging it via the Supervisor certificate trust mechanism), configure tls.verify On and tls.ca_file <path> to enable full mutual verification.

Step 4 — Apply the Configuration

Generate and apply the complete fluentbit-config-custom ConfigMap. All four configuration sections are included in a single manifest:

cat << 'EOF' > supervisor-fluentbit-config-custom.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-config-custom
  namespace: vmware-system-logging

data:
  ###########################################################################
  # parsers-custom.conf
  #
  # Keep this file present even when no custom parsers are required.
  # Some Supervisor Fluent Bit versions expect this file to exist at mount
  # time; an absent key causes a configuration load error on pod startup.
  ###########################################################################
  parsers-custom.conf: |
    # No custom parsers required.

  ###########################################################################
  # inputs-custom.conf
  ###########################################################################
  inputs-custom.conf: |
    #########################################################################
    # Kubernetes API audit logs — Supervisor (Photon OS 5.0)
    #
    # Each line is a JSON audit event conforming to audit.k8s.io/v1:
    #   "apiVersion":"audit.k8s.io/v1"
    #   "verb":"create" | "delete" | "get" | "update" | ...
    #   "requestURI":"/api/v1/..."
    #
    # Correct path: /var/log/vmware/audit/kube-apiserver.log
    # Note: /var/log/apiserver-audit/kube-apiserver-audit.log does not exist
    #       on this Supervisor version.
    #########################################################################
    [INPUT]
        Name              tail
        Alias             audit_apiserver_tail
        Tag               audit.apiserver.supervisor
        Path              /var/log/vmware/audit/kube-apiserver.log
        Parser            json
        DB                /var/log/vmware/fluentbit/flb_audit_apiserver.db
        Buffer_Chunk_Size 1MB
        Buffer_Max_Size   12MB
        Mem_Buf_Limit     64MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Rotate_Wait       30
        Read_from_Head    Off

    #########################################################################
    # Kubernetes API server container logs — Supervisor (Photon OS 5.0)
    #
    # Runtime/container logs from the kube-apiserver process: startup and
    # shutdown messages, admission webhook decisions, certificate rotation
    # activity, authentication errors, and plugin messages. These are not
    # audit events — they are Go runtime log lines in CRI format.
    #########################################################################
    [INPUT]
        Name              tail
        Alias             kube_apiserver_container_tail
        Tag               container.kube-apiserver.supervisor
        Path              /var/log/containers/*kube-apiserver*.log
        Parser            cri
        DB                /var/log/vmware/fluentbit/flb_kube_apiserver_container.db
        Buffer_Chunk_Size 256KB
        Buffer_Max_Size   2MB
        Mem_Buf_Limit     32MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Rotate_Wait       30
        Read_from_Head    Off

  ###########################################################################
  # filters-custom.conf
  #
  # Enriches each log event with structured metadata fields. These fields
  # enable targeted searches in VCF Operations for Logs without requiring
  # full-text parsing of the raw log content.
  ###########################################################################
  filters-custom.conf: |
    #########################################################################
    # Metadata enrichment — Kubernetes API audit logs
    #########################################################################
    [FILTER]
        Name           record_modifier
        Alias          audit_apiserver_modifier
        Match          audit.apiserver.*
        Record         hostname ${NODE_NAME}
        Record         appname kube-apiserver-audit
        Record         filename kube-apiserver-audit.log
        Record         log_category kube-apiserver-audit
        Record         vks_cluster supervisor
        Record         environment vcf902

    [FILTER]
        Name           modify
        Alias          audit_apiserver_text
        Match          audit.apiserver.*
        Add            text "Supervisor API Server Audit Logs"

    #########################################################################
    # Metadata enrichment — Kubernetes API server container logs
    #########################################################################
    [FILTER]
        Name           record_modifier
        Alias          kube_apiserver_container_modifier
        Match          container.kube-apiserver.*
        Record         hostname ${NODE_NAME}
        Record         appname kube-apiserver-container
        Record         filename kube-apiserver-container.log
        Record         log_category kube-apiserver-container
        Record         vks_cluster supervisor
        Record         environment vcf902

    [FILTER]
        Name           modify
        Alias          kube_apiserver_container_text
        Match          container.kube-apiserver.*
        Add            text "Supervisor API Server Container Logs"

  ###########################################################################
  # outputs-custom.conf
  #
  # Forwards logs to VCF Operations for Logs via HTTPS.
  #
  # Destination:
  #   Host: 192.168.200.40
  #   Port: 9543
  #   URI:  /api/v2/events
  #
  # Security note:
  #   tls.verify Off encrypts traffic but does not validate the server
  #   certificate. For production, configure tls.verify On and tls.ca_file
  #   once the Operations for Logs CA certificate can be mounted into the
  #   Fluent Bit pods.
  ###########################################################################
  outputs-custom.conf: |
    #########################################################################
    # Output — Kubernetes API audit logs
    #########################################################################
    [OUTPUT]
        Name          http
        Alias         audit_apiserver_output_vcf_ops_logs
        Match         audit.apiserver.*
        Host          192.168.200.40
        Port          9543
        URI           /api/v2/events
        Format        json
        tls           On
        tls.verify    Off
        Retry_Limit   False
        json_date_key timestamp

    #########################################################################
    # Output — Kubernetes API server container logs
    #########################################################################
    [OUTPUT]
        Name          http
        Alias         kube_apiserver_container_output_vcf_ops_logs
        Match         container.kube-apiserver.*
        Host          192.168.200.40
        Port          9543
        URI           /api/v2/events
        Format        json
        tls           On
        tls.verify    Off
        Retry_Limit   False
        json_date_key timestamp
EOF

Apply the ConfigMap:

kubectl apply -f supervisor-fluentbit-config-custom.yaml -n vmware-system-logging

Expected output:

configmap/fluentbit-config-custom configured

When the ConfigMap is updated, the Fluent Bit pods detect the change and reload their configuration. This typically takes 30–60 seconds. The pods do not restart unless there is a configuration parse error.

Step 5 — Verify Log Forwarding

5.1 Confirm Pods Are Running

kubectl get pods -n vmware-system-logging

All pods should remain in Running status with no increase in restart count after the ConfigMap update. An unexpected restart indicates a configuration parse error — check the pod logs immediately (see section 5.2).

5.2 Inspect Fluent Bit Logs

Check the Fluent Bit output on all pods for forwarding activity and errors:

kubectl -n vmware-system-logging logs -l name=fluentbit --tail=100

Look for lines confirming the inputs have been initialised and the HTTP output is connecting successfully:

[2026/06/05 17:20:01] [ info] [input:tail:audit_apiserver_tail] inotify_fs_add(): inode=12345 filename=/var/log/vmware/audit/kube-apiserver.log
[2026/06/05 17:20:01] [ info] [input:tail:kube_apiserver_container_tail] inotify_fs_add(): inode=67890 filename=/var/log/containers/kube-apiserver-...log
[2026/06/05 17:20:02] [ info] [output:http:audit_apiserver_output_vcf_ops_logs] worker #0 started

To stream live logs from all pods:

kubectl -n vmware-system-logging logs -l name=fluentbit -f

Step 6 — Query VCF Operations for Logs

VCF Operations for Logs exposes a REST API for querying ingested events. The API uses session-based authentication: first obtain a session token, then use it in subsequent queries.

6.1 Obtain a Session Token

LOGS_HOST='192.168.200.40'
LOGS_USER='admin'
LOGS_PASS='<password>'

SESSION_ID=$(curl -sk \
  -H "Content-Type: application/json" \
  -X POST "https://${LOGS_HOST}:9543/api/v2/sessions" \
  -d "{
    \"username\":\"${LOGS_USER}\",
    \"password\":\"${LOGS_PASS}\",
    \"provider\":\"Local\"
  }" | jq -r '.sessionId')

echo "Session ID: ${SESSION_ID}"

6.2 Search for API Audit Log Events

Search for events in the last 60 minutes that contain the text label set by the filter:

SEARCH="Supervisor API Server Audit Logs"

curl -sk \
  -H "Authorization: Bearer ${SESSION_ID}" \
  "https://${LOGS_HOST}:9543/api/v1/events/text/CONTAINS%20$(python3 -c "import urllib.parse; print(urllib.parse.quote('${SEARCH}'))")/timestamp/LAST%203600000" \
| jq .

Example event returned:

{
  "text": "Supervisor API Server Audit Logs",
  "timestamp": 1780680667279,
  "timestampString": "2026-06-05 17:31:07.279 GMT+00:00",
  "fields": [
    {
      "name": "log",
      "content": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"RequestResponse\",\"auditID\":\"c6a3d79f-823a-4d64-9e70-11d9113023e7\",\"stage\":\"RequestReceived\",\"requestURI\":\"/apis/storage.k8s.io/v1/storageclasses/vsan-esa-default-policy-raid5-latebinding\",\"verb\":\"get\",\"user\":{\"username\":\"system:serviceaccount:vmware-system-csi:vsphere-csi-controller\",\"groups\":[\"system:serviceaccounts\",\"system:serviceaccounts:vmware-system-csi\",\"system:authenticated\"]}}"
    },
    { "name": "hostname",      "content": "423e886a1ddac28a7ce11bd30f6cf21c" },
    { "name": "environment",   "content": "vcf902" },
    { "name": "filename",      "content": "kube-apiserver-audit.log" },
    { "name": "appname",       "content": "kube-apiserver-audit" },
    { "name": "log_category",  "content": "kube-apiserver-audit" },
    { "name": "vks_cluster",   "content": "supervisor" },
    { "name": "source",        "content": "192.168.200.51" }
  ]
}

The screenshot below shows a Supervisor API server audit event as it appears in VCF Operations for Logs. The log field carries the raw audit.k8s.io/v1 JSON object as a string — this is characteristic of the Supervisor pipeline, where the json parser reads the entire audit record as a single structured document and the HTTP output serialises it under the log key. The inner audit fields (verb, requestURI, user, auditID) are embedded within that string rather than promoted to individual top-level facets as they are in the VKS guest cluster pipeline.

The surrounding metadata fields — hostname, vks_cluster, log_category, appname, filename, environment — are added by the record_modifier filter and are indexed as discrete searchable facets in VCF Operations for Logs. These fields enable targeted queries without requiring JSON parsing of the nested log value: filtering on vks_cluster = supervisor and log_category = kube-apiserver-audit isolates all Supervisor API audit events from VKS workload cluster logs when multiple clusters forward to the same Operations for Logs instance.

The hostname field reflects the Supervisor control plane node identity at runtime via the ${NODE_NAME} environment variable injected into the pod. The source field shows the node’s IP address (192.168.200.51 in the example), which is useful for correlating audit events with specific control plane node activity during multi-node incidents.

Note: The ${NODE_NAME} variable is provided by the Kubernetes downward API and is correctly resolved at pod startup. Using ${HOSTNAME} instead — a common error in community examples — resolves to the pod’s internal hostname rather than the node name, producing misleading values in the hostname field.

6.3 Search for API Server Container Log Events

SEARCH="Supervisor API Server Container Logs"

curl -sk \
  -H "Authorization: Bearer ${SESSION_ID}" \
  "https://${LOGS_HOST}:9543/api/v1/events/text/CONTAINS%20$(python3 -c "import urllib.parse; print(urllib.parse.quote('${SEARCH}'))")/timestamp/LAST%203600000" \
| jq .

Example events returned:

{
  "text": "Supervisor API Server Container Logs",
  "timestamp": 1780679991604,
  "timestampString": "2026-06-05 17:19:51.604 GMT+00:00",
  "fields": [
    {
      "name": "log",
      "content": "2026-06-05T17:19:50.764323889Z stderr F I0605 17:19:50.763433       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.data.packaging.carvel.dev: Rate Limited Requeue."
    },
    { "name": "hostname",      "content": "423e36fcfce0c74ea4c0e3ad2a0525de" },
    { "name": "environment",   "content": "vcf902" },
    { "name": "filename",      "content": "kube-apiserver-container.log" },
    { "name": "appname",       "content": "kube-apiserver-container" },
    { "name": "log_category",  "content": "kube-apiserver-container" },
    { "name": "vks_cluster",   "content": "supervisor" },
    { "name": "source",        "content": "192.168.200.52" }
  ]
}

The screenshot below shows Supervisor API server container log events in VCF Operations for Logs. Container log lines arrive in CRI format — <RFC3339Nano timestamp> <stream> <flags> <message> — and are parsed by the Supervisor Fluent Bit’s built-in cri parser before enrichment. The source field shows the IP address of the Supervisor control plane node that produced the log, allowing events to be correlated with a specific node during multi-node control plane incidents.

Container logs complement the audit stream in a critical way: while the audit log records what API operations were performed, the container log records how the API server behaved at the process level. Certificate rotation events, admission webhook invocation and rejection decisions, OpenAPI aggregation activity, and plugin initialisation failures all appear exclusively in the container log stream. These events are absent from the audit trail by design. The example event shown — an OpenAPI AggregationController rate-limited requeue from the Carvel packaging service account — is the type of operational signal that is invisible in the audit log but immediately visible here.

This stream is particularly useful when diagnosing VKS cluster provisioning failures: the kube-apiserver admission webhooks that gate VKS cluster creation run in the Supervisor, and any webhook timeout or RBAC rejection is logged here before propagating as a provisioning error to the VKS cluster object.

Troubleshooting

Reference