Log Insights: Kubernetes with OpenTelemetry (Vector)

This page is the detailed reference for how the Vector configuration itself works, referenced from the CloudNativePG Log Insights setup guide as part of its step-by-step walkthrough.

If you are using Vector for log forwarding from CloudNativePG (CNPG) containers you will need to configure it correctly to map input data collected from Kubernetes logs to the OpenTelemetry OTLP log record format, optionally filtering out clusters/pods you don’t intend to monitor with pganalyze, and then forward it to the pganalyze-collector.

You can find a full example with Helm charts at the end of this page, where all of the configuration steps described below are implemented together inside a single values.yaml file.

Configuring Vector to send OpenTelemetry logs

The following steps describe the logical structure of the Vector configuration. When deploying Vector via Helm, all of these steps are expressed together inside the Helm chart’s values.yaml file.

First, you need to ensure your Vector configuration is sourcing logs from your Kubernetes cluster, like this:

  sources:
    kubernetes_logs:
      type: kubernetes_logs
      exclude_paths_glob_patterns:
        - "/var/log/pods/kube-system_*/**"

Second, we recommend filtering the logs to just the CNPG output, and limiting it down to the clusters you want to monitor in pganalyze. Each entry in the includes([...]) list below is the name of a whole Cluster resource (the cnpg.io/cluster label, shared by every instance/replica of that cluster, not a per-instance name). Most setups only have one entry here. The example below shows two entries, payments-prod and payments-staging, purely to illustrate monitoring more than one separate cluster through the same Vector deployment; don’t treat this as a pattern to imitate for a single cluster. Note that you can send the logs of multiple clusters/pods to the same pganalyze-collector installation, and then use the collector configuration to split it up by cluster or Kubernetes pod labels.

  transforms:
    cnpg_route:
      type: route
      inputs:
        - kubernetes_logs
      reroute_unmatched: false
      route:
        cnpg: |
          exists(.kubernetes.pod_labels."cnpg.io/cluster") &&
          exists(.kubernetes.pod_labels."cnpg.io/podRole") &&
          .kubernetes.pod_labels."cnpg.io/podRole" == "instance" &&
          ## Optionally filter out replicas
          # exists(.kubernetes.pod_labels."cnpg.io/instanceRole") &&
          # .kubernetes.pod_labels."cnpg.io/instanceRole" == "primary" &&
          includes(
            [
              "payments-prod",
              "payments-staging",
            ],
            .kubernetes.pod_labels."cnpg.io/cluster"
          )

Third, the CNPG log output needs to be transformed to inline the jsonlog output produced by CNPG, and modify it to match the expected OTLP log record format. This is accomplished through another transform step that uses a custom VRL script, found in full in the collector repository.

   cnpg_remap:
      type: remap
      inputs:
        - cnpg_route.cnpg
      source: |
        ... VRL Script omitted ...
        ... Copy from collector repository (contrib/vector/cnpg_remap.yaml) ...

Last, the log output needs to be sent to your pganalyze collector configuration that has the OTLP log server configured, correctly wrapping the log records in OTLP JSON encoding:

  sinks:
    pganalyze:
      inputs:
        - cnpg_remap
      type: opentelemetry
      protocol:
        type: http
        # ! Update this to point to the pganalyze-collector Service with db_log_otel_server / LOG_OTEL_SERVER configured.
        # ! Use the namespace-qualified form if Vector and the collector are in different namespaces (likely) —
        # ! a bare Service name only resolves via Kubernetes DNS within the same namespace as the caller.
        uri: "http://pganalyze-collector-otel-service.<collector-namespace>.svc.cluster.local:4318/v1/logs"
        method: post
        encoding:
          codec: json
        framing:
          method: "character_delimited"
          character_delimited:
            delimiter: ","
        payload_prefix: '{"resourceLogs":'
        payload_suffix: '}'
        batch:
          max_events: 1
          timeout_secs: 5
        request:
          headers:
            content-type: "application/json"

batch.max_events must be 1. Vector’s opentelemetry sink does not currently support batching more than one event per request with the json codec (confirmed by a Vector maintainer in vectordotdev/vector#22054); any request carrying more than one event is malformed and gets rejected with HTTP 400, and dropped rather than retried. This means every log line is sent as its own HTTP request, higher request volume than true batching, but it’s the only combination that reliably works today.

Deploying Vector through Helm chart

The easiest way to deploy Vector in a Kubernetes environment is using Helm charts.

First, fetch the example values.yaml that works with pganalyze from the collector repository:

wget https://github.com/pganalyze/collector/raw/refs/heads/main/contrib/vector/values.yaml

Important: Next, customize the values.yaml file to use your cluster names in the cnpg_route step, and point to the correct pganalyze-collector installation in the sink.

Then, use the following commands to deploy Vector:

helm repo add vector https://helm.vector.dev
helm repo update
helm install vector vector/vector \
  --namespace vector \
  --create-namespace \
  --values values.yaml