Vector to Nightlamp

Vector by Datadog (now an OSS project) collects, transforms, and ships observability data. This guide wires Vector's kubernetes_logs source through a Loki sink targeting Nightlamp — same wire protocol as Promtail/Fluent Bit, different config language.

Prerequisites

A Kubernetes cluster you can apply manifests to. (Non-k8s setups work too — skip to the "Non-Kubernetes setups" section below.)
A Nightlamp app registered — app ID and DSN key.
Vector 0.34+ (Nov 2023). The Loki sink's request.headers map landed before this version, but earlier releases had bugs around env-var expansion that made the DSN flow fragile.

How it works

Vector tails container log files on each node, enriches with Kubernetes metadata, then pushes batches to Nightlamp's Loki-compatible push endpoint. Two headers identify the source app:

Required request headers

X-Nightlamp-App-Id: <your-app-id>
X-Nightlamp-Dsn-Key: <your-dsn-key>

We gate on an opt-in pod label (nightlamp.app/id) so rolling Vector out across a cluster doesn't accidentally ship every legacy workload's stdout.

1. ServiceAccount + RBAC

rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vector
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vector-read
rules:
  - apiGroups: [""]
    resources: ["namespaces", "nodes", "pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vector-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vector-read
subjects:
  - kind: ServiceAccount
    name: vector
    namespace: default

2. Secret with your DSN

Create the Secret

kubectl create secret generic vector-dsn \
  --namespace=default \
  --from-literal=dsn=<your-dsn-key>

3. ConfigMap (TOML)

Vector uses TOML by default (YAML and JSON are also supported). The pipeline: kubernetes_logs source → filter on the opt-in label → remap to drop health-probe noise → loki sink.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-config
  namespace: default
data:
  vector.toml: |
    # — Sources ————————————————————————————————————————
    [sources.kube]
      type = "kubernetes_logs"
      # Only read files for pods labeled with our opt-in tag.
      extra_label_selector = "nightlamp.app/id"

    # — Transforms ——————————————————————————————————————
    # Drop 2xx /health* probe spam.
    [transforms.no_health]
      type = "filter"
      inputs = ["kube"]
      condition = '''
        !match(string!(.message), r'"/health[^"]*" 2\d\d')
      '''

    # Flatten kubernetes_labels.* into a couple of fields Nightlamp uses
    # for grouping. Anything else stays under .kubernetes for debug.
    [transforms.tag]
      type = "remap"
      inputs = ["no_health"]
      source = '''
        .app = .kubernetes.pod_labels."nightlamp.app/id"
        .pod = .kubernetes.pod_name
        .ns = .kubernetes.pod_namespace
        .container = .kubernetes.container_name
      '''

    # — Sink ———————————————————————————————————————————
    [sinks.nightlamp]
      type = "loki"
      inputs = ["tag"]
      endpoint = "https://api.nightlamp.app"
      path = "/api/loki/api/v1/push"
      encoding.codec = "json"
      labels.job = "vector"
      labels.app = "{{ app }}"
      labels.pod = "{{ pod }}"
      labels.namespace = "{{ ns }}"
      labels.container = "{{ container }}"
      remove_label_fields = true

      [sinks.nightlamp.request.headers]
        X-Nightlamp-App-Id = "<your-app-id>"
        X-Nightlamp-Dsn-Key = "${NIGHTLAMP_DSN}"

Multiple Nightlamp apps in one cluster? Duplicate the [sinks.*] block — each instance gets its own X-Nightlamp-App-Id literal and a condition-driven router transform upstream partitioning events by their kubernetes.pod_labels.

4. DaemonSet

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector
  namespace: default
spec:
  selector:
    matchLabels: { app: vector }
  template:
    metadata:
      labels: { app: vector }
    spec:
      serviceAccountName: vector
      tolerations:
        - operator: Exists
          effect: NoSchedule
        - operator: Exists
          effect: NoExecute
      containers:
        - name: vector
          image: timberio/vector:0.34.0-debian
          args: ["-c", "/etc/vector/vector.toml"]
          env:
            - name: NIGHTLAMP_DSN
              valueFrom:
                secretKeyRef:
                  name: vector-dsn
                  key: dsn
          resources:
            requests: { cpu: 100m, memory: 96Mi }
            limits:   { cpu: 500m, memory: 384Mi }
          volumeMounts:
            - { name: config,   mountPath: /etc/vector }
            - { name: varlog,   mountPath: /var/log, readOnly: true }
            - { name: data,     mountPath: /var/lib/vector }
      volumes:
        - name: config
          configMap: { name: vector-config }
        - { name: varlog, hostPath: { path: /var/log } }
        - { name: data,   hostPath: { path: /var/lib/vector, type: DirectoryOrCreate } }

5. Opt your workloads in

In each app's Deployment manifest

spec:
  template:
    metadata:
      labels:
        nightlamp.app/id: <your-app-id>

Non-Kubernetes setups

Outside Kubernetes, swap the kubernetes_logs source for whatever fits — file, journald, docker_logs, syslog, or an http server source. The Loki sink at the bottom of the pipeline doesn't change.

Example: systemd journal source on a single host

[sources.journald]
  type = "journald"
  current_boot_only = true

[sinks.nightlamp]
  type = "loki"
  inputs = ["journald"]
  endpoint = "https://api.nightlamp.app"
  path = "/api/loki/api/v1/push"
  encoding.codec = "json"
  labels.job = "journald"
  labels.unit = "{{ _SYSTEMD_UNIT }}"
  labels.host = "{{ host }}"

  [sinks.nightlamp.request.headers]
    X-Nightlamp-App-Id = "<your-app-id>"
    X-Nightlamp-Dsn-Key = "${NIGHTLAMP_DSN}"

Troubleshooting

Vector boots but no lines arrive. Run vector top against the running pod (kubectl exec -it <pod> -- vector top) to inspect throughput on each component. A non-zero kube source rate with zero on the sink means the filter or remap stage is dropping everything — most often a mistyped label selector.
401 on the Loki sink. Vector swallows the response body by default; enable internal_log_rate_limit = false and inspect vector's own logs for the request.failed events that include the upstream status. A 401 is always our HMAC check rejecting the DSN — double-check the env-var expansion in request.headers.
Multi-line stack traces split into individual events. Vector's kubernetes_logs source has a multiline block — add a start_pattern matching your timestamp prefix so continuation lines get glued onto the parent event.