Promtail on Kubernetes

Ship every node's container logs to Nightlamp using Promtail — Grafana's classic Loki agent. Covers RBAC, a scrape_configs walkthrough, the DaemonSet shape, and verifying that lines land in your Issue queue.

Prerequisites

A Kubernetes cluster you can apply manifests to.
A Nightlamp app registered — you'll need its app ID and a DSN key.
Promtail 2.7+ (Oct 2022). Earlier versions silently drop the clients[].headers map, which is how we authenticate the push.

Already on Grafana Alloy? Alloy is Promtail's successor and ships the same Loki source. Use the Grafana Alloy guide instead — the configuration shape is different (River vs. YAML) but the wire protocol is identical.

How it works

Promtail tails container log files under /var/log/pods and /var/log/containers, decorates them with Kubernetes pod metadata pulled from the API server, then POSTs to Nightlamp's Loki-compatible push endpoint at /api/loki/api/v1/push with two headers identifying the source app.

Required request headers

X-Nightlamp-App-Id: <your-app-id>
X-Nightlamp-Dsn-Key: <your-dsn-key>

As with Fluent Bit, we gate on an opt-in pod label so rolling Promtail across a cluster doesn't accidentally fire-hose every legacy workload's stdout into Nightlamp.

1. ServiceAccount + RBAC

Promtail's kubernetes_sd_configs needs read access to pods and (optionally) namespaces. The Endpoints permission is only required if you scrape Services in addition to Pods.

rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-read
rules:
  - apiGroups: [""]
    resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: promtail-read
subjects:
  - kind: ServiceAccount
    name: promtail
    namespace: default

2. Secret with your DSN

Promtail reads the DSN key from a Secret so the ConfigMap (committed) and the credential (out-of-band) stay separate.

Create the Secret

kubectl create secret generic promtail-dsn \
  --namespace=default \
  --from-literal=dsn=<your-dsn-key>

3. ConfigMap

Two parts: a clients[] entry pointing at Nightlamp, and a scrape_configs[] entry that discovers pods, keeps only those with the nightlamp.app/id label, and labels each line with the discovered metadata.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: default
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      log_level: info

    positions:
      filename: /run/promtail/positions.yaml

    clients:
      - url: https://api.nightlamp.app/api/loki/api/v1/push
        # The DSN key is injected at runtime via the NIGHTLAMP_DSN env
        # var; promtail's YAML config doesn't natively interpolate env
        # vars, so we use ${VAR} substitution with --config.expand-env.
        headers:
          X-Nightlamp-App-Id: <your-app-id>
          X-Nightlamp-Dsn-Key: ${NIGHTLAMP_DSN}

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          # Multiline glue: lines that don't start with a timestamp are
          # appended to the previous record (Java/Python stack traces).
          - multiline:
              firstline: '^\d{4}-\d{2}-\d{2}'
              max_wait_time: 3s
          # Drop 2xx /health* probe spam before it leaves the node.
          - drop:
              expression: '\"/health[^\"]*\" 2\d\d'
        relabel_configs:
          # Keep only pods opted in via the nightlamp.app/id label.
          - source_labels:
              - __meta_kubernetes_pod_label_nightlamp_app_id
            action: keep
            regex: .+
          - source_labels:
              - __meta_kubernetes_pod_label_nightlamp_app_id
            target_label: app
          - source_labels:
              - __meta_kubernetes_pod_name
            target_label: pod
          - source_labels:
              - __meta_kubernetes_namespace
            target_label: namespace
          - source_labels:
              - __meta_kubernetes_pod_container_name
            target_label: container
          - replacement: /var/log/pods/*$1/*.log
            separator: /
            source_labels:
              - __meta_kubernetes_pod_uid
              - __meta_kubernetes_pod_container_name
            target_label: __path__

Multiple Nightlamp apps? Promtail's clients[] is a flat list — add one entry per app with that app's literal X-Nightlamp-App-Id header and the matching DSN env var. The same DaemonSet pushes to all targets; deduplication happens server-side per app id.

4. DaemonSet

One Promtail pod per node, mounting /var/log from the host. The --config.expand-env=true flag is required so \${NIGHTLAMP_DSN} in the ConfigMap is substituted from the env var at startup.

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: default
spec:
  selector:
    matchLabels: { app: promtail }
  template:
    metadata:
      labels: { app: promtail }
    spec:
      serviceAccountName: promtail
      tolerations:
        - operator: Exists
          effect: NoSchedule
        - operator: Exists
          effect: NoExecute
      containers:
        - name: promtail
          image: grafana/promtail:3.0.0
          args:
            - -config.file=/etc/promtail/promtail.yaml
            - -config.expand-env=true
          env:
            - name: NIGHTLAMP_DSN
              valueFrom:
                secretKeyRef:
                  name: promtail-dsn
                  key: dsn
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 200m, memory: 200Mi }
          volumeMounts:
            - { name: config,  mountPath: /etc/promtail }
            - { name: varlog,  mountPath: /var/log }
            - { name: pods,    mountPath: /var/log/pods, readOnly: true }
            - { name: runstate, mountPath: /run/promtail }
      volumes:
        - { name: varlog,   hostPath: { path: /var/log } }
        - { name: pods,     hostPath: { path: /var/log/pods } }
        - { name: runstate, hostPath: { path: /run/promtail, type: DirectoryOrCreate } }
        - name: config
          configMap: { name: promtail-config }

5. Opt your workloads in

Add nightlamp.app/id to the pod template of every Deployment, StatefulSet, or Job whose logs you want to ship:

In your app's Deployment manifest

spec:
  template:
    metadata:
      labels:
        nightlamp.app/id: <your-app-id>

6. Apply + verify

Apply the manifests

rbac → secret → configmap → daemonset

kubectl apply -f rbac.yaml
kubectl apply -f configmap.yaml
kubectl apply -f daemonset.yaml

Wait for the rollout

DaemonSet should report Ready on every node

kubectl rollout status daemonset/promtail -n default --timeout=120s

Trigger a test log line
Log a recognizable string from a labeled pod, then open the app's Issue queue in Nightlamp. The first match typically shows up within a few seconds.

Troubleshooting

No lines arriving? Check kubectl logs daemonset/promtail for level=error push failures. The most common cause is an outdated Promtail (pre-2.7) silently dropping the headers map — confirm the image tag.
Lines from non-opted-in pods are showing up? The keep relabel rule above requires the nightlamp.app/id label to be present on the pod itself, not the controller (Deployment / StatefulSet). Add the label to the spec.template.metadata.labels block, not just the controller's top-level labels.
Health-probe noise still leaking? Promtail's drop stage matches the raw log line, not the parsed message. If your access-log format puts the status code in a different position, adjust the regex.