Grafana Loki: Log Aggregation Without the Price Tag

Loki is what you reach for when Elasticsearch feels like overkill and you're tired of paying cloud logging bills that scale with log volume rather than query frequency. The design philosophy is deliberately different from ELK: Loki indexes only labels (like Prometheus), not log content. The full text is stored compressed. This keeps index size tiny and storage costs low — you only pay for the logs themselves, not for indexing every word.

The tradeoff is that full-text search is slower than Elasticsearch. But for most use cases — tailing logs, filtering by service/namespace/pod, correlating logs with metrics — Loki is more than fast enough.

Deploying Loki in Kubernetes

The easiest production-ready deployment is via the loki-stack Helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install loki-stack grafana/loki-stack \
  --namespace monitoring \
  --create-namespace \
  --set loki.enabled=true \
  --set promtail.enabled=true \
  --set grafana.enabled=false \  # assuming Grafana already deployed
  --values loki-values.yaml

For production, configure storage properly in loki-values.yaml:

# loki-values.yaml
loki:
  auth_enabled: false  # set true for multi-tenant

  commonConfig:
    replication_factor: 1  # 3 for HA

  storage:
    type: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      bucketnames: your-loki-logs-bucket
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}

  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  compactor:
    working_directory: /var/loki/compactor
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

  limits_config:
    retention_period: 30d
    ingestion_rate_mb: 16
    ingestion_burst_size_mb: 32
    max_query_length: 721h   # 30 days
    max_streams_per_user: 10000

  rulerConfig:
    storage:
      type: local
      local:
        directory: /var/loki/rules
    rule_path: /var/loki/rules-temp
    alertmanager_url: http://alertmanager:9093
    enable_api: true

The S3 backend is important for anything beyond dev. SQLite/filesystem storage doesn't survive pod restarts cleanly.

Promtail: Getting Logs Into Loki

Promtail is the agent that ships logs from pods to Loki. It runs as a DaemonSet and reads from /var/log/pods/:

# promtail-config (simplified from Helm values)
promtail:
  config:
    snippets:
      pipelineStages:
        - cri: {}   # parse CRI-O/containerd log format

      extraScrapeConfigs: |
        - job_name: kubernetes-pods
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            # Keep only pods in specific namespaces
            - source_labels: [__meta_kubernetes_namespace]
              regex: (default|production|staging|monitoring)
              action: keep

            # Add namespace label
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace

            # Add pod name label
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod

            # Add container name label
            - source_labels: [__meta_kubernetes_pod_container_name]
              target_label: container

            # Add app label from pod labels
            - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
              target_label: app

            # Construct log path
            - source_labels:
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
              target_label: __path__
              separator: /
              replacement: /var/log/pods/*${1}/${2}/*.log

          pipeline_stages:
            - cri: {}
            - json:
                expressions:
                  level: level
                  msg: message
                  ts: timestamp
            - labels:
                level:
            - timestamp:
                source: ts
                format: RFC3339Nano

The relabeling pipeline here is important. You want at minimum namespace, pod, and container labels on every log stream — these are what you'll filter by in queries. The app label from pod labels is also useful for service-level filtering.

LogQL: Querying Logs

LogQL is Loki's query language. It has two parts: a log stream selector and an optional filter/processing pipeline.

Basic Log Stream Selectors

# All logs from a specific namespace
{namespace="production"}

# All logs from a specific app
{namespace="production", app="myapp"}

# All logs from multiple pods (regex)
{namespace="production", pod=~"myapp-.*"}

# All error logs from a service
{namespace="production", app="myapp"} |= "error"

Filtering Pipelines

# Case-insensitive filter
{namespace="production", app="myapp"} |~ "(?i)error"

# Exclude pattern (not containing)
{namespace="production"} != "health check"

# JSON parsing and field filtering
{namespace="production", app="myapp"}
  | json
  | level="error"

# Parse a specific field from structured logs
{namespace="production", app="myapp"}
  | json
  | line_format "{{.level}} {{.message}} {{.request_id}}"

Log Metrics (Rate Queries)

LogQL can also compute metrics from log streams — useful for dashboards without having to instrument your application:

# Error log rate per minute
sum(rate({namespace="production", app="myapp"} |= "error" [5m]))

# Error rate by pod
sum by (pod) (
  rate({namespace="production", app="myapp"} |= "level=\"error\"" [5m])
)

# HTTP 500s from nginx access logs
sum(
  rate({namespace="production", app="nginx"}
    | pattern `<ip> - - [<time>] "<method> <path> <proto>" <status> <size>`
    | status >= 500
  [5m])
)

# Count distinct request IDs (approximate unique users)
count_over_time(
  {namespace="production", app="myapp"}
    | json
    | user_id != ""
  [1h]
)

Useful LogQL Patterns

# Last 100 lines from a crashing pod
{namespace="production", pod="myapp-abc123"} | limit 100

# Logs with a specific trace ID (for distributed tracing correlation)
{namespace="production"} |= "trace_id=abc123def456"

# Extract duration from log line and filter slow requests
{app="myapp"}
  | logfmt
  | duration > 2s

# Multiline stack traces (Loki 2.8+)
{app="myapp"} | multiline firstline=`^\d{4}-\d{2}-\d{2}` maxwaittime=3s

Adding Loki as a Grafana Data Source

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "name": "Loki",
    "type": "loki",
    "url": "http://loki-stack:3100",
    "access": "proxy",
    "isDefault": false,
    "jsonData": {
      "maxLines": 1000,
      "derivedFields": [
        {
          "matcherRegex": "trace_id=(\\w+)",
          "name": "TraceID",
          "url": "http://tempo:3100/api/traces/${__value.raw}",
          "datasourceUid": "tempo"
        }
      ]
    }
  }' \
  "https://grafana.internal/api/datasources"

The derivedFields config is worth noting — it parses trace IDs from log lines and turns them into clickable links to your tracing backend (Tempo, Jaeger, Zipkin). This is the "correlation" feature that ties logs, metrics, and traces together in Grafana.

Log-Based Alerting with Loki Ruler

Loki can evaluate alerting rules directly against log streams — no need to route everything through Prometheus:

# loki-rules/myapp.yaml
groups:
  - name: myapp.log.rules
    rules:
      # Alert on high error log rate
      - alert: HighLogErrorRate
        expr: |
          sum(
            rate({namespace="production", app="myapp"} |= "level=\"error\"" [5m])
          ) > 1
        for: 5m
        labels:
          severity: warning
          service: myapp
        annotations:
          summary: "myapp is logging more than 1 error/sec"
          runbook_url: "https://runbooks.internal/myapp/errors"

      # Alert on OOM kill in logs
      - alert: OOMKillDetected
        expr: |
          count_over_time(
            {namespace="production"} |= "OOMKilled" [5m]
          ) > 0
        labels:
          severity: critical
        annotations:
          summary: "OOMKill detected in production logs"

Apply the rules via the Loki Ruler API:

curl -X POST \
  -H "Content-Type: application/yaml" \
  -H "X-Scope-OrgID: fake" \
  --data-binary @loki-rules/myapp.yaml \
  "http://loki-stack:3100/loki/api/v1/rules/production"

Cost Optimization

Loki's architecture makes it naturally cheap to run, but there are a few optimizations worth knowing:

Chunk compression: Loki compresses log chunks before storing to S3. Default is snappy; switch to zstd for better compression at a slight CPU cost:

loki:
  chunkStoreConfig:
    chunkCacheConfig:
      embeddedCache:
        enabled: true
        maxSizeMB: 500
  storageConfig:
    boltdbShipper:
      activeIndexDirectory: /var/loki/index
      cacheLocation: /var/loki/index_cache

Drop noisy logs before ingestion: Use Promtail pipeline stages to drop logs you don't need:

pipeline_stages:
  - cri: {}
  - match:
      selector: '{namespace="kube-system"}'
      stages:
        - drop:
            expression: ".*kube-proxy.*normal.*"  # drop normal kube-proxy events
  - match:
      selector: '{app="nginx"}'
      stages:
        - drop:
            expression: ".*GET /health.*200.*"   # drop health check hits

Dropping health check and readiness probe logs alone can reduce your Loki ingest volume by 20-30% in a typical Kubernetes cluster.

Retention by label: Configure different retention periods for different namespaces:

loki:
  limits_config:
    retention_period: 30d    # default
    per_stream_rate_limit: 5MB
    per_stream_rate_limit_burst: 15MB

  # Override retention per tenant (requires auth_enabled: true)
  # Or use stream selectors in ruler retention rules

Loki's combination of low operational overhead, Prometheus-compatible label model, and native Grafana integration makes it the right choice for most Kubernetes teams. You're not getting Elasticsearch's full-text search power, but for the 95% of log queries that are "show me errors from this service in the last hour," Loki handles it cleanly and cheaply.