Running Kafka on Kubernetes with Strimzi Operator

Running Kafka on bare VMs is straightforward until you need to scale, patch, or recover from a broker failure at 3am. Strimzi brings Kafka into the Kubernetes operator pattern — meaning the cluster reconciles toward desired state automatically, broker replacements are handled for you, and rolling upgrades happen without manual coordination. This guide covers a production-ready Strimzi deployment from install to first message in flight.

Why Strimzi Over a Managed Kafka Service?

Managed Kafka (MSK, Confluent Cloud, Aiven) is the right call for many teams. Strimzi makes sense when:

• You need Kafka co-located with your workloads inside a private cluster (compliance, latency)
• You are already running Kubernetes and want a unified control plane
• You want to avoid per-message egress costs at high throughput
• You need custom Kafka configurations that managed services lock down

The tradeoff is operational responsibility. Strimzi removes a lot of it but not all of it.

Installing the Strimzi Operator

Strimzi uses CRDs and a controller deployment. Install via Helm:

helm repo add strimzi https://strimzi.io/charts/
helm repo update

kubectl create namespace kafka

helm install strimzi-operator strimzi/strimzi-kafka-operator \
  --namespace kafka \
  --version 0.41.0 \
  --set watchAnyNamespace=false \
  --set defaultImageRegistry=quay.io \
  --wait

Verify the operator is running:

kubectl get pods -n kafka
# NAME                                        READY   STATUS    RESTARTS
# strimzi-cluster-operator-6d4f5b5c6b-x9k2p  1/1     Running   0

kubectl get crds | grep strimzi
# kafkas.kafka.strimzi.io
# kafkatopics.kafka.strimzi.io
# kafkausers.kafka.strimzi.io
# kafkaconnects.kafka.strimzi.io

Deploying a Production Kafka Cluster

The Kafka CRD is the core resource. Everything about your cluster — broker count, storage, listeners, Zookeeper (or KRaft) — lives in one YAML file:

# kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod-kafka
  namespace: kafka
spec:
  kafka:
    version: 3.7.0
    replicas: 3

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls

    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      # Retention
      log.retention.hours: 168
      log.retention.bytes: "10737418240"   # 10 GB per partition
      # Performance
      num.network.threads: 8
      num.io.threads: 16
      socket.send.buffer.bytes: "102400"
      socket.receive.buffer.bytes: "102400"
      socket.request.max.bytes: "104857600"

    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 500Gi
          class: fast-ssd
          deleteClaim: false

    resources:
      requests:
        memory: 8Gi
        cpu: "2"
      limits:
        memory: 12Gi
        cpu: "4"

    jvmOptions:
      -Xms: 4096m
      -Xmx: 4096m
      gcLoggingEnabled: false

    rack:
      topologyKey: topology.kubernetes.io/zone

    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - prod-kafka-kafka
                topologyKey: kubernetes.io/hostname

  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 50Gi
      class: fast-ssd
      deleteClaim: false
    resources:
      requests:
        memory: 2Gi
        cpu: "500m"
      limits:
        memory: 2Gi
        cpu: "1"

  entityOperator:
    topicOperator: {}
    userOperator: {}

Apply and wait for the cluster to become ready:

kubectl apply -f kafka-cluster.yaml

kubectl wait kafka/prod-kafka \
  --for=condition=Ready \
  --timeout=300s \
  -n kafka

The rack configuration combined with podAntiAffinity ensures brokers spread across availability zones and no two brokers share a node — critical for surviving a node failure without data loss.

KRaft Mode (No Zookeeper)

For new clusters on Kafka 3.7+, use KRaft mode. It removes the Zookeeper dependency entirely:

spec:
  kafka:
    version: 3.7.0
    replicas: 3
    metadataVersion: 3.7-IV4

    # KRaft mode: controllers embedded in broker pods
    # Remove the entire zookeeper section

  # No zookeeper block needed

KRaft is stable and recommended for all new Strimzi deployments. The main caveat: migrating an existing ZooKeeper cluster to KRaft requires Strimzi's migration procedure, not a simple config change.

Managing Topics with KafkaTopic CRD

Forget kafka-topics.sh in production. Use the KafkaTopic CRD so topic configuration is version-controlled and reconciled:

# topic-orders.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  namespace: kafka
  labels:
    strimzi.io/cluster: prod-kafka
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: "604800000"        # 7 days
    retention.bytes: "5368709120"    # 5 GB per partition
    cleanup.policy: delete
    compression.type: lz4
    min.insync.replicas: "2"
    segment.bytes: "1073741824"      # 1 GB segments

kubectl apply -f topic-orders.yaml

# Verify topic was created
kubectl get kafkatopic orders -n kafka -o yaml

If you delete a KafkaTopic resource, the operator will delete the actual Kafka topic. In production, set strimzi.io/managed: false on topics you want to protect from accidental deletion.

TLS Authentication with KafkaUser

Strimzi generates and rotates TLS certificates automatically. Define a user with a client certificate:

# kafka-user-producer.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: orders-producer
  namespace: kafka
  labels:
    strimzi.io/cluster: prod-kafka
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: orders
          patternType: literal
        operations: [Write, Describe, Create]
        host: "*"
      - resource:
          type: transactionalId
          name: orders-producer
          patternType: prefix
        operations: [Write, Describe]
        host: "*"

kubectl apply -f kafka-user-producer.yaml

# The operator creates a Secret with the client cert
kubectl get secret orders-producer -n kafka -o jsonpath='{.data.user\.crt}' | base64 -d

Your producer application mounts this secret and uses it for mTLS authentication — no passwords, no rotating API keys manually.

Producer Configuration for Durability

# producer.properties
bootstrap.servers=prod-kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
security.protocol=SSL
ssl.truststore.location=/mnt/certs/ca.p12
ssl.truststore.password=${CA_PASSWORD}
ssl.keystore.location=/mnt/certs/user.p12
ssl.keystore.password=${USER_PASSWORD}

# Durability settings
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000

# Performance
batch.size=65536
linger.ms=5
compression.type=lz4
buffer.memory=67108864

acks=all with min.insync.replicas=2 (set on the broker) means a message is only acknowledged once written to at least 2 replicas. Combined with enable.idempotence=true, this gives you exactly-once delivery semantics for producers.

Monitoring with Prometheus and Grafana

Strimzi exposes JMX metrics via a Prometheus JMX exporter. Enable it in the Kafka spec:

spec:
  kafka:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics-config
          key: kafka-metrics-config.yml

# Scrape metrics
curl -s http://prod-kafka-kafka-0.prod-kafka-kafka-brokers.kafka.svc.cluster.local:9404/metrics | \
  grep kafka_server_BrokerTopicMetrics | head -20

Key Kafka metrics to alert on:

Metric	Alert Threshold	Meaning
kafka_server_ReplicaManager_UnderReplicatedPartitions	> 0	Partitions missing replicas
kafka_controller_KafkaController_ActiveControllerCount	!= 1	Controller election problem
kafka_server_BrokerTopicMetrics_MessagesInPerSec	Sudden drop to 0	Producer connectivity issue
kafka_network_RequestMetrics_RequestQueueTimeMs_p99	> 100ms	Broker under heavy load
kafka_log_Log_LogEndOffset	Not advancing	Stalled consumer check
JVM heap used %	> 80%	GC pressure building

Rolling Upgrade Procedure

Strimzi handles rolling upgrades automatically when you update the version field:

# Update Kafka version in the CRD
kubectl patch kafka prod-kafka -n kafka \
  --type merge \
  -p '{"spec":{"kafka":{"version":"3.8.0"}}}'

# Watch the rolling update
kubectl get pods -n kafka -w
# Strimzi will restart one broker at a time, waiting for ISR to recover

The operator pauses between each broker restart and only proceeds once partition leadership has redistributed and all partitions are in-sync. This is the single biggest operational advantage over managing Kafka manually.

Cluster Sizing Reference

Throughput	Brokers	vCPU/broker	RAM/broker	Storage/broker
< 50 MB/s	3	4	16 GB	200 GB NVMe
50-200 MB/s	3-5	8	32 GB	500 GB NVMe
200 MB/s - 1 GB/s	6-12	16	64 GB	2 TB NVMe
> 1 GB/s	12+	32	128 GB	4 TB NVMe

Always provision at least 3 brokers for fault tolerance. With min.insync.replicas=2 and replication.factor=3, your cluster survives one broker failure with zero data loss.

Wrapping Up

Strimzi turns Kafka cluster management from a bespoke operational discipline into a declarative Kubernetes workflow. You define the desired state, the operator reconciles it, and rolling upgrades happen without scripting. The key production decisions are:

• Use podAntiAffinity and rack configuration to spread brokers across zones.
• Always set acks=all, min.insync.replicas=2, and replication.factor=3 together.
• Manage topics and users as CRDs so configuration is version-controlled.
• Monitor UnderReplicatedPartitions — it is the earliest warning of cluster trouble.

The operational overhead of running Kafka on Kubernetes with Strimzi is real but predictable. And predictable is what you want when your event stream carries production-critical data.