Jenkins Kubernetes Agents: Dynamic Build Pods for Scalable CI/CD

The Problem with Static Agents

Traditional Jenkins uses static agents — dedicated VMs or physical machines that sit idle between builds. This wastes money, creates snowflake machines that drift over time, and caps your concurrency at the number of nodes you've provisioned.

Kubernetes agents solve all three problems. When a build starts, Jenkins spawns a pod in your cluster. When the build finishes, the pod is deleted. You pay for compute only during builds, every build starts from a clean image, and concurrency scales to your cluster capacity.

---

Install the Kubernetes Plugin

In Jenkins: Manage Jenkins → Plugins → Available — search for Kubernetes and install it.

After installation: Manage Jenkins → Clouds → New cloud → Kubernetes.

Configure the cloud:

Field	Value
Kubernetes URL	https://kubernetes.default.svc (in-cluster) or your API endpoint
Kubernetes Namespace	jenkins
Jenkins URL	http://jenkins.jenkins.svc.cluster.local:8080
Jenkins tunnel	jenkins.jenkins.svc.cluster.local:50000

If Jenkins runs inside the cluster, leave credentials empty — the plugin uses the pod's service account. If Jenkins runs outside, provide a kubeconfig credential.

---

RBAC for In-Cluster Jenkins

Grant Jenkins permission to create and delete pods:

# jenkins-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-agent-role
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims", "events"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-agent-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-agent-role
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: jenkins

kubectl apply -f jenkins-rbac.yaml

---

Default Pod Template

Define a default pod template in the Kubernetes cloud configuration, or in a Jenkinsfile. The minimum viable template:

// Jenkinsfile using the default JNLP agent
pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
            - name: jnlp
              image: jenkins/inbound-agent:latest
              resources:
                requests:
                  cpu: 100m
                  memory: 256Mi
                limits:
                  cpu: 500m
                  memory: 512Mi
      '''
    }
  }
  stages {
    stage('Hello') {
      steps {
        sh 'echo "Running on Kubernetes agent"'
        sh 'hostname'
      }
    }
  }
}

The jnlp container is special — it's the agent that connects back to Jenkins. You always need it. Everything else you add is a sidecar.

---

Multi-Container Pods

The real power: run multiple tool containers in a single pod and switch between them with container('name'):

pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
            - name: jnlp
              image: jenkins/inbound-agent:latest
            - name: maven
              image: maven:3.9-eclipse-temurin-21
              command: ["sleep"]
              args: ["infinity"]
            - name: kaniko
              image: gcr.io/kaniko-project/executor:debug
              command: ["sleep"]
              args: ["infinity"]
            - name: kubectl
              image: bitnami/kubectl:latest
              command: ["sleep"]
              args: ["infinity"]
      '''
    }
  }

  stages {
    stage('Build JAR') {
      steps {
        container('maven') {
          sh 'mvn package -DskipTests'
        }
      }
    }

    stage('Build Image') {
      steps {
        container('kaniko') {
          sh '''
            /kaniko/executor \
              --context=dir://${WORKSPACE} \
              --dockerfile=Dockerfile \
              --destination=registry.example.com/my-app:${BUILD_NUMBER}
          '''
        }
      }
    }

    stage('Deploy') {
      steps {
        container('kubectl') {
          sh "kubectl set image deployment/my-app my-app=registry.example.com/my-app:${BUILD_NUMBER}"
        }
      }
    }
  }
}

Notice command: ["sleep"] args: ["infinity"] — this prevents containers from exiting immediately. The jnlp container manages its own lifecycle, so don't set this on it.

---

Shared Workspace via emptyDir

All containers in a pod share /home/jenkins/agent (the default workspace) via a shared emptyDir volume. Files created in maven are visible in kaniko. No extra volume configuration needed.

For persistent data across builds (like a local Maven cache to speed up builds), use a PVC:

yaml '''
  apiVersion: v1
  kind: Pod
  spec:
    volumes:
      - name: maven-cache
        persistentVolumeClaim:
          claimName: maven-local-repo-pvc
    containers:
      - name: maven
        image: maven:3.9-eclipse-temurin-21
        command: ["sleep"]
        args: ["infinity"]
        volumeMounts:
          - name: maven-cache
            mountPath: /root/.m2
'''

---

Pod Templates in Jenkins UI

For templates reused across many pipelines, define them in the Kubernetes cloud config instead of each Jenkinsfile:

1. Go to Manage Jenkins → Clouds → (your K8s cloud) → Pod Templates → Add
2. Set Label (e.g., maven-agent) — this is what pipelines reference
3. Add containers, environment variables, volumes, resource limits

Then in Jenkinsfiles:

pipeline {
  agent { label 'maven-agent' }
  // ...
}

---

Resource Limits and Node Selectors

Always set resource requests and limits. Without them, pods can starve other workloads or consume unbounded resources:

yaml '''
  apiVersion: v1
  kind: Pod
  spec:
    nodeSelector:
      kubernetes.io/os: linux
      node-role: ci-builders
    tolerations:
      - key: ci-only
        operator: Exists
        effect: NoSchedule
    containers:
      - name: jnlp
        image: jenkins/inbound-agent:latest
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 2
            memory: 2Gi
'''

Use a dedicated node pool for CI builds (tainted with ci-only) to isolate build workloads from production services.

---

Docker-in-Docker vs Kaniko

Building container images inside a Kubernetes agent has two approaches:

Docker-in-Docker (DinD) — runs a Docker daemon as a sidecar. Works but requires privileged: true, which is a security risk in multi-tenant clusters.

Kaniko — builds images without a daemon, no root, no privileges needed. Recommended for Kubernetes:

container('kaniko') {
  sh '''
    /kaniko/executor \
      --context=dir://${WORKSPACE} \
      --dockerfile=${WORKSPACE}/Dockerfile \
      --destination=${REGISTRY}/${APP}:${BUILD_NUMBER} \
      --cache=true \
      --cache-repo=${REGISTRY}/${APP}/cache
  '''
}

For registry authentication, mount a Kubernetes secret as a volume at /kaniko/.docker/config.json.

---

Parallel Stages on Separate Pods

Each parallel branch can run on its own pod:

pipeline {
  agent none

  stages {
    stage('Test') {
      parallel {
        stage('Unit Tests') {
          agent {
            kubernetes {
              yaml '''
                spec:
                  containers:
                    - name: jnlp
                      image: jenkins/inbound-agent:latest
                    - name: node
                      image: node:20-alpine
                      command: ["sleep"]
                      args: ["infinity"]
              '''
            }
          }
          steps {
            container('node') {
              sh 'npm ci && npm run test:unit'
            }
          }
        }
        stage('E2E Tests') {
          agent {
            kubernetes {
              yaml '''
                spec:
                  containers:
                    - name: jnlp
                      image: jenkins/inbound-agent:latest
                    - name: cypress
                      image: cypress/included:latest
                      command: ["sleep"]
                      args: ["infinity"]
              '''
            }
          }
          steps {
            container('cypress') {
              sh 'cypress run'
            }
          }
        }
      }
    }
  }
}

Each parallel branch gets a dedicated pod, so they truly run concurrently without sharing resources.

---

Troubleshooting

Pod stuck in Pending — check if the cluster has enough capacity (kubectl describe pod <build-pod>). Check node selectors and tolerations match actual node labels.

JNLP connection refused — ensure the Jenkins tunnel address is reachable from within the cluster. The port 50000 must be open on the Jenkins service.

Workspace empty in second container — you likely set a different workingDir. All containers default to /home/jenkins/agent. Set workingDir explicitly on all containers to match.

Build takes forever to start — the image pull is slow. Use a local registry mirror or pre-pull agent images onto your CI nodes.

Out of memory kills — increase the memory limit or check if your build process spawns unexpected subprocesses. Use kubectl top pod <build-pod> during a build to see actual usage.