Google GKE: Production Kubernetes Cluster on Google Cloud

GKE's Unique Position

GKE (Google Kubernetes Engine) is the OG managed Kubernetes — Google invented Kubernetes and GKE was the first managed offering. Its distinctive advantages:

• Autopilot mode: Google manages nodes, scaling, and resource allocation — you pay per pod, not per node
• GKE Dataplane V2: eBPF-based networking (built on Cilium) for better performance and network policy
• Binary Authorization: Cryptographically verify only trusted container images can run
• Google Kubernetes Engine Hub: Multi-cluster fleet management
• Best BigQuery/Vertex AI integration: Native connectors for ML workloads

---

Prerequisites

# Install Google Cloud CLI
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init

# Install kubectl (via gcloud)
gcloud components install kubectl

# Install gke-gcloud-auth-plugin
gcloud components install gke-gcloud-auth-plugin

# Set project
gcloud config set project my-project-id
gcloud config set compute/region us-central1

---

Mode 1: Autopilot (Recommended for Most Teams)

Autopilot manages node provisioning, scaling, and security hardening automatically. You only define pods; GKE provisions and pays per pod resource rather than per node:

gcloud container clusters create-auto production-cluster \
  --region us-central1 \
  --release-channel regular \
  --workload-pool my-project-id.svc.id.goog

# Get credentials
gcloud container clusters get-credentials production-cluster \
  --region us-central1

Autopilot enforces best practices: no privileged containers, no host networking, automatic resource limit enforcement. The trade-off is less control over node configuration.

Cost model: ~$0.10/vCPU-hour and ~$0.01/GB-hour for running pods (similar to Fargate). No idle node costs.

---

Mode 2: Standard Cluster

Standard mode gives you control over node configuration. Use this when you need specific machine types, GPUs, or custom node configurations:

gcloud container clusters create production-cluster \
  --region us-central1 \
  --release-channel regular \
  --num-nodes 1 \
  --machine-type e2-standard-4 \
  --disk-size 100GB \
  --disk-type pd-ssd \
  --enable-autoscaling \
  --min-nodes 1 \
  --max-nodes 3 \
  --workload-pool my-project-id.svc.id.goog \
  --enable-ip-alias \
  --enable-network-policy \
  --dataplane-v2 \
  --enable-shielded-nodes \
  --no-enable-basic-auth \
  --no-issue-client-certificate \
  --enable-private-nodes \
  --master-ipv4-cidr 172.16.0.0/28 \
  --enable-master-authorized-networks \
  --master-authorized-networks 10.0.0.0/8

Key flags:

• --enable-private-nodes: Worker nodes have no public IPs (VPN/Cloud NAT for egress)
• --dataplane-v2: eBPF networking with built-in network policies
• --enable-shielded-nodes: Secure Boot, vTPM, integrity monitoring
• --workload-pool: Required for Workload Identity

---

Multiple Node Pools

# Add a high-memory node pool for data processing
gcloud container node-pools create data-nodes \
  --cluster production-cluster \
  --region us-central1 \
  --machine-type n2-highmem-8 \
  --num-nodes 0 \
  --enable-autoscaling \
  --min-nodes 0 \
  --max-nodes 10 \
  --node-labels role=data-processing \
  --node-taints data=true:NoSchedule \
  --disk-type pd-ssd \
  --disk-size 200GB

# Add a Spot node pool for cost savings
gcloud container node-pools create spot-nodes \
  --cluster production-cluster \
  --region us-central1 \
  --machine-type e2-standard-4 \
  --spot \
  --enable-autoscaling \
  --min-nodes 0 \
  --max-nodes 20 \
  --node-labels spot=true

# Scale an existing node pool
gcloud container clusters resize production-cluster \
  --region us-central1 \
  --node-pool default-pool \
  --num-nodes 5

---

Workload Identity: Pods to GCP Services

Workload Identity lets Kubernetes ServiceAccounts impersonate GCP Service Accounts — no credentials in pods:

# Verify Workload Identity is enabled
gcloud container clusters describe production-cluster \
  --region us-central1 \
  --format="value(workloadIdentityConfig.workloadPool)"

# Create a GCP Service Account
gcloud iam service-accounts create myapp-gsa \
  --display-name "MyApp GCP Service Account"

# Grant it permissions (e.g., Cloud Storage read)
gcloud projects add-iam-policy-binding my-project-id \
  --member "serviceAccount:[email protected]" \
  --role "roles/storage.objectViewer"

# Create Kubernetes ServiceAccount
kubectl create serviceaccount myapp-ksa -n production

# Bind KSA to GSA
gcloud iam service-accounts add-iam-policy-binding [email protected] \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project-id.svc.id.goog[production/myapp-ksa]"

# Annotate the KSA
kubectl annotate serviceaccount myapp-ksa \
  --namespace production \
  iam.gke.io/gcp-service-account=[email protected]

Use the ServiceAccount in your pod spec and it automatically has GCP permissions.

---

Terraform Configuration

# main.tf
resource "google_container_cluster" "main" {
  name     = "production-cluster"
  location = "us-central1"

  # Remove default node pool, use separately managed pools
  remove_default_node_pool = true
  initial_node_count       = 1

  # Enable Workload Identity
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  # GKE Dataplane V2 (eBPF)
  datapath_provider = "ADVANCED_DATAPATH"

  # Private cluster
  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }

  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "10.0.0.0/8"
      display_name = "Internal"
    }
  }

  # VPC-native networking
  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  network    = google_compute_network.main.name
  subnetwork = google_compute_subnetwork.main.name

  release_channel {
    channel = "REGULAR"
  }

  # Binary Authorization
  binary_authorization {
    evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE"
  }

  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }

  monitoring_config {
    enable_components = ["SYSTEM_COMPONENTS"]
    managed_prometheus {
      enabled = true
    }
  }
}

resource "google_container_node_pool" "system" {
  name     = "system-pool"
  location = "us-central1"
  cluster  = google_container_cluster.main.name

  initial_node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 5
  }

  node_config {
    machine_type = "e2-standard-4"
    disk_size_gb = 100
    disk_type    = "pd-ssd"

    workload_metadata_config {
      mode = "GKE_METADATA"  # Required for Workload Identity
    }

    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    labels = { role = "system" }
    taint {
      key    = "CriticalAddonsOnly"
      value  = "true"
      effect = "NO_SCHEDULE"
    }
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }
}

resource "google_container_node_pool" "app" {
  name     = "app-pool"
  location = "us-central1"
  cluster  = google_container_cluster.main.name

  initial_node_count = 3

  autoscaling {
    min_node_count = 2
    max_node_count = 20
  }

  node_config {
    machine_type = "e2-standard-8"
    disk_size_gb = 200
    disk_type    = "pd-ssd"

    workload_metadata_config {
      mode = "GKE_METADATA"
    }

    labels = { role = "application" }
    spot   = false  # Set true for Spot instances
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }
}

---

GKE-Specific Add-ons

Cloud SQL Auth Proxy (database connectivity)

Instead of opening firewall rules, use the Cloud SQL Auth Proxy sidecar:

containers:
  - name: myapp
    image: gcr.io/my-project/myapp:latest
    env:
      - name: DB_HOST
        value: "127.0.0.1"
  - name: cloud-sql-proxy
    image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2
    args:
      - "--structured-logs"
      - "--port=5432"
      - "my-project:us-central1:my-db-instance"
    securityContext:
      runAsNonRoot: true

Google Cloud Managed Service for Prometheus

GKE integrates with Google Cloud Managed Prometheus out of the box:

# Enable on existing cluster
gcloud container clusters update production-cluster \
  --enable-managed-prometheus \
  --region us-central1

No Prometheus server to manage — just deploy PodMonitoring resources and metrics flow to Google Cloud Monitoring.

---

Upgrading GKE

# Check available upgrade targets
gcloud container get-server-config --region us-central1

# Upgrade control plane
gcloud container clusters upgrade production-cluster \
  --master \
  --cluster-version 1.30 \
  --region us-central1

# Upgrade node pool (in-place rolling update)
gcloud container clusters upgrade production-cluster \
  --node-pool default-pool \
  --cluster-version 1.30 \
  --region us-central1

# Check upgrade status
gcloud container operations list \
  --filter="status=RUNNING"

With REGULAR release channel, GKE automatically upgrades the control plane within weeks of a new minor version release. Node pool upgrades trigger automatically based on the channel schedule unless you use --no-enable-autoupgrade.