Azure AKS: Production Kubernetes Cluster Setup and Configuration

AKS Architecture Overview

AKS (Azure Kubernetes Service) manages the Kubernetes control plane at no extra cost — you pay only for worker node VMs. The key differentiators from EKS:

• Free control plane (EKS charges $0.10/hour)
• Deep Entra ID integration — use Azure AD groups for RBAC
• Azure CNI — pods get real VNet IPs (better for Azure-native workloads)
• Managed identities — no service principal secrets to rotate
• Node auto-provisioning — similar to Karpenter, built into AKS

---

Prerequisites

# Install Azure CLI
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Install kubectl
az aks install-cli

# Login
az login
az account set --subscription "your-subscription-id"

---

Create a Resource Group and Cluster (CLI)

# Resource group
az group create \
  --name production-rg \
  --location eastus

# Create AKS cluster
az aks create \
  --resource-group production-rg \
  --name production-aks \
  --kubernetes-version 1.29 \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --enable-managed-identity \
  --network-plugin azure \
  --network-policy azure \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 10 \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --enable-addons monitoring \
  --workspace-resource-id /subscriptions/.../resourceGroups/.../providers/Microsoft.OperationalInsights/workspaces/my-workspace \
  --generate-ssh-keys

# Get credentials
az aks get-credentials \
  --resource-group production-rg \
  --name production-aks

kubectl get nodes

---

Production Cluster with Multiple Node Pools

AKS supports multiple node pools — different VM sizes, OS types, and taints for different workloads:

# System node pool is created with the cluster
# Add a user node pool for application workloads
az aks nodepool add \
  --resource-group production-rg \
  --cluster-name production-aks \
  --name appnodes \
  --node-count 3 \
  --node-vm-size Standard_D8s_v3 \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 20 \
  --node-taints workload=application:NoSchedule \
  --labels role=application

# Add a Spot node pool for batch workloads
az aks nodepool add \
  --resource-group production-rg \
  --cluster-name production-aks \
  --name spotnodes \
  --node-count 0 \
  --node-vm-size Standard_D4s_v3 \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --enable-cluster-autoscaler \
  --min-count 0 \
  --max-count 10

Reserve the system node pool for critical system pods (CoreDNS, metrics-server) by tainting it:

az aks nodepool update \
  --resource-group production-rg \
  --cluster-name production-aks \
  --name nodepool1 \
  --node-taints CriticalAddonsOnly=true:NoSchedule

---

Terraform Configuration

# main.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "main" {
  name     = "production-rg"
  location = "East US"
}

resource "azurerm_kubernetes_cluster" "main" {
  name                = "production-aks"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "production-aks"
  kubernetes_version  = "1.29"

  # System node pool
  default_node_pool {
    name                = "system"
    node_count          = 3
    vm_size             = "Standard_D4s_v3"
    enable_auto_scaling = true
    min_count           = 2
    max_count           = 5
    os_disk_size_gb     = 128
    os_disk_type        = "Managed"
    vnet_subnet_id      = azurerm_subnet.aks.id
    node_labels = {
      role = "system"
    }
    node_taints = ["CriticalAddonsOnly=true:NoSchedule"]
  }

  # Use managed identity
  identity {
    type = "SystemAssigned"
  }

  # Azure CNI networking
  network_profile {
    network_plugin    = "azure"
    network_policy    = "azure"
    load_balancer_sku = "standard"
    service_cidr      = "10.100.0.0/16"
    dns_service_ip    = "10.100.0.10"
  }

  # Azure Monitor integration
  oms_agent {
    log_analytics_workspace_id = azurerm_log_analytics_workspace.main.id
  }

  # Enable workload identity + OIDC
  oidc_issuer_enabled       = true
  workload_identity_enabled = true

  # Azure RBAC for Kubernetes (Entra ID integration)
  azure_active_directory_role_based_access_control {
    managed                = true
    azure_rbac_enabled     = true
    admin_group_object_ids = [var.aks_admin_group_id]
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# Application node pool
resource "azurerm_kubernetes_cluster_node_pool" "app" {
  name                  = "appnodes"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id
  vm_size               = "Standard_D8s_v3"
  node_count            = 3
  enable_auto_scaling   = true
  min_count             = 2
  max_count             = 20
  vnet_subnet_id        = azurerm_subnet.aks.id

  node_labels = { role = "application" }
  node_taints = ["workload=application:NoSchedule"]

  tags = {
    Environment = "production"
  }
}

---

Azure Container Registry (ACR) Integration

# Create ACR
az acr create \
  --resource-group production-rg \
  --name productionregistry \
  --sku Premium

# Grant AKS pull access (attach ACR to cluster)
az aks update \
  --resource-group production-rg \
  --name production-aks \
  --attach-acr productionregistry

# Build and push image
az acr build \
  --registry productionregistry \
  --image myapp:v1.0 \
  .

No credentials needed in Kubernetes — the managed identity handles authentication to ACR.

---

Workload Identity (OIDC-based, replaces Pod Identity)

Workload Identity lets pods authenticate to Azure services without secrets:

# Get OIDC issuer URL
OIDC_ISSUER=$(az aks show \
  --resource-group production-rg \
  --name production-aks \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

# Create managed identity
az identity create \
  --name myapp-identity \
  --resource-group production-rg

CLIENT_ID=$(az identity show \
  --name myapp-identity \
  --resource-group production-rg \
  --query clientId -o tsv)

# Grant the identity access to Key Vault
az keyvault set-policy \
  --name my-keyvault \
  --secret-permissions get list \
  --spn $CLIENT_ID

# Federate the identity with the Kubernetes ServiceAccount
az identity federated-credential create \
  --name myapp-federated-credential \
  --identity-name myapp-identity \
  --resource-group production-rg \
  --issuer $OIDC_ISSUER \
  --subject "system:serviceaccount:production:myapp-sa"

Create the ServiceAccount with the client ID annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
  annotations:
    azure.workload.identity/client-id: "<client-id-from-above>"

Label your pod to use workload identity:

spec:
  serviceAccountName: myapp-sa
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"

The pod gets Azure credentials injected automatically and can call Azure Key Vault, Storage, and other services.

---

Azure Secrets Store CSI Driver

Mount Key Vault secrets as files in pods:

# Enable the secrets store add-on
az aks enable-addons \
  --resource-group production-rg \
  --name production-aks \
  --addons azure-keyvault-secrets-provider

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: myapp-keyvault
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "<managed-identity-client-id>"
    keyvaultName: "my-keyvault"
    tenantId: "<tenant-id>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
          objectVersion: ""
  secretObjects:
    - secretName: myapp-secret
      type: Opaque
      data:
        - objectName: db-password
          key: DB_PASSWORD

volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: myapp-keyvault

---

Monitoring with Azure Monitor

# Enable Container Insights on existing cluster
az aks enable-addons \
  --resource-group production-rg \
  --name production-aks \
  --addons monitoring \
  --workspace-resource-id /subscriptions/.../workspaces/my-workspace

# Enable Prometheus metrics
az aks update \
  --resource-group production-rg \
  --name production-aks \
  --enable-azure-monitor-metrics

Container Insights provides pre-built dashboards for node CPU/memory, pod status, and container logs in the Azure portal under the AKS cluster's Insights tab.

---

Upgrading AKS

# List available versions
az aks get-upgrades \
  --resource-group production-rg \
  --name production-aks

# Upgrade control plane first
az aks upgrade \
  --resource-group production-rg \
  --name production-aks \
  --kubernetes-version 1.30 \
  --control-plane-only

# Then upgrade node pools
az aks nodepool upgrade \
  --resource-group production-rg \
  --cluster-name production-aks \
  --name nodepool1 \
  --kubernetes-version 1.30

# Watch progress
az aks show \
  --resource-group production-rg \
  --name production-aks \
  --query "provisioningState"

Enable auto-upgrade for non-production clusters:

az aks update \
  --resource-group staging-rg \
  --name staging-aks \
  --auto-upgrade-channel patch