Kubernetes Vertical Pod Autoscaler: Automating Resource Request Tuning in Production

Let me be direct with you: most Kubernetes clusters I audit are hemorrhaging money because of poorly configured resource requests. I've seen teams running pods with 4 CPU cores requested that consistently use 200 millicores. That's not a minor inefficiency — that's a 95% waste rate on your most expensive line item.

The Vertical Pod Autoscaler (VPA) is one of the most underutilized tools in the Kubernetes ecosystem, and it directly solves this problem. In this guide, I'll show you how to deploy, configure, and operate VPA in production — including the parts most tutorials conveniently skip.

What VPA Actually Does (And What It Doesn't)

Before we write a single line of YAML, let's establish a mental model. VPA monitors your pod's actual resource consumption over time and adjusts its resource requests and limits accordingly. It answers the question: "How much CPU and memory does this workload actually need?"

Here's what it does not do:

• It doesn't scale the number of pod replicas (that's HPA's job)
• It doesn't resize pods in-place without a restart (by default — more on this later)
• It doesn't replace the need for load testing and profiling

VPA has three operational modes:

Mode	What Happens
Off	Recommendations are generated but nothing is changed
Initial	Resources are set only at pod creation time
Auto	Resources are updated and pods are restarted when needed

For most teams, I recommend starting with Off mode to build confidence in the recommendations, then graduating to Auto for appropriate workloads.

Installing VPA

VPA isn't bundled with Kubernetes — you need to install it separately. The official installation is straightforward:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

This installs three components:

• VPA Admission Controller: Mutates pod specs at creation time
• VPA Recommender: Watches metrics and generates recommendations
• VPA Updater: Evicts pods that need resource adjustment

Verify the installation:

kubectl get pods -n kube-system | grep vpa

# Expected output:
# vpa-admission-controller-xxxx   1/1   Running
# vpa-recommender-xxxx            1/1   Running
# vpa-updater-xxxx                1/1   Running

If you're using Helm (which you should be in production):

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace vpa \
  --create-namespace \
  --set "recommender.extraArgs.storage=prometheus" \
  --set "recommender.extraArgs.prometheus-address=http://prometheus.monitoring.svc.cluster.local:9090"

Connecting VPA to Prometheus gives you a much longer historical window for recommendations — the default in-memory history only goes back 8 days.

Your First VPA Object

Let's start with a real-world example. Imagine you have a Node.js API that someone initially configured with these requests:

# Before VPA: blindly over-provisioned
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

Here's the VPA object you'd create to start monitoring it:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nodejs-api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: nodejs-api
  updatePolicy:
    updateMode: "Off"  # Start in observation mode
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 2
          memory: 2Gi
        controlledResources: ["cpu", "memory"]
        controlledValues: RequestsAndLimits

After deploying this, check the recommendations after a few hours (ideally 7+ days for production workloads):

kubectl describe vpa nodejs-api-vpa -n production

You'll see output like this:

Recommendation:
  Container Recommendations:
    Container Name:  nodejs-api
    Lower Bound:
      Cpu:     60m
      Memory:  105Mi
    Target:
      Cpu:     125m
      Memory:  262Mi
    Uncapped Target:
      Cpu:     125m
      Memory:  262Mi
    Upper Bound:
      Cpu:     600m
      Memory:  1Gi

That "Target" is VPA saying: "Based on actual usage, this container needs 125 millicores and 262Mi — not the 500m/512Mi you've been paying for." On a 100-replica deployment, that's a massive difference in your node count and cloud bill.

Production-Grade VPA Configuration

Let me walk through a more complete configuration that addresses the real production concerns.

Handling Multiple Containers

Most production pods have sidecars — Envoy proxies, logging agents, secrets managers. VPA can handle each container independently:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: web-service
  updatePolicy:
    updateMode: "Auto"
    minReplicas: 2  # Never evict if fewer than 2 replicas are available
  resourcePolicy:
    containerPolicies:
      # Main application container
      - containerName: "web-app"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4
          memory: 4Gi
        controlledValues: RequestsAndLimits
      
      # Envoy sidecar - we know its profile well
      - containerName: "envoy-proxy"
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 500m
          memory: 512Mi
        controlledValues: RequestsOnly  # Don't touch limits for the proxy
      
      # Logging sidecar - exclude entirely from VPA management
      - containerName: "fluentd"
        mode: "Off"

The controlledValues: RequestsOnly option is something I use frequently. It lets VPA tune requests (which affect scheduling) without touching limits (which affect throttling behavior). This is useful when you want predictable throttling behavior but more efficient scheduling.

Configuring the Recommender for Your Workload Profile

The VPA recommender uses a histogram-based algorithm. You can tune its behavior with flags:

# In your Helm values or recommender deployment
containers:
- name: recommender
  args:
    - --v=4
    # How much historical data to use
    - --history-length=336h  # 14 days instead of default 8 days
    # CPU recommendation safety margin
    - --recommendation-margin-fraction=0.15
    # Percentile for CPU recommendations (default 0.9)
    - --target-cpu-percentile=0.9
    # Don't recommend more than 3x current request
    - --pod-recommendation-min-cpu-millicores=25
    - --pod-recommendation-min-memory-mb=250

The --target-cpu-percentile=0.9 setting is critical. At P90, VPA recommends enough CPU for 90% of your traffic patterns. For APIs with burst patterns, you might want P95 or P99. For batch jobs, P50 might be perfectly fine.

The VPA and HPA Compatibility Problem (And How to Solve It)

Here's where many teams get burned: VPA and HPA conflict when both target CPU or memory.

If HPA is scaling based on CPU utilization percentage and VPA is simultaneously changing the CPU request, you get an unstable feedback loop. HPA sees high utilization → adds replicas. VPA sees increased requests → changes the baseline. The system never stabilizes.

The solutions:

Option 1: Use HPA for scaling, VPA in Off mode for recommendations only

This is the safest approach. Use VPA purely as a recommendation engine, periodically apply its suggestions to your deployment manifests:

# Script to extract and apply VPA recommendations
#!/bin/bash

VPA_NAME=$1
NAMESPACE=${2:-default}

# Get the target recommendation
TARGET_CPU=$(kubectl get vpa $VPA_NAME -n $NAMESPACE \
  -o jsonpath='{.status.recommendation.containerRecommendations[0].target.cpu}')
TARGET_MEMORY=$(kubectl get vpa $VPA_NAME -n $NAMESPACE \
  -o jsonpath='{.status.recommendation.containerRecommendations[0].target.memory}')

echo "VPA Recommendation for $VPA_NAME:"
echo "  CPU: $TARGET_CPU"
echo "  Memory: $TARGET_MEMORY"

# Optionally patch the deployment (with human review first!)
# kubectl patch deployment $VPA_NAME -n $NAMESPACE \
#   -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"app\",\"resources\":{\"requests\":{\"cpu\":\"$TARGET_CPU\",\"memory\":\"$TARGET_MEMORY\"}}}]}}}}"

Option 2: Use HPA on custom metrics, VPA on CPU/memory

If your HPA is scaling on application-level metrics (requests per second, queue depth), you can safely run VPA on CPU/memory simultaneously:

# HPA scaling on custom metric - safe to use with VPA
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: External
    external:
      metric:
        name: requests_per_second
        selector:
          matchLabels:
            service: web-service
      target:
        type: AverageValue
        averageValue: "1000"

Option 3: Multidimensional Pod Autoscaler (MPA)

For teams running on GKE, Google's MPA coordinates HPA and VPA together. For everyone else, stick with Option 1 or 2.

Handling the Restart Problem

The biggest operational objection to VPA in Auto mode is pod restarts. When VPA updates resources, it evicts the pod, and a new one starts with the new resource spec. For stateful workloads or services where restarts are painful, this is a real concern.

Mitigation strategies:

1. Use Pod Disruption Budgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-service-pdb
spec:
  minAvailable: "75%"  # Always keep 75% of replicas running
  selector:
    matchLabels:
      app: web-service

VPA respects PDBs. If evicting a pod would violate your PDB, VPA waits. This is your primary safety mechanism.

2. Set Conservative Update Windows

You can configure the VPA updater to only apply changes when the deviation from current resources is significant:

# Updater configuration
containers:
- name: updater
  args:
    # Only update if recommendation differs by more than this fraction
    - --eviction-tolerance=0.5
    # Require at least this many pods in the group before evicting
    - --min-replicas=2

3. In-Place Pod Resizing (Kubernetes 1.27+)

This is exciting news for the VPA ecosystem. Kubernetes 1.27 introduced the InPlacePodVerticalScaling feature gate, which allows CPU (and in some cases memory) changes without a pod restart:

# Requires feature gate: InPlacePodVerticalScaling=true
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "InPlaceOrRecreate"  # Try in-place first, restart if needed

As of writing, this is still maturing, but it's the future of VPA operations. Memory resizing still typically requires a restart due to OS-level constraints.

Real-World Scenario: Rightsizing a Java Microservices Fleet

Here's a realistic scenario I encounter frequently. A team has 40 Java microservices, all configured with identical resource requests because "we don't know what they need" and nobody wants to be responsible for an OOM kill.

The current state:

• Every service: requests: {cpu: 1000m, memory: 2Gi}, limits: {cpu: 2000m, memory: 4Gi}
• 40 services × average 5 replicas = 200 pods
• Actual observed usage: ~150m CPU, ~512Mi memory per pod

First, deploy VPA for all services in Off mode using a templated approach:

# vpa-template.yaml - apply with kustomize or helm for each service
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: {{ .serviceName }}-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: {{ .serviceName }}
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: "app"
        minAllowed:
          cpu: 50m
          memory: 256Mi  # Java needs a reasonable minimum for the JVM
        maxAllowed:
          cpu: 4
          memory: 8Gi
        controlledValues: RequestsAndLimits

After 2 weeks of data collection, write a script to harvest all recommendations:

#!/bin/bash
# harvest-vpa-recommendations.sh

echo "Service,Container,Target CPU,Target Memory,Lower CPU,Lower Memory,Upper CPU,Upper Memory"

for vpa in $(kubectl get vpa -n production -o name); do
  VPA_NAME=$(echo $vpa | cut -d/ -f2)
  
  # Get container recommendations
  kubectl get vpa $VPA_NAME -n production -o json | \
    jq -r --arg name "$VPA_NAME" '
      .status.recommendation.containerRecommendations[]? |
      [$name, .containerName,
       .target.cpu, .target.memory,
       .lowerBound.cpu, .lowerBound.memory,
       .upperBound.cpu, .upperBound.memory] |
      @csv'
done

Run this, import to a spreadsheet, and you'll typically see something like:

Service	Target CPU	Target Memory	Current CPU	Savings/Pod
auth-service	95m	480Mi	1000m	~90% CPU
payment-service	380m	1.2Gi	1000m	~62% CPU
notification-service	45m	290Mi	1000m	~95% CPU

On a cluster running on c5.2xlarge instances ($0.34/hour), rightsizing from 200 pods at the original requests to 200 pods at actual needs can reduce your required node count dramatically. I've seen this exercise save teams $15,000-$40,000 per month on mid-sized fleets.

Monitoring VPA in Production

Don't fly blind. VPA exposes metrics via Prometheus. Key metrics to dashboard:

# PrometheusRule for VPA alerting
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vpa-alerts
spec:
  groups:
  - name: vpa
    rules:
    # Alert when VPA can't get recommendations (data freshness issue)
    - alert: VPANoRecommendation
      expr: |
        kube_verticalpodautoscaler_status_recommendation_containerrecommendations_target == 0
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: "VPA has no recommendation for {{ $labels.verticalpodautoscaler }}"
    
    # Alert when actual usage is far above VPA recommendation (something changed)
    - alert: VPAUnderProvisioned  
      expr: |
        (
          container_memory_working_set_bytes
          / on(pod, container, namespace) 
          kube_verticalpodautoscaler_status_recommendation_containerrecommendations_target{resource="memory"}
        ) > 1.5
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} memory usage is 50%+ above VPA recommendation"

Also track this Grafana query to measure the ongoing impact of VPA:

# Cost efficiency: ratio of requested vs used CPU across VPA-managed pods
sum(
  kube_pod_container_resource_requests{resource="cpu", namespace="production"}
  * on(pod) group_left() kube_pod_labels{label_app!=""}
) 
/ 
sum(
  rate(container_cpu_usage_seconds_total{namespace="production"}[5m])
  * on(pod) group_left() kube_pod_labels{label_app!=""}
)

A ratio above 3.0 means you're requesting 3x more CPU than you're using. That's your waste multiplier.

Common Pitfalls and How to Avoid Them

Pitfall 1: Deploying VPA Auto mode on Day 1

I see this constantly. Teams install VPA, immediately set everything to Auto, and then get paged at 2 AM because VPA evicted pods during a traffic spike. Start with Off, build confidence, then move to Initial, then Auto.

Pitfall 2: Not setting maxAllowed

Without maxAllowed, VPA can recommend arbitrarily large resource requests. If a batch job briefly spikes to 32 CPU cores due to a bug or unusual workload, VPA might recommend 32 cores as the new baseline. Always bound your recommendations:

maxAllowed:
  cpu: 8        # Never recommend more than 8 cores
  memory: 16Gi  # Never recommend more than 16Gi

Pitfall 3: Using VPA with StatefulSets carelessly

StatefulSets have stable network identities and persistent storage, and restarts are more disruptive. Use Initial mode for StatefulSets in most cases, and ensure you have proper PDBs:

updatePolicy:
  updateMode: "Initial"  # Set at creation, don't auto-restart StatefulSet pods

Pitfall 4: Ignoring the OOM Kill pattern

VPA will increase memory requests when it detects OOM kills. But if your app has a memory leak, VPA will just keep recommending more and more memory. VPA is not a substitute for fixing memory leaks — it will just make them more expensive. Monitor your OOM kill rate separately and treat sustained increases as application bugs, not configuration problems.

Pitfall 5: Short observation windows for seasonal workloads

If your application has weekly patterns (higher load on weekdays), VPA needs at least 2-3 weeks of data. If it has monthly or seasonal patterns, VPA alone won't capture the peak. In these cases, use VPA recommendations as a floor and set your limits higher based on known peak patterns.

Integrating VPA into Your GitOps Workflow

In a proper GitOps setup, you want VPA recommendations to flow back into your manifests, not silently mutate your pods. Here's a pattern I use with Argo CD:

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-service
spec:
  source:
    repoURL: https://github.com/your-org/k8s-configs
    path: apps/web-service
    plugin:
      name: vpa-recommender  # Custom plugin that injects VPA recommendations

The custom plugin approach is advanced, but a simpler path is a scheduled CI/CD job:

#!/bin/bash
# sync-vpa-recommendations.sh - runs in CI weekly

NAMESPACE="production"
BRANCH="vpa-recommendations-$(date +%Y%m%d)"

git checkout -b $BRANCH

for deployment in $(kubectl get deployments -n $NAMESPACE -o name); do
  DEP_NAME=$(echo $deployment | cut -d/ -f2)
  VPA_NAME="${DEP_NAME}-vpa"
  
  # Check if VPA exists for this deployment
  if kubectl get vpa $VPA_NAME -n $NAMESPACE &>/dev/null; then
    TARGET_CPU=$(kubectl get vpa $VPA_NAME -n $NAMESPACE \
      -o jsonpath='{.status.recommendation.containerRecommendations[0].target.cpu}')
    TARGET_MEM=$(kubectl get vpa $VPA_NAME -n $NAMESPACE \
      -o jsonpath='{.status.recommendation.containerRecommendations[0].target.memory}')
    
    # Update the kustomize patch file
    cat > "apps/${DEP_NAME}/vpa-patch.yaml" << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: $DEP_NAME
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          requests:
            cpu: $TARGET_CPU
            memory: $TARGET_MEM
EOF
  fi
done

git add -A
git commit -m "chore: weekly VPA recommendation sync for $NAMESPACE"
git push origin $BRANCH
# Create PR for human review before merging
gh pr create --title "Weekly VPA Recommendations" --body "Auto-generated from VPA. Review before merging."

This gives you the observability of VPA recommendations with the auditability and control of GitOps.

The Bottom Line

VPA is a powerful tool that, used correctly, can systematically eliminate one of the largest sources of cloud waste in Kubernetes environments. The key principles:

1. Start in Off mode and build confidence before enabling automatic updates
2. Always set min/max bounds to prevent runaway recommendations
3. Respect your PDBs and use them as a safety mechanism
4. Don't fight HPA — use them for different dimensions or use HPA on custom metrics
5. Monitor VPA itself — it's infrastructure that needs oversight like everything else
6. Feed recommendations back into Git — your source of truth should reflect reality

The teams I work with that implement VPA correctly typically see 30-60% reduction in CPU requests and 20-40% reduction in memory requests within the first quarter. On a cluster with meaningful scale, that's the kind of saving that makes CFOs send thank-you notes.

Your cluster is almost certainly running pods that are begging for less. VPA helps you listen.

---

Have questions about VPA configuration for your specific workload patterns? The edge cases are where the interesting problems live. The scenarios above cover 80% of production situations, but every fleet has its quirks — batch workloads, GPU containers, and JVM-heavy services each have their own tuning considerations worth a dedicated deep dive.