Elasticsearch Cluster Sizing for Production: Nodes, Shards, and Memory

Most Elasticsearch performance problems are not bugs — they are sizing decisions made at 2am during an incident that were never revisited. Over-sharding, under-memory hot nodes, and single-role clusters are the three patterns that kill most production deployments. This guide gives you the formulas and decisions to get sizing right before things go wrong.

Node Roles: Stop Running Everything on One Box

Elasticsearch supports multiple node roles, and separating them is the most impactful production decision you can make. By default, a new node takes every role — that means your master election is competing with your indexing load, and your search queries are landing on nodes that are also merging segments.

Role	Responsibility	Recommended Hardware
master	Cluster state, shard allocation decisions	4-8 vCPU, 16 GB RAM, SSD (small)
data_hot	Active writes, recent data queries	16-32 vCPU, 64-128 GB RAM, NVMe SSD
data_warm	Recent but infrequently queried data	8-16 vCPU, 32-64 GB RAM, SSD or dense HDD
data_cold	Archival data, rare queries	4-8 vCPU, 16-32 GB RAM, HDD or object storage
ingest	Pipeline processing (grok, enrich)	8-16 vCPU, 32 GB RAM, moderate disk
coordinating	Query fan-out, result merging	8-16 vCPU, 32-64 GB RAM, fast network

For a production logging cluster ingesting 50 GB/day, a minimum viable separation looks like:

• 3 dedicated master nodes (never store data)
• 3-5 hot data nodes
• 2 coordinating nodes in front of Kibana

# elasticsearch.yml for a dedicated master node
node.roles: [master]
node.name: master-1
cluster.name: prod-logging

# Disable data-related roles explicitly
# (setting node.roles overrides defaults)

# elasticsearch.yml for a hot data node
node.roles: [data_hot, data_content]
node.name: hot-1
cluster.name: prod-logging

path.data: /mnt/nvme/elasticsearch
path.logs: /var/log/elasticsearch

Heap Sizing: The Most Common Mistake

Elasticsearch runs on JVM. Heap that is too small causes constant GC pauses. Heap that is too large causes GC pause times that exceed 30 seconds, which triggers master timeouts and can split your cluster.

The rule: heap = 50% of RAM, capped at 31 GB.

The 31 GB cap is not arbitrary. Above 32 GB, the JVM switches from compressed ordinary object pointers (OOPS) to full 64-bit pointers, which increases object sizes and makes GC worse, not better. At 31 GB you get the largest possible heap with compressed OOPS still enabled.

# /etc/elasticsearch/jvm.options.d/heap.options
-Xms31g
-Xmx31g

Always set -Xms and -Xmx to the same value. If they differ, the JVM will resize the heap at runtime, causing GC events.

What does the remaining 50% of RAM do? It goes to the OS file system cache, which Elasticsearch uses heavily for Lucene segment reads. A node with 64 GB RAM should have 31 GB heap and 33 GB available to the file cache.

# Verify heap is applied correctly
curl -s localhost:9200/_nodes/stats/jvm | jq '.nodes[].jvm.mem.heap_max_in_bytes' | awk '{print $1/1024/1024/1024 " GB"}'

Shard Sizing: The Over-Sharding Epidemic

The most common Elasticsearch antipattern is daily index rotation with the default 1 primary shard, producing hundreds of tiny shards per week. Each shard is a Lucene index with its own file handles, memory overhead, and merge activity. Elasticsearch's own recommendation: aim for 10-50 GB per shard, with 20-40 GB being the sweet spot for search-heavy workloads.

Shard count formula:

primary_shards = ceil(daily_volume_GB / target_shard_size_GB)
total_shards   = primary_shards * (1 + replica_count)

For a cluster ingesting 100 GB/day with a target shard size of 25 GB:

primary_shards = ceil(100 / 25) = 4
total_shards   = 4 * (1 + 1)   = 8 per daily index

Over 30 days that is 240 shards — manageable. Compare that to daily indices with 10 primary shards (3,000 shards over 90 days), which is a cluster state nightmare.

# Check current shard sizes across all indices
curl -s 'localhost:9200/_cat/shards?v&h=index,shard,prirep,store&s=store:desc' | head -30

Signs you are over-sharded:

• _cat/health shows unassigned_shards > 0 frequently
• Master node CPU spikes during index creation
• Cluster state update latency over 500ms

# Check cluster state size (should be under 100MB for healthy cluster)
curl -s 'localhost:9200/_cluster/stats' | jq '.nodes.count, .indices.shards.total'

Index Lifecycle Management

ILM is how you automate the hot-warm-cold-delete pipeline. Without it, disk fills up, hot nodes carry stale data, and someone deletes the wrong index manually at 3am.

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "25gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": { "data": "cold" }
          },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The shrink action in warm phase is critical. It collapses a 4-shard hot index into 1 shard once it stops receiving writes, drastically reducing shard count over time without losing data.

Capacity Planning: The Sizing Formula

For a logging cluster, here is the end-to-end capacity formula:

Daily ingestion:       100 GB/day (raw logs)
Compression ratio:     ~3:1 (Elasticsearch with LZ4)
Stored per day:        ~33 GB
Retention:             90 days
Replication factor:    1 replica (2x total storage)
Storage needed:        33 GB * 90 * 2 = ~6 TB usable
Safety buffer (20%):   ~7.2 TB total provisioned storage

Shard count:           4 primary per day * 90 days = 360 primary shards
With replicas:         720 total shards
Per hot data node (3): 240 shards (acceptable — keep under 500/node)

Per-node hardware recommendation for this scenario:

Node Type	Count	vCPU	RAM	Disk
Master	3	4	16 GB	100 GB SSD
Hot data	3	16	64 GB	1 TB NVMe
Warm data	2	8	32 GB	2 TB SSD
Coordinating	2	8	32 GB	200 GB SSD

Key Monitoring Metrics

Once your cluster is running, monitor these:

# Cluster health overview
curl -s 'localhost:9200/_cluster/health?pretty'

# JVM GC pressure (alert if old_gc_time > 300ms in 30s window)
curl -s 'localhost:9200/_nodes/stats/jvm' | \
  jq '.nodes[] | {name: .name, old_gc_ms: .jvm.gc.collectors.old.collection_time_in_millis}'

# Indexing rate (documents/second)
curl -s 'localhost:9200/_nodes/stats/indices' | \
  jq '.nodes[] | {name: .name, index_rate: .indices.indexing.index_total}'

# Search latency
curl -s 'localhost:9200/_nodes/stats/indices' | \
  jq '.nodes[] | {name: .name, query_avg_ms: (.indices.search.query_time_in_millis / .indices.search.query_total)}'

Alert thresholds to configure in Grafana or Elastic Alerting:

Metric	Warning	Critical
Cluster status	yellow	red
JVM heap used %	> 75%	> 85%
Old GC duration	> 300ms	> 1s
Unassigned shards	> 0	> 5
Disk used %	> 70%	> 85%
CPU per data node	> 70%	> 90%

Thread Pool Tuning

For write-heavy clusters, the indexing thread pool is often the first bottleneck:

# Check thread pool rejections (any rejections = problem)
GET _nodes/stats/thread_pool/write,search

If you see rejected count increasing in the write thread pool:

# elasticsearch.yml
thread_pool.write.queue_size: 1000
thread_pool.write.size: 16  # Matches vCPU count

If search thread pool shows rejections, your coordinating layer is undersized or your queries are too expensive — adding nodes is faster than tuning queries in an incident.

Summary

Proper Elasticsearch sizing comes down to four decisions made correctly upfront:

1. Separate node roles — never let masters hold data in production.
2. Heap at 50% of RAM, hard cap at 31 GB, Xms equals Xmx.
3. Target 20-40 GB per shard — over-sharding is the silent cluster killer.
4. Use ILM with shrink on the warm phase to control shard sprawl over time.

Get these four right and your cluster will survive a 10x spike in log volume without a 3am incident. Get them wrong and you will be fighting ghost problems — yellow cluster status, random GC pauses, and unassigned shards — forever.