Nexus Repository 503 Service Unavailable After JVM Heap Exhaustion: Diagnosis and Fix

If you've ever stared at a 503 error from Nexus Repository Manager right before a critical deployment, you know the panic. Your CI/CD pipeline is blocked, developers are frustrated, and the clock is ticking. Nine times out of ten, the culprit is JVM heap exhaustion — and the good news is it's completely fixable once you know what you're looking at.

This guide walks you through diagnosing the root cause and applying a permanent fix, not just restarting the service and hoping for the best.

---

Why Nexus Throws 503 After Heap Exhaustion

Nexus Repository Manager runs on Java (specifically on top of Apache Karaf and Jetty). When the JVM runs out of heap space, Nexus can't allocate new objects, which causes it to either crash outright or enter a degraded state where it stops accepting HTTP connections — hence the 503.

The sequence typically looks like this:

1. Heap fills up (often triggered by large artifact uploads, search indexing, or background cleanup tasks)
2. JVM starts aggressive garbage collection (GC) trying to free memory
3. If GC can't reclaim enough space, OutOfMemoryError is thrown
4. Jetty's thread pool becomes exhausted or the application context dies
5. Nexus returns 503 or stops responding entirely

The tricky part? The service might appear to be running (the process is alive, the port is listening) but it's functionally dead. A simple systemctl status nexus won't tell you the whole story.

---

Step 1: Confirm It's Actually Heap Exhaustion

Don't assume — verify. Check the logs first.

# Nexus log location (default installation)
tail -n 200 /opt/nexus/sonatype-work/nexus3/log/nexus.log

# Also check the JVM log
tail -n 200 /opt/nexus/sonatype-work/nexus3/log/jvm.log

You're looking for these specific error signatures:

java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC overhead limit exceeded

The GC overhead limit exceeded variant is actually more insidious — it means the JVM is spending more than 98% of its time doing garbage collection and recovering less than 2% of the heap. Nexus is still technically "running" but it's completely unusable.

You can also check for heap dump files that were automatically generated:

find /opt/nexus/sonatype-work -name "*.hprof" -ls

If you see .hprof files, that's a smoking gun confirming OOM events.

Check Current JVM Configuration

# Find the Nexus PID
ps aux | grep nexus

# Check current heap settings for the running process
cat /proc/$(pgrep -f nexus)/cmdline | tr '\0' '\n' | grep -E "Xmx|Xms"

Or look at the configuration file directly:

cat /opt/nexus/nexus-3.x.x-xx/bin/nexus.vmoptions

If you see something like -Xmx1200m on a server with 16GB RAM, you've found your problem.

---

Step 2: Understand What Nexus Actually Needs

This is where a lot of people get it wrong. Sonatype's official recommendation is not just "give it more RAM" — it's give it the right amount based on your usage.

Here's a practical sizing guide:

Usage Level	Concurrent Users	Recommended Heap	Total Server RAM
Small	< 20	2-4 GB	8 GB
Medium	20-50	4-8 GB	16 GB
Large	50+	8-12 GB	32 GB

Critical rule: Never allocate more than 50-60% of total system RAM to the JVM heap. The rest is needed for the OS, direct memory buffers, and Nexus's own off-heap storage. Setting -Xmx28g on a 32GB server will cause a different kind of pain.

---

Step 3: Apply the Fix

Edit the JVM Options File

The nexus.vmoptions file is your primary configuration target:

# Backup first — always
cp /opt/nexus/nexus-3.x.x-xx/bin/nexus.vmoptions \
   /opt/nexus/nexus-3.x.x-xx/bin/nexus.vmoptions.bak

# Edit the file
nano /opt/nexus/nexus-3.x.x-xx/bin/nexus.vmoptions

Your updated file should look something like this for a medium deployment:

-Xms4096m
-Xmx4096m
-XX:MaxDirectMemorySize=2048m
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+LogVMOutput
-XX:LogFile=../sonatype-work/nexus3/log/jvm.log
-XX:-OmitStackTraceInFastThrow
-Djava.net.preferIPv4Stack=true
-Dkaraf.home=.
-Dkaraf.base=.
-Dkaraf.etc=etc/karaf
-Djava.util.logging.config.file=etc/karaf/java.util.logging.properties
-Dkaraf.data=../sonatype-work/nexus3
-Dkaraf.log=../sonatype-work/nexus3/log
-Djava.io.tmpdir=../sonatype-work/nexus3/tmp
-Dkaraf.startLocalConsole=false
-Djdk.tls.ephemeralDHKeySize=2048

Key things to notice:

• Set -Xms and -Xmx to the same value. This prevents heap resizing overhead and makes GC behavior more predictable.
• -XX:MaxDirectMemorySize controls off-heap memory for things like file I/O buffers. Don't neglect this.

Restart Nexus

systemctl restart nexus

# Watch the startup logs in real-time
journalctl -u nexus -f

# Or tail the Nexus log directly
tail -f /opt/nexus/sonatype-work/nexus3/log/nexus.log

Nexus typically takes 2-3 minutes to fully start. You'll see this line when it's ready:

Started Sonatype Nexus OSS x.x.x

---

Step 4: Prevent It From Happening Again

Fixing the heap size is the immediate solution. But you also need to address why the heap got exhausted in the first place.

Enable GC Logging for Visibility

Add these flags to nexus.vmoptions to get detailed GC information:

-Xlog:gc*:file=../sonatype-work/nexus3/log/gc.log:time,uptime:filecount=5,filesize=20m

This rotates GC logs automatically so they don't fill your disk.

Configure Automatic Heap Dumps on OOM

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=../sonatype-work/nexus3/log/

The heap dump lets you analyze what was consuming memory if it happens again. Tools like Eclipse MAT or VisualVM can open these files.

Set Up Monitoring Alerts

Don't wait for a 503 to find out you have a memory problem. If you're using Prometheus, the Nexus metrics endpoint is available at:

http://your-nexus-host:8081/service/metrics/prometheus

Add this alert rule to catch problems before they become outages:

# prometheus-nexus-alerts.yml
groups:
  - name: nexus_jvm
    rules:
      - alert: NexusJVMHeapHigh
        expr: |
          (jvm_memory_used_bytes{area="heap"} / 
           jvm_memory_max_bytes{area="heap"}) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Nexus JVM heap usage above 85%"
          description: "Heap is {{ $value | humanizePercentage }} full. Consider increasing -Xmx."

      - alert: NexusJVMHeapCritical
        expr: |
          (jvm_memory_used_bytes{area="heap"} / 
           jvm_memory_max_bytes{area="heap"}) > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Nexus JVM heap critically high - OOM imminent"

Schedule Regular Cleanup Tasks

Accumulated blob store and database bloat are common culprits for heap pressure. In Nexus, go to Administration → System → Tasks and verify these scheduled tasks are configured:

• Admin - Compact blob store: Runs weekly, reclaims space from deleted components
• Admin - Rebuild repository search: Can cause heap spikes if it runs too frequently
• Admin - Delete unused components: Helps keep the database lean

For large installations, consider running compact blob store during off-peak hours:

# You can trigger tasks via the Nexus REST API
curl -u admin:password \
  -X POST \
  "http://your-nexus:8081/service/rest/v1/tasks/{taskId}/run" \
  -H "accept: application/json"

---

Bonus: Docker Deployment Considerations

If you're running Nexus in Docker, heap configuration works a bit differently. Don't rely on the INSTALL4J_ADD_VM_PARAMS environment variable alone — it can be overridden.

The proper approach is to mount a custom nexus.vmoptions file:

# docker-compose.yml
version: '3.8'
services:
  nexus:
    image: sonatype/nexus3:latest
    container_name: nexus
    volumes:
      - nexus-data:/nexus-data
      - ./nexus.vmoptions:/opt/sonatype/nexus/bin/nexus.vmoptions:ro
    ports:
      - "8081:8081"
    environment:
      - INSTALL4J_ADD_VM_PARAMS=-Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=2048m
    deploy:
      resources:
        limits:
          memory: 8g  # Always set container memory limits

volumes:
  nexus-data:

Make sure your container memory limit is higher than your JVM heap + direct memory combined, or you'll get OOM kills at the OS level instead of the JVM level — which is even harder to diagnose.

---

Quick Diagnosis Checklist

When you hit a 503, run through this in order:

# 1. Is the process running?
systemctl status nexus

# 2. Is it OOM?
grep -E "OutOfMemoryError|GC overhead" /opt/nexus/sonatype-work/nexus3/log/nexus.log | tail -20

# 3. What are the current heap settings?
cat /opt/nexus/nexus-*/bin/nexus.vmoptions | grep -E "Xmx|Xms"

# 4. How much RAM does the server have?
free -h

# 5. Any heap dumps generated?
find /opt/nexus -name "*.hprof" 2>/dev/null

---

The Bottom Line

JVM heap exhaustion in Nexus is annoying but predictable. The pattern is always the same: undersized heap, no monitoring, surprise outage. Break that pattern by right-sizing your heap (with -Xms and -Xmx set equal), adding GC logging, and setting up alerts before you're in crisis mode.

The fix takes about ten minutes. The prevention setup takes maybe an hour. Either way, it's better than explaining to your team why the artifact repository went down during a release window.