Fix Jenkins Agent Disconnecting Randomly

The Error: Agent Keeps Disconnecting

Builds fail with:

ERROR: Connection was broken: java.io.IOException:
Unexpected termination of the channel

Or in the agent status page:

Agent went offline because Jenkins could not connect to it.
java.net.SocketTimeoutException: connect timed out

The agent appears online briefly, runs part of a build, then drops. Builds get stuck in the queue or fail mid-step.

Root Cause

Jenkins agent disconnections usually stem from:

1. JVM heap exhaustion on the agent — the agent process runs out of memory and crashes
2. Network timeout between controller and agent — firewalls, NAT, or load balancers drop idle TCP connections
3. SSH keepalive not configured — for SSH-connected agents, idle connections are killed
4. Agent machine running out of disk space — the remoting JAR cannot write temporary files
5. Mismatched Java versions — controller and agent running incompatible JVM versions

Step-by-Step Fix

1. Check agent logs for the root cause

# Agent log location (varies by launch method)
# JNLP agent
cat /var/jenkins/agent/logs/remoting.log

# SSH agent — check from Jenkins UI
# Manage Jenkins > Nodes > <agent> > Log

Look for OutOfMemoryError, SocketTimeoutException, or Connection reset.

2. Increase agent JVM heap

The default -Xmx for the agent remoting process is often too low:

# For JNLP/inbound agents, update the launch command
java -Xmx512m -jar agent.jar \
  -url https://jenkins.example.com \
  -secret <secret> \
  -name <agent-name> \
  -workDir /var/jenkins/agent

For SSH agents, set JVM options in Jenkins UI:

• Go to Manage Jenkins > Nodes > [agent] > Configure
• Under Launch method > Advanced, set JVM Options: -Xmx512m

3. Configure TCP keepalives

Prevent firewalls from dropping idle connections. On the Jenkins controller, go to Manage Jenkins > Configure Global Security and set:

• TCP port for inbound agents: Fixed port (e.g., 50000)

On the agent machine, tune kernel keepalives:

# Keep TCP connections alive through firewalls/NAT
sudo sysctl -w net.ipv4.tcp_keepalive_time=60
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=10
sudo sysctl -w net.ipv4.tcp_keepalive_probes=6

# Make permanent
cat <<EOF | sudo tee -a /etc/sysctl.d/99-keepalive.conf
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
EOF

4. Fix SSH agent keepalives

For SSH-based agents, configure the SSH client on the controller:

# /var/lib/jenkins/.ssh/config (or equivalent)
Host *
    ServerAliveInterval 30
    ServerAliveCountMax 5
    TCPKeepAlive yes

And on the agent's SSH server:

# /etc/ssh/sshd_config
ClientAliveInterval 30
ClientAliveCountMax 5

sudo systemctl reload sshd

5. Check disk space on the agent

df -h /var/jenkins/agent

# Clean up old workspaces
find /var/jenkins/agent/workspace -maxdepth 1 -mtime +7 -exec rm -rf {} +

Jenkins stores build artifacts, workspaces, and remoting cache on the agent. If the disk fills, the agent crashes.

6. Match Java versions

# On the controller
java -version

# On the agent
java -version

Jenkins requires compatible Java versions. If the controller runs Java 17, agents should also run Java 17. Mismatches cause serialization errors and disconnects.

7. Use the availability strategy

In the agent configuration (Manage Jenkins > Nodes > [agent] > Configure):

• Set Availability to "Keep this agent online as much as possible"
• Set # of Retries for reconnection to 5+
• Set Seconds Between Retries to 15

Prevention Tips

• Monitor agent health with the Jenkins Monitoring plugin — track heap usage, thread count, and response time.
• Use ephemeral agents (Kubernetes pod agents, EC2 plugin, Docker agents) to avoid long-lived connection issues entirely.
• Set workspace cleanup policies — use the Workspace Cleanup plugin to delete workspaces after builds.
• Pin Java versions in your agent provisioning automation so controller/agent versions always match.
• Alert on agent offline events — use the Jenkins Notification plugin or a Prometheus exporter to catch disconnections before users report stuck builds.