Fix Linux 'Too Many Open Files' System-Wide

The Error

Applications start failing with errors like:

Too many open files

java.io.IOException: Too many open files

socket: too many open files

The system or a specific process has exhausted its file descriptor limit. Since Linux treats almost everything as a file (sockets, pipes, actual files), high-traffic services hit this limit frequently.

Root Cause

Linux enforces file descriptor limits at three levels:

1. System-wide limit (fs.file-max) - The absolute maximum number of file descriptors the kernel will allocate.
2. User-level limit (ulimit / /etc/security/limits.conf) - The maximum per-user, enforced by PAM.
3. Systemd service limit (LimitNOFILE) - Per-service override when running under systemd.

The default soft limit is often 1024, which is far too low for database servers, web servers, or any application handling many concurrent connections.

Step-by-Step Fix

1. Check current limits

# System-wide maximum
cat /proc/sys/fs/file-max

# Current system-wide usage
cat /proc/sys/fs/file-nr
# Output: <allocated> <free> <max>

# Current user limits
ulimit -n        # soft limit
ulimit -Hn       # hard limit

# For a specific running process
cat /proc/<PID>/limits | grep "open files"

2. Increase the system-wide limit

# Temporary (survives until reboot)
sudo sysctl -w fs.file-max=2097152

# Permanent
echo "fs.file-max = 2097152" | sudo tee -a /etc/sysctl.d/99-file-max.conf
sudo sysctl --system

3. Increase user-level limits

Edit /etc/security/limits.conf:

sudo tee -a /etc/security/limits.conf <<EOF
# Increase file descriptor limits
*         soft    nofile    65535
*         hard    nofile    65535
root      soft    nofile    65535
root      hard    nofile    65535
EOF

Ensure PAM is configured to apply these limits. Check that this line exists in /etc/pam.d/common-session:

session required pam_limits.so

The user must log out and back in for the new limits to take effect.

4. Increase limits for systemd services

For services managed by systemd, user limits from limits.conf may not apply. Override at the service level:

sudo systemctl edit nginx

Add:

[Service]
LimitNOFILE=65535

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Verify:

cat /proc/$(pgrep -f nginx | head -1)/limits | grep "open files"

5. Immediate relief for a running process (no restart)

On Linux 4.1+ with kernel support, you can use prlimit to change limits on a running process:

sudo prlimit --pid <PID> --nofile=65535:65535

6. Find the file descriptor leak (if limits are reasonable)

If the limit is already high but still being hit, you may have a file descriptor leak:

# Count open FDs for a process
ls /proc/<PID>/fd | wc -l

# See what files are open
ls -la /proc/<PID>/fd | tail -20

# Or use lsof
lsof -p <PID> | wc -l
lsof -p <PID> | awk '{print $5}' | sort | uniq -c | sort -rn | head

Prevention Tips

• Set high limits in your base image or provisioning scripts. A soft limit of 65535 is a safe default for most production servers.
• Monitor file descriptor usage. Export /proc/sys/fs/file-nr to your monitoring system and alert at 80% utilization.
• Configure application-level limits. Many applications (Nginx, MySQL, Elasticsearch) have their own max_connections or worker_connections settings that should align with the OS limit.
• Audit for FD leaks. Periodically check long-running processes for steadily increasing FD counts, which indicate a leak in connection or file handling code.