Skip to main content
DevOps Insightsguides & answers

Linux · troubleshooting

Too many open files

A process asked for one more file descriptor than it is allowed to have. Sockets cost descriptors too, so this usually arrives disguised as a network fault. Two different errors share almost the same wording, and which one you have decides whether you edit a service unit or a sysctl.

Reproduced on Python 3.9.6, . Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • The PID of the process that logged the error, and a shell on the same host, or inside the same container if the process runs in one.
  • Root, or the same user as the process, to list /proc/<pid>/fd, which is guarded by a ptrace access check. /proc/<pid>/limits is readable by every user.
  • To change a service's limit: permission to add a systemd drop-in for its unit, and a window to restart it, because systemd applies the limit when it starts the process.
  • For the system-wide limit or the inotify limits: root, to change them with sysctl and persist them in /etc/sysctl.d/.
  • A rough figure for how many connections and files the workload should hold at peak, so you can tell a leak from real demand.

Run this first

step 1 of 4
cat /proc/<pid>/limits | grep -i 'open files'

Look for Soft and hard values for Max open files

Establishes the limit the process actually has, as opposed to the one your shell has. Everything else is measured against this number.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Which wording does the error use?

The two are different errors. "in system" at the end changes the fix completely.

Cause space

5 of 5 still possible

  • The process is leaking descriptorsCommon
  • The limit is genuinely too low for the workloadCommon
  • systemd is setting a limit your shell never showed youCommon
  • The whole system is out, not the processRare
  • inotify instances ran out, not file descriptorsOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    cat /proc/<pid>/limits | grep -i 'open files'

    Establishes the limit the process actually has, as opposed to the one your shell has. Everything else is measured against this number.

    Look for Soft and hard values for Max open files

  2. Step 2

    ls /proc/<pid>/fd | wc -l

    The current usage. Compared against the soft limit, this separates "at the ceiling" from "nowhere near it, so the error came from somewhere else".

    Look for A number at or just below the soft limit

  3. Step 3

    ls -l /proc/<pid>/fd | awk 'NR>1{print $NF}' | sed -E 's/:\[[0-9]+\]$//' | sort | uniq -c | sort -rn | head -20

    Names what the descriptors point at. A leak shows as one target repeated hundreds or thousands of times.

    Look for A dominant entry — the same socket, pipe or path over and over

  4. Step 4

    cat /proc/sys/fs/file-nr

    Distinguishes the per-process limit from the system-wide one, which is the fork in the road for the fix.

    Look for Allocated versus maximum. Near the maximum means ENFILE rather than EMFILE

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The process is leaking descriptors

Common

Descriptors are opened and never closed — a connection pool that discards clients without closing them, a library that opens a file per request, a retry loop that reconnects on failure and abandons the old socket. The signature is that the count only ever climbs. Restarting clears it and the clock starts again, which is why this often presents as a service that needs restarting every few days.

Confirm

for i in 1 2 3; do ls /proc/<pid>/fd | wc -l; sleep 10; done

A count that rises across the three samples and never comes back down while the workload is steady

Fix

  • Group the open descriptors by what they point at — that normally names the leak in one line, because the leaked kind dominates the list.
  • Fix the close path in the application. A higher limit only moves the failure further out.
  • If the leak is in a dependency you cannot change, a scheduled restart is a holding measure, not a resolution — and worth recording as such so it is not mistaken for a fix later.
ls -l /proc/<pid>/fd | awk 'NR>1{print $NF}' | sed -E 's/:\[[0-9]+\]$//' | sort | uniq -c | sort -rn | head -20

Counts descriptors by target, with socket and pipe inode numbers stripped so they group. Thousands of sockets or pipes, or the same file over and over, is the leak.

The limit is genuinely too low for the workload

Common

Nothing is leaking; the process simply needs more than it was given. Reverse proxies, databases and anything holding many concurrent connections exceed conservative defaults easily, because each live connection is at least one descriptor and often two when traffic is proxied onward.

Confirm

cat /proc/<pid>/limits | grep -i 'open files'

The soft limit, compared against the descriptor count the process actually reaches under normal load

Fix

  • Raise the soft limit for that process to something above its real working set, with headroom.
  • The hard limit is the ceiling the soft limit can be raised to without privilege. /proc/sys/fs/nr_open caps the hard limit itself and defaults to 1048576, enforced for privileged and unprivileged processes alike.
  • Set it where the process is actually started — a unit file, a container runtime option, a supervisor config — not in an interactive shell.

systemd is setting a limit your shell never showed you

Common

A service started by systemd inherits LimitNOFILE= from its unit, not whatever ulimit -n reports in your session. The two are unrelated, so a limit that looks generous interactively can be modest for the service. This is the usual reason a raised limit appears to have had no effect.

Confirm

systemctl show <unit> -p LimitNOFILE -p LimitNOFILESoft

The values systemd will actually apply. Compare them with /proc/<pid>/limits for the running process

Fix

  • Set LimitNOFILE= in the unit (a single value, or soft:hard), but raise the soft limit above 1024 only for programs that do not use select(2). systemd's documentation recommends that applications raise their own soft limit up to the hard limit instead.
  • Use a drop-in rather than editing a packaged unit, so a package upgrade does not quietly revert it.
  • Reload the unit definitions and restart the service — limits are applied when the process is started, so a running process keeps the old ones.
systemctl daemon-reload && systemctl restart <unit> && cat /proc/$(systemctl show -p MainPID --value <unit>)/limits | grep -i 'open files'

Reads the limit back off the running process, which is the only confirmation that counts.

The whole system is out, not the process

Rare

/proc/sys/fs/file-max caps open files across every process. Calls that hit it fail with ENFILE and report "Too many open files in system". Raising a single service's limit does nothing here, and on a machine in this state unrelated processes start failing at the same time — which is the clearest tell.

Confirm

cat /proc/sys/fs/file-nr

Allocated handles close to the third number, which is file-max. Check the kernel log for file-max messages too

Fix

  • Raise fs.file-max via sysctl, persisting it so it survives a reboot.
  • Then find what consumed them. A system-wide exhaustion is usually one process leaking badly rather than genuine demand, so treat the raise as breathing room and keep looking.
sudo sysctl -w fs.file-max=<n> && cat /proc/sys/fs/file-nr

Add it to a file under /etc/sysctl.d/ to make it persistent. The running value alone is lost on reboot.

inotify instances ran out, not file descriptors

Occasional

inotify_init fails with EMFILE, the same Too many open files, when a user reaches fs.inotify.max_user_instances, even though the process is nowhere near its descriptor limit. File watchers in editors, build tools and on Kubernetes nodes are common consumers.

Confirm

cat /proc/sys/fs/inotify/max_user_instances
sudo find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

The second number, the inotify instances open right now across every user on the host, at or near the first. The limit is per user, so a total near it is a lead, not proof

Fix

  • Raise fs.inotify.max_user_instances with sysctl, and persist it in /etc/sysctl.d/.
  • Then find what is creating so many watchers: raising the limit hides a leak if there is one.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Every file, network connection and pipe a program has open takes up one numbered slot, and the system caps how many slots each program may hold. When a program asks for one more than its cap, the request fails with "Too many open files". Network connections use the same slots, so this often looks like a network fault rather than a file problem.

Hitting the limit on purpose

Reproduced with Python 3.9.6 on macOS 27.0.1, not Linux: a zsh subshell set a soft limit of 32 and a hard limit of 64 with ulimit, then the script filled its descriptors with /dev/null, tried one more socket, accepted a connection already waiting on 127.0.0.1:18443 (loopback), and raised its own limits. The /proc and systemd commands elsewhere on this page are Linux-only; they come from the man pages and were not run here.

Command

(ulimit -Hn 64; ulimit -Sn 32; python3 - <<'EOF'
import resource, socket
print("limit:", resource.getrlimit(resource.RLIMIT_NOFILE))
server = socket.create_server(("127.0.0.1", 18443))
client = socket.socket()
files = []
try:
    while True:
        files.append(open("/dev/null"))
except OSError as e:
    print("open():", e)
try:
    socket.socket()
except OSError as e:
    print("socket():", e)
client.connect(("127.0.0.1", 18443))
print("client connected")
try:
    server.accept()
except OSError as e:
    print("accept():", e)
resource.setrlimit(resource.RLIMIT_NOFILE, (64, 64))
print("raised to:", resource.getrlimit(resource.RLIMIT_NOFILE))
try:
    resource.setrlimit(resource.RLIMIT_NOFILE, (65, 65))
except ValueError as e:
    print("above hard:", e)
EOF
)

Output

limit: (32, 64)
open(): [Errno 24] Too many open files: '/dev/null'
socket(): [Errno 24] Too many open files
client connected
accept(): [Errno 24] Too many open files
raised to: (64, 64)
above hard: not allowed to raise maximum limit

How to read it

  • [Errno 24] Too many open files is EMFILE, the per-process limit. The system-wide error, ENFILE, reads Too many open files in system and needs a different fix, so check which one your log says.
  • The socket and the accept failed with the same message as the file, because sockets use descriptors too. The client's connect still succeeded, because the kernel queues incoming connections for a listening socket; only the process could not take one off the queue. From outside, that looks like a hung server.
  • ulimit changed only the subshell and the Python it started; the terminal around it kept its own limit. That is why ulimit -n in your shell says nothing about a service started elsewhere. On Linux, read /proc/<pid>/limits for the process itself.
  • The process raised its soft limit to the hard limit without any privilege, but not allowed to raise maximum limit stopped it going higher. getrlimit(2) describes the same rule on Linux.

Words used on this page

File descriptor
The small number a process uses for something it has open: a file, a socket, a pipe, or a special handle such as an inotify instance. Each one counts against the limit.
RLIMIT_NOFILE
The per-process limit on descriptors. getrlimit(2) defines it as one more than the highest descriptor number the process may open. On Linux, /proc/<pid>/limits shows it for a running process.
Soft and hard limit
The soft limit is the one the kernel enforces. A process may raise its own soft limit up to the hard limit; raising the hard limit needs privilege (CAP_SYS_RESOURCE on Linux).
EMFILE
The error for reaching the per-process limit. Its message is Too many open files.
ENFILE
The error for reaching the system-wide limit, fs.file-max on Linux. Its message is Too many open files in system.
LimitNOFILE=
The systemd unit setting for a service's limit, either one value or soft:hard. A unit that does not set it gets DefaultLimitNOFILE=, which defaults to 1024:524288.
Descriptor leak
Descriptors that are opened and never closed, so the count only climbs until it reaches the limit.
inotify instance
A file-watching handle. It is a descriptor too, and a separate per-user limit, fs.inotify.max_user_instances, also produces Too many open files.
What is actually happening

Every open file, pipe, directory and socket a process holds costs one file descriptor, and the kernel caps how many it may hold at once. That cap is RLIMIT_NOFILE, which getrlimit(2) defines as a value one greater than the highest descriptor number the process can open. Attempts to go past it — open(2), pipe(2), dup(2) and, importantly, accepting a connection — fail with EMFILE, whose message is "Too many open files".

The single most useful thing to notice is that there are two errors here, not one. EMFILE is the per-process limit. ENFILE is the system-wide one, reported as "Too many open files in system", and it comes from /proc/sys/fs/file-max — a limit on open files across every process on the box. The wording is close enough that people routinely raise the wrong ceiling and watch the error come straight back.

Because sockets are descriptors, a server that has run out will usually keep running while failing to accept anything new. The log line is often accept4: too many open files rather than anything about a file, and from outside it looks like the process has hung or the network has broken. It has not: it is alive, and refusing.

There is also a difference between a limit that is too low and a limit that is being leaked into. A busy proxy legitimately holding tens of thousands of connections needs a higher number. A process whose descriptor count only ever climbs is not going to be fixed by a higher number — it will take longer to hit, which looks like a fix for a few days and is not one.

One more trap, specific to anything run as a service: ulimit -n in your shell describes your shell. A unit started by systemd takes its limits from LimitNOFILE= in the unit, or, if the unit does not set it, from DefaultLimitNOFILE= in systemd-system.conf, which may be set to a value, a soft:hard pair, or infinity. Checking the limit in a terminal and concluding the service has the same one is a reliable way to spend an afternoon.

How to tell this is your problem
WhereWhat you see
Application logaccept4: too many open files, socket: too many open files, or an open failure on a path that definitely exists
cat /proc/<pid>/limitsA "Max open files" row with a soft and a hard value
ls /proc/<pid>/fd | wc -lA count sitting at, or just under, the soft limit
cat /proc/sys/fs/file-nrThree numbers — allocated, free, and the system maximum. Allocated near the maximum means the system-wide form

How to know it is actually fixed

  • The descriptor count under normal load sits comfortably below the soft limit, with headroom rather than just inside it.
  • cat /proc/<pid>/limits on the running process shows the new value — not the config file, the process.
  • The error stops appearing in the log under the load that produced it, rather than merely not having reappeared yet.
  • Where a leak was suspected, the count is flat over a sustained period instead of climbing more slowly.

Stopping it happening again

  • Alert on descriptor usage as a fraction of the limit, not on the error. The error is the end of the story and the ratio climbing is the beginning of it.
  • Set limits where the process is started — unit file, runtime option, supervisor config — and treat an interactive ulimit as a diagnostic, never as a fix.
  • Watch for a flat count rather than a rising one when load is steady. A limit raised on a leaking process buys time and hides the cause.
  • Remember the other per-user ceilings that look similar when a file-watching process misbehaves: proc_sys_fs_inotify(5), linked from inotify(7), documents separate per-user limits on instances and watches in /proc/sys/fs/inotify/, which are counted independently of RLIMIT_NOFILE.

Security considerations

  • Descriptor exhaustion is a denial of service: anyone who can open connections to the service can push it to its limit, and then nobody else gets in. Pair a sensible limit with per-client connection limits in the server or proxy rather than relying on a bigger number.
  • Do not raise a leaking process's limit as far as it will go. It only delays the failure, and a large enough leak then exhausts the system-wide fs.file-max, which makes unrelated processes fail with Too many open files in system.
  • ls -l /proc/<pid>/fd lists the path of every file the process has open, including key, certificate and credential files. Redact paths before pasting the output into a ticket or chat.
  • fs.file-max and the fs.inotify limits apply to the whole host and every user on it. Change them in a file under /etc/sysctl.d/ with a comment saying why, so the change is reviewed and not lost on reboot.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next