How to track down a file descriptor leak?
I was troubleshooting an application that crashed with "Too many open files" error. The cause is clear - the application is leaking file descriptors. But how do I find out where the leak is happening?
Number of open file descriptors
The default limitation of open file descriptors is set to 1024 on most Linux systems. It is set as a resource limit [1] and can be set from the terminal using the ulimit command:
1ulimit -n 2048/proc/<PID>/fd
/proc/pid/fd is a subdirectory containing symbolic links representing the file descriptors opened by the process. You can use it to see which files are open and how many file descriptors are currently in use.
As you know, "Everything is a file", so this includes not only regular files, but also sockets, pipes, and other types of file descriptors.
Some file descriptors do not have a corresponding inode (e.g. file descriptors produced by epoll_create, eventfd, signalfd etc.) and will have a symbolic link with contents of the form:
anon_inode:file-type
For example anon_inode:[eventpoll] for a file descriptor created by epoll and anon_inode:[eventfd] for a file descriptor created by eventfd.
Find out what is leaking
So, by looking at the contents of /proc/<pid>/fd you can see which file descriptors are currently open and what they are. If there is an increasing number of file descriptors of a specific type, then you probably have a leak somewhere.
In my case, I had a lot of file descriptors of the type anon_inode:[eventfd], which were steadily increased every time the application took a certain code path.
strace is your friend
strace [2] is a fantastic tool. It lets you trace system calls and signals of a process and are nearly always available on any Linux system.
As we know that we are leaking eventfd file descriptors, we can use strace to trace only calls to eventfd and eventfd2 system calls, which are used to create event file descriptors. We can use the -k option to print the stack trace of each system call, which will help us to find out where in the code the leak is happening.
An example
Consider the following code snippet:
1void bar() {
2 int i = eventfd(0,0);
3}
4
5void foo() {
6 bar();
7}
8
9void main() {
10 foo();
11 sleep(1000);
12}When we run this code, it will create an event file descriptor every time it is executed, and then sleep for a while - just to give us some time to inspect the content of /proc/<pid>/fd.
Start the application with strace:
1$ strace --stack-trace -e eventfd,eventfd2 ./main
2eventfd2(0, 0) = 3
3 > /usr/lib/libc.so.6(eventfd+0xb) [0x11d50b]
4 > /home/marcus/tmp/example/main(bar+0x17) [0x1150]
5 > /home/marcus/tmp/example/main(foo+0x9) [0x115f]
6 > /home/marcus/tmp/example/main(main+0x9) [0x116b]
7 > /usr/lib/libc.so.6() [0x27741]
8 > /usr/lib/libc.so.6(__libc_start_main+0x89) [0x27879]
9 > /home/marcus/tmp/example/main(_start+0x25) [0x1065]As we can see, we get a nice stack trace to the point where the file descriptor is created, and we also see that the file descriptor has the number 3.
If we look at the content of /proc/<pid>/fd we can see that file descriptor 3 is indeed an eventfd:
1$ ls -al /proc/514724/fd
2total 0
3dr-x------ 2 marcus marcus 4 21 maj 16.16 .
4dr-xr-xr-x 9 marcus marcus 0 21 maj 16.16 ..
5lrwx------ 1 marcus marcus 64 21 maj 16.16 0 -> /dev/pts/37
6lrwx------ 1 marcus marcus 64 21 maj 16.16 1 -> /dev/pts/37
7lrwx------ 1 marcus marcus 64 21 maj 16.16 2 -> /dev/pts/37
8lrwx------ 1 marcus marcus 64 21 maj 16.16 3 -> 'anon_inode:[eventfd]'The file descriptors 0, 1 and 2 are the standard input, output and error respectively.
Summary
Using strace for debugging is very useful. If you have a multithreaded application then you may want to use the -f option to follow child processes and threads as well. In my case, there was a leak in a 3rd party library, not too easy to find without proper tools.