Bash Shell in Java: error=24, Too many open files

Bash Shell in Java: “Cannot run program ”/bin/bash“: java.io.IOException: error=24, Too many open files”.

If you happen to see above error in your logs, there is a simple solution to fix this. It is only a matter of few lines in your code.

Recently i have faced a problem as in the title. I was using JAVA's ProcessBuilder class, which allows to use bash shell and its scripts. Unfortunately my program after a while started to log that it "Cannot run program".

If you encounter similar problem, googling a while will give you plenty of pages with some more or less accurate solutions. Most of them will tell to check how many files are currently open in the system (by unix command lsof) and how's that number related to your system limit (check by bash command ulimit -n). Increasing maximum number of open files at a time is short-term solution in my opinion.

Let's get back to the core of problem. In my code I was using below lines:

ProcessBuilder pb = new ProcessBuilder("bash", "-c", command);
pb.redirectErrorStream(true); // use this to capture messages sent to stderr
Process shell = pb.start();
InputStream shellIn = shell.getInputStream(); // this captures the output from the command
int shellExitStatus = shell.waitFor(); // wait for the shell to finish and get the return code
InputStreamReader reader = new InputStreamReader(shellIn);
BufferedReader buf = new BufferedReader(reader);
// Read lines using buf

After using new ProcessBuilder and reader of input stream we must assure that InputStreamReader will be closed after reading as it should release any associated files. So I have added following lines:

if(reader != null) {
  try {
    reader.close();
  } catch (IOException ex) {
    log.error(ex, ex);
  }
}
if(shellIn != null) {
  try {
    shellIn.close();
  } catch (IOException ex) {
    log.error(ex, ex);
  }
}

Above code should be used in finally block. This solution works but not always, because it will not close all pipes and subprocesses triggered by shell command. For example when I have used

ps aux | grep java | awk '{if($1=="root") print $11}' | grep test | wc -l

I was creating multiple unnamed pipes, which stayed open and were increasing the total number of open files. To cleanup after executing shell command all subprocesses have to be stopped:

if(shell != null) {
  try {
    shell.destroy();
  } catch (Exception ex) {
    log.error(ex, ex);
  }
}

Destroying all subprocesses will do the thing. It will not impact on the result of shell command as in the code we have following line

int shellExitStatus = shell.waitFor();

To summarize, if we want to avoid this problem we should use something similar to below function:

  public String bashCommand(String command) {
        ProcessBuilder pb = new ProcessBuilder("bash", "-c", command);
        pb.redirectErrorStream(true); // use this to capture messages sent to stderr
        Process shell = null;
        String result = "";
        InputStream shellIn = null;
        InputStreamReader reader = null;
        try {
            shell = pb.start();
            shellIn = shell.getInputStream(); // this captures the output from the command
            int shellExitStatus = shell.waitFor();
            reader = new InputStreamReader(shellIn);
            BufferedReader buf = new BufferedReader(reader);
            String line;
            while ((line = buf.readLine()) != null) {
                result += line + "\n";
            }
        } catch (InterruptedException ig) {
            // Handle error
        } catch (IOException ignoreMe) {
            // Handle error
        } finally {
            if(reader != null) {
                try {
                    reader.close();
                } catch (IOException ex) {
                    // Handle error
                }
            }
            if(shellIn != null) {
                try {
                    shellIn.close();
                } catch (IOException ex) {
                    // Handle error
                }
            }
            if(shell != null) {
                try {
                    shell.destroy();
                } catch (Exception ex) {
                    // Handle error
                }
            }
        }
        return result;
    }

That's all. Hope it will help :)

Posted in: Bash, Java, Java tips, Shell