Need To Avoid Subprocess Deadlock Without Communicate

I need a execute a command that produces a lot of output and takes a lot of time to execute (> 30 minutes). I was thinking of using subprocess.Popen to do it. I need to capture

Solution 1:

• to workaround python bug #1124861 on Python2.4 you could attach stdin to a NUL device

import os
from subprocess import PIPE, STDOUT, Popen

lines = []
p = Popen(cmd, bufsize=1, stdin=open(os.devnull), stdout=PIPE, stderr=STDOUT)
for line in iter(p.stdout.readline, ''):
      print line,          # print to stdout immediately
      lines.append(line)   # capture for later
p.stdout.close()
p.wait()

Solution 2:

Have you tried pexpect?

Solution 3:

It sounds like you need to do a non-blocking read on the filehandles attached to the pipes.

This question addresses some ways to do that for windows & linux: Non-blocking read on a subprocess.PIPE in python

Solution 4:

To avoid the pipe buffers filling up, just launch a background thread in the parent process. That thread can either just continuously read from stdout (and stderr) to keep the pipe buffers from filling up, or you can invoke communicate() from it. Either way, the main thread is free to continue with ordinary processing and the child process won't block on an output operation.

Converting a synchronous IO operation into an asynchronous one (from the point of view of the main thread) is one of the best use cases for threads. Even async frameworks like Twisted will sometimes use it as a last resort solution when no native asynchronous interface is available for a given operation.

Solution 5:

You might consider using multiple threads. Assign one thread to read from stdout, one from stderr, and use a third thread to detect the timeout:

whiletime.time() - last_output_time < 20 * 60:
    time.sleep( 20 * 60 - (time.time() - last_output_time) )
print'No output detected in the last 20 minutes. Terminating execution'
sys.exit(1)