Skip to content Skip to sidebar Skip to footer
Home / Multiprocessing / Multithreading / Parallel Processing / Python / User Interface

In Python, How Do I Know When A Process Is Finished?

Post a Comment
From within a Python GUI (PyGTK) I start a process (using multiprocessing). The process takes a long time (~20 minutes) to finish. When the process is finished I would like to cl

Solution 1:

I think as a part of making python multi-platform, simple things like SIGCHLD must be done yourself. Agreed, this is a little more work when all you want to do is know when the child is done, but it really isn't THAT painful. Consider the following that uses a child process to do the work, two multiprocessing.Event instances, and a thread to check if the child process is done:

import threading
from multiprocessing import Process, Event
from time import sleep

defchildsPlay(event):
    print"Child started"for i inrange(3):
        print"Child is playing..."
        sleep(1)
    print"Child done"
    event.set()

defcheckChild(event, killEvent):
    event.wait()
    print"Child checked, and is done playing"if raw_input("Do again? y/n:") == "y":
        event.clear()
        t = threading.Thread(target=checkChild, args=(event, killEvent))
        t.start()
        p = Process(target=childsPlay, args=(event,))
        p.start()
    else:
        cleanChild()
        killEvent.set()

defcleanChild():
    print"Cleaning up the child..."if __name__ == '__main__':
    event = Event()
    killEvent = Event()

    # process to do work
    p = Process(target=childsPlay, args=(event,))
    p.start()

    # thread to check on child process
    t = threading.Thread(target=checkChild, args=(event, killEvent))
    t.start()

    try:
        whilenot killEvent.is_set():
            print"GUI running..."
            sleep(1)
    except KeyboardInterrupt:
        print"Quitting..."
        exit(0)
    finally:
        print"Main done"

EDIT

Joining to all processes and threads created is a good practice because it will help indicate when zombie (never-finishing) processes/threads are being created. I've altered the above code making a ChildChecker class that inherits from threading.Thread. It's sole purpose is to start a job in a separate process, wait for that process to finish, and then notify the GUI when everything is complete. Joining on the ChildChecker will also join the process it is "checking". Now, if the process doesn't join after 5 seconds, the thread will force terminate the process. Enter "y" creates starts a child process running "endlessChildsPlay" that must demonstrate force termination. 

import threading
from multiprocessing import Process, Event
from time import sleep

defchildsPlay(event):
    print"Child started"for i inrange(3):
        print"Child is playing..."
        sleep(1)
    print"Child done"
    event.set()

defendlessChildsPlay(event):
    print"Endless child started"whileTrue:
        print"Endless child is playing..."
        sleep(1)
        event.set()
    print"Endless child done"classChildChecker(threading.Thread):
    def__init__(self, killEvent):
        super(ChildChecker, self).__init__()
        self.killEvent = killEvent
        self.event = Event()
        self.process = Process(target=childsPlay, args=(self.event,))

    defrun(self):
        self.process.start()

        whilenot self.killEvent.is_set():
            self.event.wait()
            print"Child checked, and is done playing"if raw_input("Do again? y/n:") == "y":
                self.event.clear()
                self.process = Process(target=endlessChildsPlay, args=(self.event,))
                self.process.start()
            else:
                self.cleanChild()
                self.killEvent.set()

    defjoin(self):
        print"Joining child process"# Timeout on 5 seconds
        self.process.join(5)

        if self.process.is_alive():
            print"Child did not join!  Killing.."
            self.process.terminate()
        print"Joining ChildChecker thread"super(ChildChecker, self).join()


    defcleanChild(self):
        print"Cleaning up the child..."if __name__ == '__main__':
    killEvent = Event()
    # thread to check on child process
    t = ChildChecker(killEvent)
    t.start()

    try:
        whilenot killEvent.is_set():
            print"GUI running..."
            sleep(1)
    except KeyboardInterrupt:
        print"Quitting..."
        exit(0)
    finally:
        t.join()
        print"Main done"

Solution 2:

This answer is really simple! (It just took me days to work it out.)

Combined with PyGTK's idle_add(), you can create an AutoJoiningThread. The total code is borderline trivial:

classAutoJoiningThread(threading.Thread):defrun(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)

If you want to do more than just join (such as collecting results) then you can extend the above class to emit signals on completion, as is done in the following example:

import threading
import time
import sys
import gobject
gobject.threads_init()

classChild:
    def__init__(self):
        self.result = Nonedefplay(self, count):
        print"Child starting to play."for i inrange(count):
            print"Child playing."
            time.sleep(1)
        print"Child finished playing."
        self.result = 42defget_result(self, obj):
        print"The result was "+str(self.result)

classAutoJoiningThread(threading.Thread, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
        }

    def__init__(self, *args, **kwargs):
        threading.Thread.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    defrun(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)
        gobject.idle_add(self.emit, 'finished')

    defjoin(self):
        threading.Thread.join(self)
        print"Called Thread.join()"if __name__ == '__main__':
    print"Creating child"
    child = Child()
    print"Creating thread"
    thread = AutoJoiningThread(target=child.play,
                               args=(3,))
    thread.connect('finished', child.get_result)
    print"Starting thread"
    thread.start()
    print"Running mainloop (Ctrl+C to exit)"
    mainloop = gobject.MainLoop()

    try:
        mainloop.run()
    except KeyboardInterrupt:
        print"Received KeyboardInterrupt.  Quiting."
        sys.exit()

    print"God knows how we got here.  Quiting."
    sys.exit()

The output of the above example will depend on the order the threads are executed, but it will be similar to:

Creating child
Creating thread
Starting thread
Child starting to play.
 Child playing.
Running mainloop (Ctrl+C to exit)
Child playing.
Child playing.
Child finished playing.
Called Thread.join()
The result was 42
^CReceived KeyboardInterrupt.  Quiting.

It's not possible to create an AutoJoiningProcess in the same way (because we cannot call idle_add() across two different processes), however we can use an AutoJoiningThread to get what we want:

classAutoJoiningProcess(multiprocessing.Process):defstart(self):
        thread = AutoJoiningThread(target=self.start_process)
        thread.start() # automatically joinsdefstart_process(self):
        multiprocessing.Process.start(self)
        self.join()

To demonstrate AutoJoiningProcess here is another example:

import threading
import multiprocessing
import time
import sys
import gobject
gobject.threads_init()

classChild:
    def__init__(self):
        self.result = multiprocessing.Manager().list()

    defplay(self, count):
        print"Child starting to play."for i inrange(count):
            print"Child playing."
            time.sleep(1)
    print"Child finished playing."
        self.result.append(42)

    defget_result(self, obj):
        print"The result was "+str(self.result)

classAutoJoiningThread(threading.Thread, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
    }

    def__init__(self, *args, **kwargs):
        threading.Thread.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    defrun(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)
        gobject.idle_add(self.emit, 'finished')

    defjoin(self):
        threading.Thread.join(self)
        print"Called Thread.join()"classAutoJoiningProcess(multiprocessing.Process, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
        }

    def__init__(self, *args, **kwargs):
        multiprocessing.Process.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    defstart(self):
        thread = AutoJoiningThread(target=self.start_process)
        thread.start()

    defstart_process(self):
        multiprocessing.Process.start(self)
        self.join()
        gobject.idle_add(self.emit, 'finished')

    defjoin(self):
        multiprocessing.Process.join(self)
        print"Called Process.join()"if __name__ == '__main__':
    print"Creating child"
    child = Child()
    print"Creating thread"
    process = AutoJoiningProcess(target=child.play,
                               args=(3,))
    process.connect('finished',child.get_result)
    print"Starting thread"
    process.start()
    print"Running mainloop (Ctrl+C to exit)"
    mainloop = gobject.MainLoop()

    try:
        mainloop.run()
    except KeyboardInterrupt:
        print"Received KeyboardInterrupt.  Quiting."
        sys.exit()

    print"God knows how we got here.  Quiting."
    sys.exit()

The resulting output will be very similar to the example above, except this time we have both the process joining and it's attendant thread joining too:

Creating child
Creating thread
Starting thread
Running mainloop (Ctrl+C to exit)
 Child starting to play.
Child playing.
Child playing.
Child playing.
Child finished playing.
Called Process.join()
The result was [42]
Called Thread.join()
^CReceived KeyboardInterrupt.  Quiting.

Unfortunately:

  1. This solution is dependent on gobject, due to the use of idle_add(). gobject is used by PyGTK.
  2. This is not a true parent/child relationship. If one of these threads is started by another thread, then it will nonetheless be joined by the thread running the mainloop, not the parent thread. This problem holds true for AutoJoiningProcess too, except there I imagine an exception would be thrown.

Thus to use this approach, it would be best to only create threads/process from within the mainloop/GUI.

Solution 3:

You can use a queue to communicate with child processes. You can stick intermediate results on it, or messages indicating that milestones have been hit (for progress bars) or just a message indicating that the process is ready to be joined. Polling it with empty is easy and fast.

If you really only want to know if it's done, you can watch the exitcode of your process or poll is_alive().

Solution 4:

In my efforts to try to find an answer to my own question, I stumbled across PyGTK's idle_add() function. This gives me the following possibility:

  1. Create a new child process that communicates via a Queue.
  2. Create a listener thread that listens to the Queue, when the child process sends the listener a message saying that it is finished, the listener calls idle_add() that sets up a callback.
  3. During the next time around the main loop the parent process will call the callback.
  4. The callback can extract results, join the child process and join the listener-thread.

This seems an overly complex way to re-create Unix's call-callback-when-child-process-is-done.

This must be an uber-common problem with GUIs in Python. Surely there is a standard pattern to solve this problem?

Solution 5:

have a look at the subprocess module:

http://docs.python.org/library/subprocess.html

import subprocess
let pipe = subprocess.Popen("ls -l", stdout=subprocess.PIPE)
allText = pipe.stdout.read()
pipe.wait()
retVal = pipe.returncode

Post a Comment for "In Python, How Do I Know When A Process Is Finished?"