Here is my (or the) solution to monitor stdout and stderr of a subprocess in Python 3. Python's documentation recommends using subprocess.Popen.communicate() rather than subprocess.Popen.stdout.read() to avoid blocking the application, but you only get to use the output once the subprocess ends. Things get complicated if you need to monitor both stdout and stderr. I use select.select() to solve this. I don't think it works with MS Windows but that's your problem, not mine :).

import subprocess
import logging
import select

p = subprocess.Popen(["spam", "--verbose"],
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE)
outputs = {p.stdout: {"EOF": False, "logcmd": logging.info},
           p.stderr: {"EOF": False, "logcmd": logging.error}}
while (not outputs[p.stdout]["EOF"] and
       not outputs[p.stderr]["EOF"]):
    for fd in select.select([p.stdout, p.stderr], [], [])[0]:
        output = fd.readline()
        if output == b"":
            outputs[fd]["EOF"] = True
        else:
            outputs[fd]["logcmd"](output.decode().rstrip())

Resources:

Edit : Also read subprocess source code located at /usr/lib/python3.2/subprocess.py. You will learn how to do it with the threading module.

2012-06-02 edit : Mr Eric Pruitt was kind enough to share his improvement on this code. Thank you sir!

Comments

Comment Atom Feed
Avatar
Eric Pruitt

Hey Alexandre,

I was looking for some information on logging subprocess output when I found your post. I re-wrote the code in a little more compact form, and I thought you might be interested. Here is my version:

def iterate_fds(handles, functions):
    methods = dict(zip(handles, functions))
    while methods:
        for handle in select.select(methods.keys(), tuple(), tuple())[0]:
            line = handle.readline()
            if line:
                methods[handle](line[:-1])
            else:
                methods.pop(handle)

In my program, I am calling the code like this:

iterate_fds((rsync.stderr, rsync.stdout), (logging.warning, logging.info))

Eric

Add a Comment

You can use the Markdown syntax to format your comment.