Python: Make System Call

By Xah Lee. Date: . Last updated: .

To call a unix shell command, first import subprocess then call one of the following, depending on whether you want Python to return the exit code or command output.

Call Shell Command, Get Output

subprocess.check_output([cmd, arg1, arg2, etc])
Run shell command. Wait for command to finish, then return its output. If the exit code is not 0, then raises CalledProcessError. 
# call unix shell command and get its output

import subprocess

output = subprocess.check_output(["ls", "-al"])

print(output.decode("utf-8"))

Call Shell Command, Ignore Output, Get Exit Code

subprocess.call([cmd, arg1, arg2, etc])
Run shell command. Wait for command to finish, then return the shell command exit code. 
# call unix shell command and get its exit code. (ignore output)
import subprocess
exitCode = subprocess.call(["ls", "-al"])
print(exitCode)

subprocess.call() will run successfully even if the shell command did not run successfully.

# call unix shell command and get its exit code.

import subprocess

# here, ls will exit with error code 2, because there is no option -y
exitCode = subprocess.call(["ls", "-y"])

# python will not raise error
print exitCode  # 2

(Linux shell commands have exit code of 0 if ran successfully.)

Call Shell Command, Ignore Output, Stop on Error

subprocess.check_call([cmd, arg1, arg2, etc])
Run shell command. Wait for command to finish. If the return code is 0 then return 0, else raise CalledProcessError. To get exit code, call CalledProcessError.returncode. 
# call unix shell command. Raise exception if the command failed

import subprocess

# ls does not have -y option
exitCode = subprocess.check_call(["ls", "-y"])

# this will not be printed
print("done")

# python exit error

# ls: illegal option -- y
# usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
# Traceback (most recent call last):
#   File "/Users/xah/web/xahlee_info/python/xxtemp.20190322.272.py3", line 8, in <module>
#     exitCode = subprocess.check_call(["ls", "-y"])
#   File "/Users/xah/anaconda3/lib/python3.7/subprocess.py", line 341, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ls', '-y']' returned non-zero exit status 1.

Advanced: subprocess.Popen()

All the following commands:

are wrappers to the more general function subprocess.Popen(). Here's the code:

def call(*popenargs, **kwargs):
    """Run command with arguments.  Wait for command to complete, then
    return the returncode attribute.

    The arguments are the same as for the Popen constructor.  Example:

    retcode = call(["ls", "-l"])
    """
    return Popen(*popenargs, **kwargs).wait()
def check_call(*popenargs, **kwargs):
    """Run command with arguments.  Wait for command to complete.  If
    the exit code was zero then return, otherwise raise
    CalledProcessError.  The CalledProcessError object will have the
    return code in the returncode attribute.

    The arguments are the same as for the Popen constructor.  Example:

    check_call(["ls", "-l"])
    """
    retcode = call(*popenargs, **kwargs)
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd)
    return 0
def check_output(*popenargs, **kwargs):
    r"""Run command with arguments and return its output as a byte string.

    If the exit code was non-zero it raises a CalledProcessError.  The
    CalledProcessError object will have the return code in the returncode
    attribute and output in the output attribute.

    The arguments are the same as for the Popen constructor.  Example:

    >>> check_output(["ls", "-l", "/dev/null"])
    'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

    The stdout argument is not allowed as it is used internally.
    To capture standard error in the result, use stderr=STDOUT.

    >>> check_output(["/bin/sh", "-c",
    ...               "ls -l non_existent_file ; exit 0"],
    ...              stderr=STDOUT)
    'ls: non_existent_file: No such file or directory\n'
    """
    if 'stdout' in kwargs:
        raise ValueError('stdout argument not allowed, it will be overridden.')
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
    output, unused_err = process.communicate()
    retcode = process.poll()
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd, output=output)
    return output

Examples of using: subprocess.Popen()

Call and Don't Wait

Suppose you want to decompress a file. gzip -d x.txt.gz

import subprocess

# call a unix command but not wait for it to finish
subprocess.Popen([r"gzip","-d", "x.txt.gz"])

Call and Wait for Command to Finish

Suppose you want to decompress a file. gzip -d x.txt.gz

import subprocess
subprocess.Popen([r"gzip","-d", "x.txt.gz"]).wait()

The subprocess.Popen([…]) creates a object. The .wait() makes the code wait until the system call finished.

Get Output

To make a system call and get its output, pass the argument stdout=subprocess.PIPE, and get output from the method communicate()[0].

import subprocess

# call a unix command, wait for it to finish, get its output

myOutput=subprocess.Popen([r"tail","-n 1", "x.txt"], stdout=subprocess.PIPE).communicate()[0]

The method communicate() returns a tuple (stdoutdata, stderrdata), but you must pass arguments stdout=subprocess.PIPE and stderr=subprocess.PIPE to subprocess.Popen().

Full Example

Following is a complete example. The code makes a system call to decompress a gzip file (and wait for it), then get the last line of the file by calling the unix util “tail”, then gzip the file again.

import subprocess

# decompress a file
subprocess.Popen([r"gzip","-d", "x.txt.gz"]).wait()

# get last line
last_line = subprocess.Popen([r"tail","-n 1", "x.txt"], stdout=subprocess.PIPE).communicate()[0]

# compress the file
subprocess.Popen([r"gzip","x.txt"])

print(last_line)

Note: Python has a gzip module. If you really want to compress/decompress a file, use that. [see Python: Compress/Decompress Gzip Files]

Get Output, Stderr, Exit Code, Process Id

Here's a example of getting both output and exit code.

import subprocess

# call shell command
subP = subprocess.Popen(["du", "--si", "-s", "/usr/bin"],  stdout=subprocess.PIPE , stderr=subprocess.PIPE)

stdout, stderr = subP.communicate()

exitCode = subP.returncode

pid = subP.pid

print("--------------------\n output is:", stdout)

print("--------------------\n stderr is:", stderr)

print("--------------------\n exit code is:", exitCode)

print("--------------------\n pid is:", pid)