Python: Make System Call

By Xah Lee. Date: . Last updated: .

To call a unix shell command, first import subprocess then call one of the following, depending on whether you want Python to return the exit code or command output.

Call Shell Command, Get Output

subprocess.check_output([cmd, arg1, arg2, …]) → Run shell command cmd arg1 arg2. Wait for command to complete, then return its output. If the return code is not 0, then raises CalledProcessError.

# python 3

# call unix shell command and get its output

import subprocess

output = subprocess.check_output(["ls", "-al"])

print(output.decode("utf-8"))

Call Shell Command, Ignore Output, Get Exit Code

subprocess.call([cmd, arg1, arg2, …]) → Run shell command cmd arg1 arg2. Wait for command to complete, then return the shell command exit code.

# python 3

# call unix shell command and get its exit code. (ignore output)
import subprocess
exitCode = subprocess.call(["ls", "-al"])
print(exitCode)

subprocess.call() will run successfully even if the shell command did not run successfully.

# python 3

# call unix shell command and get its exit code.

import subprocess

# here, ls will exit with error code 2, because there's no option -y
exitCode = subprocess.call(["ls", "-y"])

# python will not raise error
print exitCode  # 2

(Linux shell commands have exit code of 0 if ran successfully.)

Call Shell Command, Ignore Output, Stop on Error

subprocess.check_call([cmd, arg1, arg2, …]) → Run shell command cmd arg1 arg2. Wait for command to complete. If the return code is 0 then return 0, else raise CalledProcessError. To get exit code, call CalledProcessError.returncode.

# python 3

# call unix shell command. Raise exception if the command failed

import subprocess

# ls does not have -y option
exitCode = subprocess.check_call(["ls", "-y"])

# this will not be printed
print("done")

# python exit error

# ls: illegal option -- y
# usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
# Traceback (most recent call last):
#   File "/Users/xah/web/xahlee_info/python/xxtemp.20190322.272.py3", line 8, in <module>
#     exitCode = subprocess.check_call(["ls", "-y"])
#   File "/Users/xah/anaconda3/lib/python3.7/subprocess.py", line 341, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['ls', '-y']' returned non-zero exit status 1.

Advanced: 「subprocess.Popen()」

All the following commands:

are wrappers to the more general function subprocess.Popen(). Here's the code:

def call(*popenargs, **kwargs):
    """Run command with arguments.  Wait for command to complete, then
    return the returncode attribute.

    The arguments are the same as for the Popen constructor.  Example:

    retcode = call(["ls", "-l"])
    """
    return Popen(*popenargs, **kwargs).wait()
def check_call(*popenargs, **kwargs):
    """Run command with arguments.  Wait for command to complete.  If
    the exit code was zero then return, otherwise raise
    CalledProcessError.  The CalledProcessError object will have the
    return code in the returncode attribute.

    The arguments are the same as for the Popen constructor.  Example:

    check_call(["ls", "-l"])
    """
    retcode = call(*popenargs, **kwargs)
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd)
    return 0
def check_output(*popenargs, **kwargs):
    r"""Run command with arguments and return its output as a byte string.

    If the exit code was non-zero it raises a CalledProcessError.  The
    CalledProcessError object will have the return code in the returncode
    attribute and output in the output attribute.

    The arguments are the same as for the Popen constructor.  Example:

    >>> check_output(["ls", "-l", "/dev/null"])
    'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

    The stdout argument is not allowed as it is used internally.
    To capture standard error in the result, use stderr=STDOUT.

    >>> check_output(["/bin/sh", "-c",
    ...               "ls -l non_existent_file ; exit 0"],
    ...              stderr=STDOUT)
    'ls: non_existent_file: No such file or directory\n'
    """
    if 'stdout' in kwargs:
        raise ValueError('stdout argument not allowed, it will be overridden.')
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
    output, unused_err = process.communicate()
    retcode = process.poll()
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd, output=output)
    return output

Examples of using 「subprocess.Popen(…)」

Call and Don't Wait

Suppose you want to decompress a file. gzip -d x.txt.gz

# python 3

import subprocess

# call a unix command but not wait for it to finish
subprocess.Popen([r"gzip","-d", "x.txt.gz"])

Call and Wait for Command to Finish

Suppose you want to decompress a file. gzip -d x.txt.gz

# python 3

import subprocess
subprocess.Popen([r"gzip","-d", "x.txt.gz"]).wait()

The subprocess.Popen([…]) creates a object. The .wait() makes the code wait until the system call finished.

Get Output

To make a system call and get its output, pass the argument stdout=subprocess.PIPE, and get output from the method communicate()[0].

# python 3

import subprocess

# call a unix command, wait for it to finish, get its output

myOutput=subprocess.Popen([r"tail","-n 1", "x.txt"], stdout=subprocess.PIPE).communicate()[0]

The method communicate() returns a tuple (stdoutdata, stderrdata), but you must pass arguments stdout=subprocess.PIPE and stderr=subprocess.PIPE to subprocess.Popen().

Full Example

Following is a complete example. The code makes a system call to decompress a gzip file (and wait for it), then get the last line of the file by calling the unix util “tail”, then gzip the file again.

# python 3

import subprocess

# decompress a file
subprocess.Popen([r"gzip","-d", "x.txt.gz"]).wait()

# get last line
last_line = subprocess.Popen([r"tail","-n 1", "x.txt"], stdout=subprocess.PIPE).communicate()[0]

# compress the file
subprocess.Popen([r"gzip","x.txt"])

print(last_line)

Note: Python has a gzip module. If you really want to compress/decompress a file, use that. [see Python: Compress/Decompress Gzip Files]

Get Output, Stderr, Exit Code, Process Id

Here's a example of getting both output and exit code.

# python 3

import subprocess

# call shell command
subP = subprocess.Popen(["du", "--si", "-s", "/usr/bin"],  stdout=subprocess.PIPE , stderr=subprocess.PIPE)

stdout, stderr = subP.communicate()

exitCode = subP.returncode

pid = subP.pid

print("--------------------\n output is:", stdout)

print("--------------------\n stderr is:", stderr)

print("--------------------\n exit code is:", exitCode)

print("--------------------\n pid is:", pid)

If you have a question, put $5 at patreon and message me.

Python

  1. Python 3 Basics
  2. Python 2 Basics
  3. Python 2 and 3 Difference
  4. Print Version
  5. Builtin Help
  6. Quote String
  7. String Methods
  8. Format String
  9. Operators
  10. Complex Numbers
  11. True, False
  12. if then else
  13. Loop
  14. List Basics
  15. Loop Thru List
  16. Map f to List
  17. Copy Nested List
  18. List Comprehension
  19. List Methods
  20. Sort
  21. Dictionary
  22. Loop Thru Dict
  23. Dict Methods
  24. Tuple
  25. Sets
  26. Function
  27. Closure
  28. 2 Closure
  29. Decorator
  30. Class
  31. Object, ID, Type
  32. List Modules
  33. Write a Module
  34. Unicode 🐍

Regex

  1. Regex Basics
  2. Regex Reference

Text Processing

  1. Read/Write File
  2. Traverse Directory
  3. File Path
  4. Process Unicode
  5. Convert File Encoding
  6. Find Replace in dir
  7. Find Replace by Regex
  8. Count Word Frequency

Web

  1. Send Email
  2. GET Web Page
  3. Web Crawler
  4. HTTP POST

Misc

  1. JSON
  2. Find Script Path
  3. Get Env Var
  4. System Call
  5. Decompress Gzip
  6. Append String in Loop
  7. Timing f timeit
  8. Keyword Arg Default Value Unstable
  9. Check Page Load Size
  10. Thumbnail Generation