Python: run shell commands using the subprocess package

In this article, you’ll learn how to execute shell commands using the subprocess package in Python.

This article is part of a two-part series related to running shell commands from within Python.

• Part 1: Execute shell commands with the os package
• Part 2: Execute shell commands with the subprocess package

As I explained in part 1 of this series, the popen method of the os package uses the subprocess package to run shell commands. In this article, we’ll use the subprocess package ourselves.

The popen method of the subprocess package executes a child program in a new subprocess. This process can then be accessed (i.e.) via the communicate method. This method waits for the process to finish and finally prints the stdout and stderr as a tuple.

import subprocess
process = subprocess.Popen(['echo', '"Hello World!"'],
                     stdout = subprocess.PIPE, 
                     stderr = subprocess.PIPE,
                     text = True,
                     shell = True)
std_out, std_err = process.communicate()
std_out.strip(), std_err

❗ If you don’t set the shell parameter to True for the echo command, you’ll run into the following error:

FileNotFoundError: [WinError 2] The system cannot find the file specified

This is because the command/program takes arguments. Running the Popen method requires that you specify a single string (the path to the program), or explicitly set shell to True if your command contains spaces (e.g. arguments).

Instead of waiting for a command to finish, you could also use poll(). This method checks if a process is finished. The following script will run a command (in this case ping) and keep printing the output if it hasn’t finished. If an error is encountered, it will print the process exit code and the error message.

process = subprocess.Popen(['ping', 'facebook.com'], 
                           stdout = subprocess.PIPE,
                           stderr = subprocess.PIPE,
                           text = True,
                           shell = True)

while True:
    output = process.stdout.readline().strip()
    if output != '':
      print(output)
    return_code = process.poll()
    if return_code is not None:
        print(f'Returned the following return code: {return_code}')
        for std_output in process.stdout.readlines():
            print(std_output)
        for std_err in process.stderr.readlines():
            print(std_err)
        break

Finally, if splitting your command in separate strings is too cumbersome, use the split function of the shlex package.

import shlex
import subprocess
shell_cmd = shlex.split('echo "Hello, World!"')

import subprocess
process = subprocess.Popen(shell_cmd,
                     stdout = subprocess.PIPE, 
                     stderr = subprocess.PIPE,
                     text = True,
                     shell = True
                     )
std_out, std_err = process.communicate()
std_out.strip(), std_err

Great success!