How to Execute Shell Commands With Python?

 If you need to execute a shell command with Python, there are two ways. You can either use the subprocess module or the run command( command.run() ) function.

The first option is easier to run one line of code and exit, but it isn’t as flexible when using arguments or producing text output. The second option is more complicated to code, but it will allow for more customization in your script.

I’ll be going through how to do both in this blog post!

I recommend you check out Shell Scripting: How to Automate Command Line Tasks Using Bash Scripting and Shell Programming. It helped me expand my understanding of shell scripts.

Other shell scripting resources
Linux Command Line and Shell Scripting Bible
Wicked Cool Shell Scripts

If you’re working with Python and shell scripts, you may also want to check the four different ways to run Python scripts inside a terminal. It’s the opposite of what we have in this post—how to run Python scripts from a terminal. Also, check this if you want to run scripts over ssh on a remote computer.

What is the purpose of shell commands?

Shell is a command-line interface (CLI) for interacting with the operating system. It provides an environment for running text-based commands and typically offers features not available in graphical user interfaces (GUIs), such as piping and redirection.

Shell environment is often called the command prompt in Windows and terminal in Linux systems.

To open the command prompt in Windows, press Win+R, type cmd, and hit Enter. You can open a Linux terminal by pressing Ctrl+Alt+T. Press Cmd+Space, type terminal, and hit Enter if you are on Mac OS.

While most users prefer GUI-based tools, advanced operations often command prompt usage. For instance, data scientists, programmers, and system engineers use shell commands to manage files, directories, and software. Additionally, CLIs can be used in batch scripts or cron jobs to automate repetitive tasks.

How to execute a shell command in Python?

Programmers often want to execute a shell command in more extensive Python code. There are two options to do so:

Use the subprocess module.

The subprocess module is a part of the standard library in Python. It provides facilities for executing shell commands and capturing their output or error.

Here’s how to run a shell script using Python. In this example, we’re copying a file from ‘dirA’ to ‘dirB.’

import subprocess

# This is our shell command, executed in subprocess.

p = subprocess.run("cp", "dirA/file", "dirB")

While run() is a much quicker way to execute shell scripts, the Popen constructor could be more flexible.

The Popen constructor allows you to pass in additional arguments, such as shell=True, which will cause the command to run in a sub-shell.

Additionally, you can use stdout=PIPE to have Popen capture the returned output instead of simply displaying it on the screen.

Here’s a quick example of how to run a shell command and capture its output:

import subprocess

# This is our shell command, executed by Popen.

p = subprocess.Popen("ls -lh", stdout=subprocess.PIPE, shell=True)

print(p.communicate())

The above Python program executes the shell command ls -lh and prints the result. Note: If we don’t specify shell=True, Popen will only capture the first output line.

It’s often very convenient to start Popen objects as context managers. We can use the with statement in Python. Here’s an example.

with Popen(["ifconfig"], stdout=subprocess.PIPE) as proc:
   print(proc.stdout.read())

In the above code snippet, we use ifconfig as our shell command. The with statement ensures that the proc object is closed correctly when we exit the block.

This is especially important when you are working with large amounts of data. If you don’t close the proc object, your program will close before all the data is read from stdout. Additionally, it will leave a file descriptor open for your OS.

Use the Command module.

The Command library is another convenient way to execute shell scripts inside Python. The Command module is a wrapper around the Popen method of the subprocess module. But, Command has outstanding capabilities for logging and debugging.

You can install the Command library from PyPI using the below command.

pip install Command

The following Python script executes a shell command.

import command 

res = command.run(['ls']) 

print(res.output) 
print(res.exit)

Also, you could set the debug parameter to a function to handle the logs differently. The following example prints the debug logs on the screen.

import command

def debugger(text):     
    print(text)
  
res = command.run(['ls'], debug=debugger) 

print(res.output) 
print(res.exit)

Use the os module.

The os module in Python provides access to many low-level operating system features. You can use it to list directories, change directories ( cd ), list information about files ( stat ), create files, and more.

Here’s an example of using the os module to get the list of files in the current directory.

Warning: This Python method to execute bash commands is now deprecated.

import os

directories = os.system("ls -lh")
print(directories)

# You could also use the os.popen() method
with os.popen("ls -lh") as f:
    print(f.readlines())

Note: os.popen returns a file object, and we use the read() method to get the data from stdout.

os.system vs. subprocess: What should you use?

The subprocess module is always preferable compared to the os.system one. This is because the latter has a hike risk of shell injection attack.

You can pass more than one command to the system utility by operating them out using a pipe. The following is an example of listing the files and creating one there.

os.system('ls -h | touch dangerous_file')

In addition to getting the list of files and folders in the current directory, the above command also creates a file. Attackers may use this vulnerability to access your servers and do harmful activities.

Thus avoid os.system unless it’s necessary and you understand the risks—it’s not the answer to exec shell scripts.

How to run a script with Python?

Often you might have to run other scripts inside a Python program as you’re running it from the shell. To do this, you can use the subprocess module as well.

In this example, we’ll show how to run a bash script file called myscript.sh that takes an input filename as an argument.

import subprocess

subprocess.run(["myscript.sh", "--input_file=input.txt"])

If you’re using an older version of Python (before 3.5), you should instead use the subprocess.call method.

Thanks for the read, friend. It seems you and I have lots of common interests. Say Hi to me on LinkedIn, Twitter, and Medium.

Not a Medium member yet? Please use this link to become a member because I earn a commission for referring at no extra cost for you.