How to Execute Shell Commands With Python?

Dec. 23, 2021

How to Execute Shell Commands With Python?

If you need to execute a shell command with Python, there are two ways. You can either use the subprocess module or the run command( command.run() ) function.

The first option is easier to run one line of code and exit, but it isn’t as flexible when using arguments or producing text output. The second option is more complicated to code, but it will allow for more customization in your script.

I’ll’ be going through how to do both in this blog post!

I recommend you check out Shell Scripting: How to Automate Command Line Tasks Using Bash Scripting and Shell Programming. It helped me expand my understanding of shell scripts.

Other shell scripting resources
Linux Command Line and Shell Scripting Bible
Wicked Cool Shell Scripts

If you're working with Python and shell scripts, you may also want to check the four different ways to run Python scripts inside a terminal. It's the opposite of what we have in this post—hot to run python scripts from a terminal. Also, check this if you want to run scripts over ssh on a remote computer.

What is the purpose of shell commands?

Shell is a command-line interface (CLI) for interacting with the operating system. It provides an environment for running text-based commands and typically offers features not available in graphical user interfaces (GUIs), such as piping and redirection.

Shell environment is often called the command prompt in Windows and terminal in Linux systems.

To open the command prompt in Windows, press Win+R, type cmd, and hit Enter. You can open a Linux terminal by pressing Ctrl+Alt+T. Press Cmd+Space, type terminal, and hit Enter if you are on Mac OS.

While most users prefer GUI-based tools, advanced operations often command prompt usage. For instance, data scientists, programmers, and system engineers use shell commands to manage files, directories, and software. Additionally, CLIs can be used in batch scripts or cron jobs to automate repetitive tasks.

How to execute a shell command in Python?

Programmers often want to execute a shell command in more extensive Python code. There are two options to do so:

Use the subprocess module.

The subprocess module is a part of the standard library in Python. It provides facilities for executing shell commands and capturing their output or error.

Here’s how to run a shell script using Python. In this example, we’re copying a file from ‘dirA’ to ‘dirB.’

import subprocess

# This is our shell command, executed in subprocess.

p = subprocess.run("cp", "dirA/file", "dirB")

While run() is a much quicker way to execute shell scripts, you should use the Popen constructor for more flexibility.

The Popen constructor allows you to pass in additional arguments, such as shell=True, which will cause the command to run in a sub-shell.

Additionally, you can use stdout=PIPE to have Popen capture the returned output instead of simply displaying it on the screen.

Here's a quick example of how to run a shell command and capture its output:

import subprocess

# This is our shell command, executed by Popen.

p = subprocess.Popen("ls -lh", stdout=subprocess.PIPE, shell=True)

print(p.communicate())

The above Python program executes a shell command, ls -lh and gets an output. Note: If we don’t specify shell=True, Popen will only capture the first output line.

It's often very convenient to start Popen objects as context managers. We can use the with statement in Python. Here's an example.

with Popen(["ifconfig"], stdout=subprocess.PIPE) as proc:
   print(proc.stdout.read())

In the above code snippet, we use ifconfig as our shell command. The with statement ensures that the proc object is closed properly when we exit the block.

This is especially important when you are working with large amounts of data. If you don’t close the proc object, your program will close before all the data is read from stdout. Additionally, it will leave a file descriptor open for your OS.

Use the Command module.

The Command library is another convenient way to execute shell scripts inside Python. The Command module is a wrapper around the Popen method of the subprocess module. But, Command has outstanding capabilities for logging and debugging.

You can install the Command library from PyPI using the below command.

pip install Command

The following Python script executes a shell command.

import command 

res = command.run(['ls']) 

print(res.output) 
print(res.exit)

Also, you could set the debug parameter to a function to handle the logs differently. The following example prints the debug logs on the screen.

import command

def debugger(text):     
    print(text)
  
res = command.run(['ls'], debug=debugger) 

print(res.output) 
print(res.exit)

Use the os module.

The os module in Python provides access to many low-level operating system features. You can use it to list directories, change directories ( cd ), list information about files ( stat ), create files, and more.

Here’s an example of how to use os to get the list of files in the current directory.

Warning: This python method to execute bash commands is now deprecated.

import os

directories = os.system("ls -lh")
print(directories)

# You could also use the os.popen() method
with os.popen("ls -lh") as f:
    print(f.readlines())

Note: os.popen returns a file object, and we use the read() method to get the data from stdout.

os.system vs subprocess: What should you use?

The subprocess module is always preferable compared to the os.system one. This is because the latter has a hike risk of shell injection attack.

You can pass more than one command to the system utility by operating them out using a pipe. The following is an example of listing the files and creating one there.

os.system('ls -h | touch dangerous_file')

In addition to getting the list of files and folders in the current directory, the above command also creates a file. Attackers may use this vulnerability to access your servers and do harmful activities.

Thus avoid os.system unless it's necessary and you understand the risks—it's not the answer to exec shell scripts.

How to run a script with Python?

Often you might have to run other scripts inside a python program as you're running it from the shell. To do this, you can use the subprocess module as well.

In this example, we'll show how to run a bash script file called myscript.sh that takes an input filename as an argument.

import subprocess

subprocess.run(["myscript.sh", "--input_file=input.txt"])

If you're using an older version of Python (before 3.5), you should instead use the subprocess.call method.

Thanks for the read, friend. It seems you and I have lots of common interests. Say Hi to me on LinkedIn, Twitter, and Medium. I’ll break the ice for you.

Not a Medium member yet? Please use this link to
become a member because I earn a commission for referring at no extra cost for you.

How we work

Readers support The Analytics Club. We earn through display ads. Also, when you buy something we recommend, we may get an affiliate commission. But it never affects your price or what we pick.

Connect with us