How to Execute Shell Commands With Python?

Dec. 23, 2021

How to Execute Shell Commands With Python?

If you need to execute a shell command with Python, there are two ways. You can either use the subprocess module or the RunShellCommand() function.

The first option is easier to run one line of code and then exit, but it isn’t as flexible when using arguments or producing text output. The second option is more complicated to code, but it will allow for more customization in your script.

I’ll’ be going through how to do both in this blog post!

Did you know, like running shell commands in Python, there are four different ways to run Python commands in a terminal?

What is the purpose of shell commands?

Shell is a command-line interface (CLI) for interacting with the operating system. It provides an environment for running text-based commands and typically offers several features not available in graphical user interfaces (GUIs), such as piping and redirection.

Shell environment is often referred to as the command prompt in Windows and terminal in Linux.

To open the command prompt in Windows, press Win+R, type cmd, and hit Enter. On Linux, you can open the terminal by pressing Ctrl+Alt+T. Press Cmd+Space, type terminal, and hit Enter if you are on Mac OS.

While most users prefer GUI-based tools, advanced operations often command prompt usage. For instance, data scientists and programmers use shell commands to manage files, directories, and software. Additionally, CLIs can be used in batch scripts or cron jobs to automate repetitive tasks.

How to execute a shell command in Python?

Programmers often want to execute a shell command as inside a more extensive Python code. There are two options to do so:

Use the subprocess module.

The subprocess module is a part of the standard library in Python. It provides facilities for executing shell commands and capturing their output or error.

Here’s how to run a shell script using Python. In this example, we’re copying a file from ‘dirA’ to ‘dirB.’

import subprocess

# This is our shell command, executed in subprocess.

p = subprocess.run("cp", "dirA/file", "dirB")

While run() is a much quicker way to execute shell scripts, you should use the Popen constructor for more flexibility.

The Popen constructor allows you to pass in additional arguments, such as shell=True, which will cause the command to run in a sub-shell.

Additionally, you can use stdout=PIPE to have Popen capture the returned output instead of simply displaying it on the screen.

Here's a quick example of how to run a shell command and capture its output:

import subprocess

# This is our shell command, executed by Popen.

p = subprocess.Popen("ls -lh", stdout=subprocess.PIPE, shell=True)

print(p.communicate())

The above Python program executes a shell command, ls -lh and gets an output. Note: If we don’t specify shell=True, Popen will only capture the first output line.

It's often very convenient to start Popen objects as context managers. We can use the with statement in Python. Here's an example.

with Popen(["ifconfig"], stdout=subprocess.PIPE) as proc:
   print(proc.stdout.read())

In the above code snippet, we use ifconfig as our shell command. The with statement ensures that the proc object is closed properly when we exit the block.

This is especially important when you are working with large amounts of data. If you don’t close the proc object properly, your program will close before all the data is read from stdout. Additionally, it will leave a file descriptor open for your OS.

Use the Command module

The Command library is another convenient way to execute shell scripts inside Python. The Command module is a wrapper around the Popen method of the subprocess module. But, Command has outstanding capabilities for logging and debugging.

You can install the Command library from PyPI using the below command.

pip install Command

The following Python script executes a shell command.

import command 

res = command.run(['ls']) 

print(res.output) 
print(res.exit)

Also, you could set the debug parameter to a function to handle the logs differently. The following example prints the debug logs on the screen.

import command

def debugger(text):     
    print(text)
  
res = command.run(['ls'], debug=debugger) 

print(res.output) 
print(res.exit)

Use the os module.

The os module in Python provides access to many low-level features of the operating system. You can use it to list directories, change directories ( cd ), list information about files ( stat ), create files, and more.

Here’s an example of how to use os to get the list of files in the current directory.

import os

directories = os.system("ls -lh")
print(directories)

# You could also use the os.popen() method
with os.popen("ls -lh") as f:
    print(f.readlines())

Note: os.popen returns a file object, and we use the read() method to get the data from stdout.

How to run a script with Python?

Often times you might have to run other scripts inside a python program as you're running it from the shell. To do this, you can use the subprocess module as well.

In this example, we'll show how to run a script called myscript.sh that takes an input filename as an argument.

import subprocess

subprocess.run(["myscript.sh", "--input_file=input.txt"])

If you're using an older version of Python (prior to 3.5) you should instead use the subprocess.call method.

Thanks for the read, friend. It seems you and I have lots of common interests. Say Hi to me on LinkedIn, Twitter, and Medium. I’ll break the ice for you.
Not a Medium member yet? Please use this link to become a member because I earn a commission for referring at no extra cost for you.

How we work

Readers support The Analytics Club. We earn through display ads. Also, when you buy something we recommend, we may get an affiliate commission. But it never affects your price or what we pick.

Connect with us