How to Download Files From a URL Using Python?

Downloading files is a common task in many programming projects, and Python provides several libraries and tools to make this process effortless.

In this blog post, we will explore three popular methods for downloading files using Python: requests, selenium, and wget SSH command. We’ll walk through each method step-by-step and provide code examples along the way. So, let’s dive in!

Downloading files using requests

The requests library is widely used for making HTTP requests, including file downloads. Here’s a simple example of how you can use requests to download a file:

import requests

url = 'https://example.com/file.txt'
response = requests.get(url)

if response.status_code == 200:
    with open('file.txt', 'wb') as file:
        file.write(response.content)
        print('File downloaded successfully!')
else:
    print('Failed to download the file.')

In this code snippet, we start by specifying the URL of the file we want to download. Then, we use the requests.get() method to send an HTTP GET request to the specified URL. If the request is successful (status code 200), we open a file in binary write mode ('wb'), write the response content to the file, and print a success message. Otherwise, we print an error message indicating the failure.

Downloading using Selenium

Selenium is primarily known for browser automation, but it can also be used to download files programmatically. Here’s an example of how you can leverage Selenium for file downloads:

from selenium import webdriver

options = webdriver.ChromeOptions()

# Run in headless mode, without opening a browser window
options.add_argument('--headless')  

driver = webdriver.Chrome(options=options)

url = 'https://example.com/file.txt'
driver.get(url)

print('File downloaded successfully!')
driver.quit()

In this code snippet, we first import the webdriver module from Selenium. Then, we create a ChromeOptions object to configure the browser’s behavior. In this case, we set the --headless option to run the browser in headless mode, meaning it won’t open a visible browser window.

Next, we create an instance of the Chrome web driver, passing in the options we defined earlier. We use the driver.get() method to navigate to the URL of the file we want to download. After the download completes, we print a success message and quit the browser using driver.quit().

Downloading using the wget shell command

The wget command is a versatile utility commonly used for downloading files from the command line. By leveraging the subprocess module, we can execute wget directly from within our Python script. This approach is particularly useful when working with large files that are not feasible to load entirely into memory.

Here’s an example code snippet for downloading a file using the wget shell command:

import subprocess

file_url = "https://example.com/file.pdf"
save_path = "path/to/save/file.pdf"

# Execute wget command in the shell
subprocess.call(["wget", file_url, "-O", save_path])

print("File downloaded successfully!")

In this example, we use the subprocess.call() function to execute the wget command in the shell. We pass the file URL as an argument to wget, along with the -O option to specify the output file path.

Downloading using the wget SSH command inside Python

If you prefer not to load file downloads into the Python runtime and want a more efficient method, you can utilize the wget command through an SSH connection. Here’s an example of using the paramiko library to establish an SSH connection and execute the wget command:

import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('your_server', username='your_username', password='your_password')

command = 'wget -P /path/to/save https://example.com/file.txt'
stdin, stdout, stderr = ssh.exec_command(command)

error = stderr.read().decode().strip()
if error:
print(f'Failed to download the file. Error: {error}')
else:
print('File downloaded successfully!')

ssh.close()

In this code snippet, we import the paramiko library, which provides an interface to SSH connections. We create an SSHClient instance and configure it to add the host key of the SSH server automatically.

Next, we establish an SSH connection by calling ssh.connect() with the appropriate server, username, and password. We define the wget command we want to execute, specifying the destination path where the file should be saved.

Then, we use ssh.exec_command() to execute the command on the remote server. We capture the standard input, output, and error streams of the command execution. If there is an error message, we print it. Otherwise, we print a success message.

Finally, we close the SSH connection using ssh.close().

Conclusion

Downloading files using Python is made simple and efficient with various libraries and tools.

In this blog post, we explored three popular methods: requests, selenium, and wget SSH command.

Depending on your project requirements, you can choose the most suitable method to effortlessly download files. Remember to handle any potential errors and ensure the security and reliability of the downloaded files.

Be sure to check my post about the best practices when programmatically downloading files using Python.

Thanks for the read, friend. It seems you and I have lots of common interests. Say Hi to me on LinkedIn, Twitter, and Medium.

Not a Medium member yet? Please use this link to become a member because I earn a commission for referring at no extra cost for you.