Python Multi-threading and Concurrency: Concurrent file downloading

Last update on December 21 2024 07:47:13 (UTC/GMT +8 hours)

Write a Python program to download multiple files concurrently using threads.

Sample Solution:

Python Code:

import threading
import urllib.request
def download_file(url, filename):
    print(f"\nDownloading {filename} from {url}...")
    urllib.request.urlretrieve(url, filename)
    print(f"\n{filename} downloaded successfully.")

# Create a list of files to download
files_to_download = [
    {"url": "https://en.wikipedia.org/wiki/British_logistics_in_the_Normandy_campaign", "filename": "i:\wfile1"},
    {"url": "https://en.wikipedia.org/wiki/Graph_(abstract_data_type)", "filename": "i:\Graph_abstract_data_type"},
    {"url": "https://example.com/", "filename": "i:\example"}
]

# Create a list to store the threads
threads = []

# Create a thread for each file and start the download
for file_info in files_to_download:
    thread = threading.Thread(
        target=download_file,
        args=(file_info["url"], file_info["filename"])
    )
    thread.start()
    threads.append(thread)

# Wait for all threads to complete
for thread in threads:
    thread.join()

Sample Output:

Downloading i:\wfile1 from https://en.wikipedia.org/wiki/British_logistics_in_the_Normandy_campaign...
Downloading i:\Graph_abstract_data_type from https://en.wikipedia.org/wiki/Graph_(abstract_data_type)...
Downloading i:\example from https://example.com/...

i:\Graph_abstract_data_type downloaded successfully.
i:\wfile1 downloaded successfully.
i:\example downloaded successfully.

Explanation:

In the above exercise -

• The "download_file()" function is defined, which takes a URL and a filename as input. The function uses urllib.request.urlretrieve to download the file from the specified URL and save it with the given filename.
• Create a list "files_to_download" that contains dictionaries specifying the URL and filename for each file we want to download.
• Create an empty list of threads to store thread objects.
• Using a loop, create a thread for each file in "files_to_download". Each thread is assigned the "download_file()" function as the target, and we pass the URL and filename as arguments.
• We start each thread by calling its start method, which initiates the file download.
• After starting all the threads, we use another loop to call the join method on each thread. By doing this, the program ensures that all threads are completed before proceeding.

Flowchart:

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Creating and managing threads.
Next: Concurrent even and odd number printing.