Speeding Up Many Downloads in Python With ThreadPoolExecutor

If your program downloads many files one after another, it spends almost all its time waiting for the network, not working. ThreadPoolExecutor lets several downloads wait at the same time. For 20 files that each take half a second, that's the difference between 10 seconds and about 2. It doesn't help with work that keeps the CPU busy, and it doesn't make any single download faster.

Is your job waiting or working?

Threads help when the slow part is waiting: downloads, API calls, database queries, reading from a network drive. While one thread waits for a reply, Python lets another one run.

They don't help when the slow part is calculating: resizing images, parsing huge files, number crunching. In standard Python, only one thread runs Python code at a time, so ten threads calculating are about as fast as one. For that, use ProcessPoolExecutor, which has the same interface.

Quick test: if the CPU sits near idle while your script runs, it's waiting. Threads will help.

A test server so the numbers are repeatable

To show real timings without hammering a real website, this tutorial downloads from a small server on your own computer that waits half a second before each reply, like a slow API. Save it and start it in a separate terminal:

import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


class SlowHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(0.5)
        body = f"report for {self.path}\n".encode() * 1000
        self.send_response(200)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


ThreadingHTTPServer.request_queue_size = 100  # let many connections wait at once
ThreadingHTTPServer(("127.0.0.1", 8765), SlowHandler).serve_forever()

python slow_server.py

Everything below works the same with real URLs. Only the address changes.

One at a time

import time
from urllib.request import urlopen

URLS = [f"http://127.0.0.1:8765/report-{n}.txt" for n in range(1, 21)]


def download(url):
    with urlopen(url, timeout=10) as response:
        return len(response.read())


start = time.perf_counter()
total = sum(download(url) for url in URLS)
print(f"{len(URLS)} files, {total:,} bytes in {time.perf_counter() - start:.1f}s")

20 files, 511,000 bytes in 10.0s

Twenty files, half a second each, ten seconds. The computer did almost nothing for those ten seconds except wait.

With a thread pool

import time
from concurrent.futures import ThreadPoolExecutor
from urllib.request import urlopen

URLS = [f"http://127.0.0.1:8765/report-{n}.txt" for n in range(1, 21)]


def download(url):
    with urlopen(url, timeout=10) as response:
        return len(response.read())


start = time.perf_counter()
with ThreadPoolExecutor(max_workers=5) as pool:
    sizes = list(pool.map(download, URLS))
print(f"{len(URLS)} files, {sum(sizes):,} bytes in {time.perf_counter() - start:.1f}s")

20 files, 511,000 bytes in 2.0s

Five workers, four rounds of half a second: about two seconds. pool.map works like the built-in map, and returns results in the same order as the input, no matter which download finished first. The with block waits for every download to finish before moving on.

The pattern with as_completed

pool.map has one weakness: if one download raises an error, you only get it when the loop reaches that item, and the remaining results are lost with it. For real downloads, where some will fail, submit each one and handle results as they finish:

from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.error import URLError
from urllib.request import urlopen

URLS = [f"http://127.0.0.1:8765/report-{n}.txt" for n in range(1, 6)]
URLS.append("http://127.0.0.1:9/missing.txt")


def download(url):
    with urlopen(url, timeout=10) as response:
        return len(response.read())


ok, failed = {}, {}
with ThreadPoolExecutor(max_workers=5) as pool:
    futures = {pool.submit(download, url): url for url in URLS}
    for future in as_completed(futures):
        url = futures[future]
        try:
            ok[url] = future.result()
        except URLError as error:
            failed[url] = error.reason

print(f"{len(ok)} downloaded, {len(failed)} failed")
for url, reason in failed.items():
    print("FAILED", url, "-", reason)

5 downloaded, 1 failed
FAILED http://127.0.0.1:9/missing.txt - [Errno 111] Connection refused

How it works:

1. **pool.submit(download, url)** starts the job and returns a Future, a placeholder for the result.
2. The dictionary maps each future back to its URL, so you know which one failed.
3. **as_completed gives you futures in the order they finish**, so fast ones aren't held up by slow ones.
4. **future.result()** returns the value, or raises the exception that happened inside the thread. That's where you catch it.

The one bad URL failed alone. The other five finished normally.

How many workers?

There's no perfect number. A few guidelines:

• Start with 5 to 10 for downloads from one server.
• More workers to the same server isn't always faster. The server or your connection becomes the limit, and some servers slow you down or block you.
• Different servers can take more workers, because no single one gets all the load.
• Measure. Try 5, 10 and 20 on your real job and keep the smallest number that's close to the fastest time.

Be polite to the server

Twenty parallel requests from a script can look like an attack to a small website. Before you run this against something you don't own, check its terms or API documentation for rate limits, keep max_workers low, set a timeout so stuck requests don't hang forever, and don't retry failed requests in a tight loop.

Common mistakes

• Threads for CPU-heavy work. No speed-up. Use ProcessPoolExecutor.
• **No timeout.** One stuck request keeps a worker busy forever.
• **Never calling result().** Errors vanish.
• Writing to the same file or list from many threads without thinking. Return values from the function and collect them in the main thread, like above. That avoids most thread-safety problems.
• **max_workers=100 "to be safe".** It's not safer. It's just more load.

What happens if you skip it

The script still works. It just takes ten times longer than it needs to, and it gets slower with every file you add. For a nightly job that downloads 2,000 reports at half a second each, that's over 16 minutes of waiting, every night.

How long does your slowest script spend waiting, and how long does it actually spend working?