Python Generators: Handle Big Files Without Running Out of Memory

Loading a 2 GB log file into a list is a bad idea. Your program reads the whole file into memory before it does anything useful, and on a small server, it simply dies. A generator reads one line, handles it, and moves on to the next. Same result, a tiny fraction of the memory.

First, is memory really your problem?

Be honest about this. It depends on the size.

• A file of a few MB: load it however you like. A list is fine, and sometimes simpler.
• Hundreds of MB or more, or a file that keeps growing (logs, exports): use generators.
• You need the data more than once, or need to sort all of it: a generator alone won't do it. You can only go through a generator once. See "When a generator is the wrong tool".

The test file

To see the difference, we need a big file. This script writes 500,000 fake log lines, about 29 MB. Small compared to a real production log, but enough to measure:

import random

random.seed(1)
levels = ["INFO"] * 18 + ["WARNING"] + ["ERROR"]
with open("app.log", "w", encoding="utf-8") as file:
    for i in range(500_000):
        level = random.choice(levels)
        file.write(f"2026-10-13 08:{i // 60000 % 60:02d}:{i // 1000 % 60:02d} {level} request {i} handled in {random.randint(5, 900)} ms\n")
print("done")

done

Step 1: the list version and the generator version

import tracemalloc


def errors_list(path):
    with open(path, encoding="utf-8") as file:
        lines = file.readlines()
    return [line for line in lines if " ERROR " in line]


def errors_gen(path):
    with open(path, encoding="utf-8") as file:
        for line in file:
            if " ERROR " in line:
                yield line


for name, func in [("list", errors_list), ("generator", errors_gen)]:
    tracemalloc.start()
    count = sum(1 for _ in func("app.log"))
    peak = tracemalloc.get_traced_memory()[1]
    tracemalloc.stop()
    print(f"{name:<10} {count:,} errors, peak memory {peak / 1_000_000:.1f} MB")

This is what I got on Python 3.13. Your numbers will differ a bit:

list       25,094 errors, peak memory 53.8 MB
generator  25,094 errors, peak memory 0.0 MB

Same 25,094 errors. The list version needed about 54 MB, more than the 29 MB file itself, because every line becomes a Python string with its own overhead. The generator stayed under 0.1 MB. Make the file 100 times bigger and the list needs 100 times more memory. The generator still needs about the same.

tracemalloc is part of the standard library. It measures the memory Python allocates. I used it here so you can see the difference instead of trusting me.

Step 2: what yield actually does

errors_gen looks like a normal function, but yield makes it a generator. Calling it doesn't run the code. It gives you an object that runs until the next yield each time something asks for a value:

def count_up(limit):
    print("starting")
    for number in range(1, limit + 1):
        print("about to yield", number)
        yield number
    print("finished")


numbers = count_up(2)
print("nothing has run yet")
print(next(numbers))
print(next(numbers))
for leftover in numbers:
    print("never printed")

nothing has run yet
starting
about to yield 1
1
about to yield 2
2
finished

Notice "starting" only prints when the first value is requested. Each next runs the function until the next yield, then pauses there. When the function ends, the generator is done, and the for loop at the end gets nothing. That's the important part. A generator can only be used once.

Step 3: chain generators into a pipeline

This is where generators really pay off. Each step is a small generator that takes the one before it. Only one line moves through the whole pipeline at a time:

def read_lines(path):
    with open(path, encoding="utf-8") as file:
        for line in file:
            yield line.rstrip("\n")


def only_level(lines, level):
    for line in lines:
        if f" {level} " in line:
            yield line


def durations(lines):
    for line in lines:
        yield int(line.split(" handled in ")[1].removesuffix(" ms"))


lines = read_lines("app.log")
errors = only_level(lines, "ERROR")
times = durations(errors)

slow = sum(1 for ms in times if ms > 800)
print(f"Slow error requests: {slow:,}")

Slow error requests: 2,840

Each function does one thing and can be tested on its own with a small list of fake lines. Nothing is read from the file until sum asks for the first value. removesuffix needs Python 3.9 or newer.

You can also write short steps as generator expressions, with round brackets instead of square ones:

with open("app.log", encoding="utf-8") as file:
    warnings = (line for line in file if " WARNING " in line)
    print(f"Warnings: {sum(1 for _ in warnings):,}")

Warnings: 24,764

Square brackets build the whole list in memory. Round brackets make a generator. One character of difference, and with a big file it's the difference between working and crashing.

When a generator is the wrong tool

• You need to sort everything, or need the 10 largest. Sorting needs all the values. If the values are small numbers, collecting just those into a list is fine. For the top 10, use heapq.nlargest(10, times), which keeps only 10 at a time.
• You need random access, like "line 300,000". A generator only goes forward.
• The data is small. A list is easier to debug: you can print it, check its length, look at it twice. Don't make simple code complicated to save memory you don't need.

Common mistakes

• file.readlines() or file.read() on a big file. That's the whole file in memory. Loop over the file object instead.
• Using a generator twice. The second time is empty.
• [...] when you meant (...). Square brackets build a list.
• Calling len() on a generator. It fails with a TypeError. Generators don't know their length until they're finished.

What happens if you skip this

It works on your test file. Then it runs on the real log on a server with 1 GB of memory, and the process gets killed halfway through, often with no Python error at all, just gone. Or the server starts swapping and everything else on it slows down too. Writing it as a generator from the start costs almost nothing. Rewriting it under pressure costs a lot more.