Python Generators and yield: One Piece at a Time

A generator produces values one at a time, only when asked, instead of building the whole result first. For small data, that difference doesn't matter. For a log file bigger than your RAM, it is the difference between a script that finishes and one that gets killed.

yield: a function that pauses

def countdown(n):
    print("starting")
    while n > 0:
        yield n
        n -= 1
    print("done")


gen = countdown(3)
print(gen)
print(next(gen))
print(next(gen))
print(list(gen))

<generator object countdown at 0x...>
starting
3
2
done
[1]

Calling countdown(3) doesn't run anything. It returns a generator, which is an iterator (see the previous lesson). Each next() runs the function until the next yield, hands out that value, and pauses there with all its local variables intact. When the function ends, the generator raises StopIteration for you. That is why done prints before [1]: list() collected the last value, then the function finished, and only then did print show the list.

The pager, rewritten

The previous lesson wrote a Pages class with __iter__ and __next__. As a generator:

def pages(items, size):
    for start in range(0, len(items), size):
        yield items[start:start + size]


for page in pages(list(range(1, 8)), 3):
    print(page)

[1, 2, 3]
[4, 5, 6]
[7]

Same result, no class, no position counter, no StopIteration. Python keeps track of where the function paused.

Why it matters: memory

Here is a test with a log file of 500,000 lines. One version reads it into a list, the other uses a generator. tracemalloc measures the peak memory Python used:

import tracemalloc

with open("app.log", "w", encoding="utf-8") as f:
    for i in range(500_000):
        level = "ERROR" if i % 1000 == 0 else "INFO"
        f.write(f"2026-10-01 12:00:00 {level} request {i} handled in 12ms\n")


def errors_list(path):
    with open(path, encoding="utf-8") as f:
        lines = f.readlines()                     # the whole file in memory
    return [line for line in lines if " ERROR " in line]


def errors_gen(path):
    with open(path, encoding="utf-8") as f:
        for line in f:                            # one line at a time
            if " ERROR " in line:
                yield line


for func in (errors_list, errors_gen):
    tracemalloc.start()
    found = sum(1 for _ in func("app.log"))
    peak = tracemalloc.get_traced_memory()[1]
    tracemalloc.stop()
    print(f"{func.__name__}: {found} errors, peak {peak / 1_000_000:.1f} MB")

errors_list: 500 errors, peak ... MB
errors_gen: 500 errors, peak 0.0 MB

When this was tested for the course, the list version peaked at about 53 MB and the generator version at well under 0.1 MB (it rounds to 0.0). Your exact numbers will differ by Python version and system. The pattern won't: the list version grows with the file, the generator version stays flat. Make the file 20 GB and the list version runs out of memory while the generator version keeps going.

Pipelines: small generators chained together

The real strength is chaining. Each step takes an iterator and yields to the next, and only one line is in flight at a time:

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        yield from f


def only_errors(lines):
    return (line for line in lines if " ERROR " in line)


def request_ids(lines):
    for line in lines:
        yield int(line.split()[4])


with open("app.log", "w", encoding="utf-8") as f:
    for i in range(10_000):
        level = "ERROR" if i % 2500 == 0 else "INFO"
        f.write(f"2026-10-01 12:00:00 {level} request {i} handled in 12ms\n")

ids = request_ids(only_errors(read_lines("app.log")))
print(list(ids))

[0, 2500, 5000, 7500]

Each function does one job and can be tested on its own with a short list instead of a file. yield from f means "yield every item of f". only_errors uses a generator expression, the round-bracket version of a comprehension, which is fine for one-line steps.

Taking only what you need

Because nothing runs until it is asked for, stopping early is free:

from itertools import islice


def squares():
    n = 1
    while True:
        print(f"computing {n}")
        yield n * n
        n += 1


print(list(islice(squares(), 3)))

computing 1
computing 2
computing 3
[1, 4, 9]

An endless generator is safe as long as the consumer stops. list(squares()) would never finish, so be careful who you hand it to.

When not to use a generator

• You need the data more than once. A generator is used up after one pass. If you need to sort, count and then process, a list is simpler, as long as it fits in memory.
• You need len() or indexing. Generators have neither.
• The data is small. A list of 200 rows is fine. Using a generator there adds nothing and makes debugging slightly harder, because you can't just print it.

Common mistakes

• Printing a generator to see what's inside. You get <generator object ...>. Convert a small one with list(), or take a few with islice.
• Calling list() on a huge generator. That loads everything into memory again and cancels the whole point.
• Expecting code before the first yield to run immediately. It runs on the first next(). Validation errors in a generator function appear when the loop starts, not when you call it.

What happens if you skip this step

Scripts that read whole files into lists work for months, while the files are small. Then the data grows, and one day the script slows down, the machine starts swapping, and the process is killed with no Python error at all. Writing it as a generator from the start costs nothing extra.

Your log file doubles in size every year. Which part of your current script would fail first, and in which year?