Python Iterators: What a for Loop Actually Does

Every for loop in Python does the same two things behind the scenes: it asks the object for an iterator with iter(), then calls next() on it until it runs out. Once you see that, a lot of odd behaviour makes sense, like why a file or a zip can only be looped over once.

What a for loop actually does

colors = ["red", "green", "blue"]

it = iter(colors)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration: the loop ends here")

red
green
blue
StopIteration: the loop ends here

That is all for color in colors: is. It calls iter() once, then next() again and again, and stops quietly when it gets StopIteration.

Iterable vs iterator

Two words that sound the same and mean different things:

• An iterable is something you can loop over: a list, a string, a dictionary, a file. It can give you a new iterator every time you ask.
• An iterator is the thing doing the looping. It remembers its position and only goes forward. When it is used up, it stays used up.

numbers = [1, 2, 3]
print(list(numbers), list(numbers))     # a list is iterable: loop as often as you like

pairs = zip(["a", "b"], [1, 2])         # zip returns an iterator
print(list(pairs), list(pairs))         # the second time, it is empty
print(iter(pairs) is pairs)             # an iterator's iter() returns itself

[1, 2, 3] [1, 2, 3]
[('a', 1), ('b', 2)] []
True

That empty second list is a real bug pattern. You pass a zip, map, filter, file or generator to a function that loops over it twice, for example once to count and once to process. The second loop gets nothing, and there is no error.

Writing your own iterator

A class becomes an iterator by having two methods: __iter__ returns the iterator (itself), and __next__ returns the next value or raises StopIteration. Here is one that hands out records in pages, like an API that returns results 100 at a time:

class Pages:
    def __init__(self, items, size):
        self.items = items
        self.size = size
        self.position = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.position >= len(self.items):
            raise StopIteration
        page = self.items[self.position:self.position + self.size]
        self.position += self.size
        return page


for page in Pages(list(range(1, 8)), size=3):
    print(page)

[1, 2, 3]
[4, 5, 6]
[7]

It works, but notice two things. First, it is a lot of code for a simple idea. Second, it is single-use: loop over the same Pages object again and you get nothing. The next lesson writes the same thing with a generator in five lines.

Making a reusable iterable

If you want an object that can be looped over many times, make __iter__ return a new iterator each time instead of self:

class Catalog:
    def __init__(self, products):
        self.products = products

    def __iter__(self):
        return iter(self.products)   # a fresh iterator every time


catalog = Catalog(["Keyboard", "Mouse"])
print(list(catalog))
print(list(catalog))
print("Mouse" in catalog)

['Keyboard', 'Mouse']
['Keyboard', 'Mouse']
True

As a bonus, in, list(), sorted(), sum() and unpacking all work, because they all use the same iter()/next() protocol.

Useful tools in itertools

The itertools module works on iterators without building lists. Three you will use:

from itertools import islice, chain, count

print(list(islice(count(start=1000), 3)))     # first 3 of an endless counter
print(list(chain([1, 2], (3, 4), "ab")))      # several iterables as one

log_lines = iter(["header", "row 1", "row 2", "row 3"])
next(log_lines)                                # skip the header
print(list(islice(log_lines, 2)))             # take only the next two

[1000, 1001, 1002]
[1, 2, 3, 4, 'a', 'b']
['row 1', 'row 2']

count() never ends. That is fine, because islice only asks for three values. An iterator only does work when someone calls next(). This laziness is what makes processing huge files possible, and it is the core of the next lesson.

Common mistakes

• Looping over an iterator twice. Covered above. The second loop is silently empty.
• Calling len() on an iterator. len(zip(a, b)) is a TypeError. Iterators don't know how many items are left until they get there.
• Changing a list while looping over it. The iterator tracks a position, and removing items shifts everything after it, so items get skipped. Loop over a copy, or build a new list.

What happens if you skip this step

You keep running into "it worked with a list, but not with this" bugs and treat each one as a separate mystery. They are all the same thing: an iterator only goes forward, once. Knowing that turns a confusing class of bugs into an obvious one.

A function counts the rows of its input, then sums them. You pass it an open file. What does the sum come out as, and why?