Reading a Large Text File in Python Line by Line

To read a big text file in Python, loop over the file object: for line in file:. It reads one line at a time, so a 10 GB log uses about as much memory as a 10 KB one. The methods that load everything at once, read() and readlines(), are fine for small files and a problem for big ones.

Make a test file

To see the difference, you need a file that's actually big. This writes about 2 million lines of fake log data, around 90 MB:

import random

random.seed(1)
levels = ["INFO"] * 90 + ["WARNING"] * 8 + ["ERROR"] * 2

with open("app.log", "w", encoding="utf-8") as file:
    for number in range(2_000_000):
        level = random.choice(levels)
        file.write(f"2026-10-13 08:{number % 60:02d}:00 {level} request {number} done\n")

print("written")

written

The slow way and the right way

import tracemalloc

tracemalloc.start()
with open("app.log", encoding="utf-8") as file:
    lines = file.readlines()
errors = sum(1 for line in lines if " ERROR " in line)
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
print(f"readlines(): {errors} errors, peak memory {peak / 1_000_000:.0f} MB")

tracemalloc.start()
errors = 0
with open("app.log", encoding="utf-8") as file:
    for line in file:
        if " ERROR " in line:
            errors += 1
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
print(f"for line in file: {errors} errors, peak memory {peak / 1_000_000:.2f} MB")

readlines(): 40148 errors, peak memory 191 MB
for line in file: 40148 errors, peak memory 0.02 MB

Same answer. readlines() built a list of 2 million strings and needed about twice the file's size in memory, because every Python string carries extra memory on top of its text. The loop kept only one line at a time, and its memory stayed tiny. With a 10 GB file, the first version doesn't get slow. It runs out of memory and the program, or the whole computer, freezes.

tracemalloc is only here to measure. You don't need it in real code.

Why the loop works

A file object is an iterator. Each turn of the for loop asks it for the next line, and it reads ahead in small blocks behind the scenes. Nothing keeps old lines around unless you store them. So the rule is simple: don't collect lines into a list unless you really need all of them at once.

That includes this, which looks innocent and isn't:

lines = [line for line in file]

It's readlines() with more typing.

Each line ends with \n

Every line you get includes its line break, except maybe the last one. Strip it before comparing or splitting:

with open("app.log", encoding="utf-8") as file:
    first = next(file)

print(repr(first))
print(repr(first.rstrip("\n")))
print(first.split()[2])

'2026-10-13 08:00:00 INFO request 0 done\n'
'2026-10-13 08:00:00 INFO request 0 done'
INFO

Use rstrip("\n") if spaces at the end matter. Use strip() if they don't. split() with no arguments ignores the line break anyway.

Counting, filtering and stopping early

Most big-file jobs are one of three shapes. Count something, copy some lines to another file, or find the first match and stop.

from collections import Counter

counts = Counter()
with open("app.log", encoding="utf-8") as source, open("errors.log", "w", encoding="utf-8") as target:
    for line in source:
        level = line.split()[2]
        counts[level] += 1
        if level == "ERROR":
            target.write(line)

print(counts.most_common())

with open("app.log", encoding="utf-8") as file:
    for line_number, line in enumerate(file, start=1):
        if "request 1500000 " in line:
            print("found on line", line_number)
            break

[('INFO', 1799820), ('WARNING', 160032), ('ERROR', 40148)]
found on line 1500001

Notice there's no list anywhere. The counts are a small dictionary, matching lines go straight to the new file, and break stops reading the moment the line is found. On a big file, stopping early is the biggest speed-up you'll ever get.

When there are no lines

Some big files are one huge line, like a minified JSON export. Then for line in file reads the whole thing as one "line" and you're back to the memory problem. For those, read fixed-size blocks:

size = 0
with open("app.log", "rb") as file:
    while chunk := file.read(1024 * 1024):
        size += len(chunk)

print(f"{size:,} bytes")

91,409,134 bytes

"rb" opens the file in binary mode, which is right for counting bytes or copying. For JSON that big, use a library built to read it in pieces, rather than splitting it yourself.

Common mistakes

• **read() or readlines() on a file that might grow.** Logs and exports always grow.
• Appending every line to a list "for later". If you only need counts or matches, keep only those.
• Opening the file inside the loop. Open once, loop inside the with.
• **No break when you only needed the first match.**

What happens if you skip this

The script works in testing on a small sample. It goes into a scheduled job. Months later the log is 6 GB, and the job starts failing at night with MemoryError or gets killed by the system with no message at all. Nothing in the code changed. The file did.

How big will your file be in a year, and would your script still run on it?