Python File Handling: Open, Read, Write, and Always Use with

Open a file with with, say which encoding it uses, and pick the mode on purpose. Those three habits prevent most file bugs: files left open, names like "José" instead of "José", and a report overwritten when you meant to add to it.

Why with matters

The old way:

f = open("notes.txt", "w", encoding="utf-8")
f.write("first line\n")
# if something fails here, close() never runs
f.close()

If an error happens before close(), the file stays open. Written data may still be sitting in a buffer and not on the disk. On Windows, an open file also can't be deleted or renamed by other programs. With with, the file is closed when the block ends, even when an error happens inside it:

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")
    f.write("second line\n")

print(f.closed)

True

Reading a file

Three ways, for three situations:

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("first line\nsecond line\n")

# 1. The whole file as one string. Fine for small files.
with open("notes.txt", encoding="utf-8") as f:
    text = f.read()
print(repr(text))

# 2. Line by line. Works for files of any size.
with open("notes.txt", encoding="utf-8") as f:
    for line in f:
        print(line.rstrip("\n"))

# 3. A list of lines without the newline characters.
with open("notes.txt", encoding="utf-8") as f:
    lines = f.read().splitlines()
print(lines)

'first line\nsecond line\n'
first line
second line
['first line', 'second line']

Each line read with method 2 still ends with \n. That is why rstrip("\n") is there. Without it, print adds its own newline and you get blank lines between everything.

The modes: pick on purpose

• "r" (the default): read. Fails if the file doesn't exist.
• "w": write. Creates the file, or empties it first if it exists.
• "a": append. Adds to the end. Creates the file if needed.
• "x": create. Fails if the file already exists.

with open("log.txt", "a", encoding="utf-8") as f:
    f.write("run 1\n")
with open("log.txt", "a", encoding="utf-8") as f:
    f.write("run 2\n")

with open("log.txt", encoding="utf-8") as f:
    print(f.read(), end="")

try:
    with open("log.txt", "x", encoding="utf-8") as f:
        f.write("this would have replaced the log\n")
except FileExistsError:
    print("log.txt already exists, nothing was overwritten")

run 1
run 2
log.txt already exists, nothing was overwritten

Always say the encoding

Without encoding=, Python uses the computer's default. On Linux and macOS that is almost always UTF-8. On Windows it is often an older code page. So the same script writes "José" correctly on one computer and garbage on another. Here is what a wrong encoding looks like:

with open("names.txt", "w", encoding="utf-8") as f:
    f.write("José\nPeña\n")

with open("names.txt", encoding="latin-1") as f:
    print(f.read(), end="")

José
Peña

No error. Just wrong text. Use encoding="utf-8" unless you know the file uses something else. Files exported from Excel on Windows are one common exception. If you see odd characters at the very start of the first line, try encoding="utf-8-sig", which removes the invisible marker Excel adds.

Handle the file that isn't there

Don't let a missing file crash the program with a traceback the user can't read. Catch the specific error and say what to do:

def read_settings(path):
    try:
        with open(path, encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        print(f"Settings file not found: {path}. Copy settings.example.txt to {path} first.")
        return None


read_settings("settings.txt")

Settings file not found: settings.txt. Copy settings.example.txt to settings.txt first.

Catch FileNotFoundError, not a bare except:. A bare except also hides permission errors and typos in your own code, and then you have no idea what actually went wrong.

Write safely: temporary file, then replace

If the program crashes halfway through writing with "w", you have a half-written file and the old version is already gone. For files that matter, write to a temporary file and swap it in at the end:

import os

lines = ["id,total", "1,1200", "2,900"]

with open("report.csv.tmp", "w", encoding="utf-8") as f:
    f.write("\n".join(lines) + "\n")
os.replace("report.csv.tmp", "report.csv")

with open("report.csv", encoding="utf-8") as f:
    print(f.read(), end="")

id,total
1,1200
2,900

os.replace swaps the file in one step. Anyone reading report.csv sees either the old version or the new one, never half of it. For a throwaway output file this is overkill. For a file other programs read, it is worth the two extra lines.

Common mistakes

• Reading a huge file with read(). A 2 GB log loaded as one string needs at least 2 GB of memory. Loop over the file line by line.
• Using "w" in a loop. Opening with "w" inside a loop empties the file on every pass, so only the last item survives.
• Relative paths that depend on where you run from. open("data.txt") looks in the current folder, which isn't always the script's folder. The next lesson fixes this.

What happens if you skip this step

Your script works on your computer with your test file. Then a coworker runs it on Windows and the names come out broken, or it crashes halfway and leaves a half-written report that the next step reads as if it were complete. None of these show up in a quick test. They show up later, with real data.

If your script crashed right in the middle of writing its output file, what would the next person who opens that file see?