Process a Large CSV in Node.js With Streams

fs.readFileSync("orders.csv", "utf8") loads the whole file into one string. That works for a few megabytes. With a multi-gigabyte export, Node either fails with an out-of-memory crash or refuses because the string would be too long. Streams read the file a chunk at a time, so it doesn't matter how big it gets.

Make a test file

The examples use a generated CSV, so you can run them anywhere:

import { createWriteStream } from "node:fs";
import { once } from "node:events";

const out = createWriteStream("orders.csv");
out.write("id,customer,status,total\n");
for (let i = 1; i <= 500_000; i++) {
  const line = `${i},C${i % 1000},${i % 20 === 0 ? "refunded" : "paid"},${(i % 50) * 10}\n`;
  if (!out.write(line)) await once(out, "drain");
}
out.end();
await once(out, "finish");
console.log("wrote orders.csv");

wrote orders.csv

Notice the if (!out.write(line)) await once(out, "drain") line. write returns false when its buffer is full. Waiting for "drain" before writing more is backpressure, and it keeps even the writer's memory flat. Save each example as an .mjs file in the same folder.

Read it line by line

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const lines = createInterface({ input: createReadStream("orders.csv"), crlfDelay: Infinity });

let count = 0;
let paid = 0;
let refunded = 0;
let isHeader = true;
for await (const line of lines) {
  if (isHeader) { isHeader = false; continue; }
  const [, , status, total] = line.split(",");
  count++;
  if (status === "paid") paid += Number(total);
  else refunded += Number(total);
}

console.log({ count, paid, refunded });
console.log("heap used under 50 MB:", process.memoryUsage().heapUsed < 50 * 1024 * 1024);

{ count: 500000, paid: 117500000, refunded: 5000000 }
heap used under 50 MB: true

readline takes the raw chunks from the read stream and gives you whole lines, even when a chunk ends in the middle of one. crlfDelay: Infinity treats Windows line endings (\r\n) as a single line break. The heap stays small because only the current chunk is in memory. Make the file ten times bigger and it stays about the same.

Write a filtered copy

To write only the refunded orders to a new file, connect the steps with pipeline. An async generator works as the middle step:

import { createReadStream, createWriteStream } from "node:fs";
import { createInterface } from "node:readline";
import { pipeline } from "node:stream/promises";

async function* onlyRefunded(lines) {
  let isHeader = true;
  for await (const line of lines) {
    if (isHeader) { isHeader = false; yield line + "\n"; continue; }
    if (line.split(",")[2] === "refunded") yield line + "\n";
  }
}

await pipeline(
  createInterface({ input: createReadStream("orders.csv"), crlfDelay: Infinity }),
  onlyRefunded,
  createWriteStream("refunded.csv"),
);

let rows = -1;
for await (const _ of createInterface({ input: createReadStream("refunded.csv") })) rows++;
console.log("refunded rows:", rows);

refunded rows: 25000

pipeline handles backpressure between every step, waits until the last byte is written, and closes all the streams if any step throws. That last part is why you use it instead of chaining .pipe() calls.

What not to do inside the loop

• Don't push every row into an array "to use later". You're rebuilding the whole file in memory.
• Don't await a slow network call per row without a limit. 500,000 rows at 50 ms each is almost 7 hours. Batch the rows (for example, 500 per request) instead.
• Don't swallow parse errors. Count bad lines and log the line numbers, so you can tell the person who sent the file.

Clean up

rm orders.csv refunded.csv

(On Windows, del orders.csv refunded.csv.)

If the totals had to be grouped per customer instead of per status, what would you keep in memory, and would it still stay flat as the file grew?