Paginate an API With an Async Generator

Most APIs don't return 10,000 records at once. They return 100 and a way to ask for the next 100: a page number, a next URL or a cursor. If every caller handles that loop itself, the paging logic gets copied everywhere, with a slightly different bug in each copy. Put it in one async generator, and callers just loop over records.

The API

To keep this runnable without someone else's server, here's a fake API with 7 customers, 3 per page, that answers like a real one would:

const customers = ["Ana", "Ben", "Carla", "Dan", "Ella", "Fe", "Gio"];

async function fetchCustomers(page) {
  console.log(`  GET /customers?page=${page}`);
  await new Promise(resolve => setTimeout(resolve, 20));
  const start = (page - 1) * 3;
  const items = customers.slice(start, start + 3);
  return { items, nextPage: start + 3 < customers.length ? page + 1 : null };
}

With a real API, this is where your fetch call goes, with response.ok checked.

The generator

const customers = ["Ana", "Ben", "Carla", "Dan", "Ella", "Fe", "Gio"];
async function fetchCustomers(page) {
  console.log(`  GET /customers?page=${page}`);
  await new Promise(resolve => setTimeout(resolve, 20));
  const start = (page - 1) * 3;
  const items = customers.slice(start, start + 3);
  return { items, nextPage: start + 3 < customers.length ? page + 1 : null };
}

async function* allCustomers() {
  let page = 1;
  while (page !== null) {
    const data = await fetchCustomers(page);
    yield* data.items;
    page = data.nextPage;
  }
}

for await (const name of allCustomers()) {
  console.log(name);
}

  GET /customers?page=1
Ana
Ben
Carla
  GET /customers?page=2
Dan
Ella
Fe
  GET /customers?page=3
Gio

(Save it as .mjs, because for await at the top level needs a module.)

The caller doesn't know there are pages. It gets one name at a time. Page 2 is only requested when the loop asks for "Dan".

Stopping early is free

const customers = ["Ana", "Ben", "Carla", "Dan", "Ella", "Fe", "Gio"];
async function fetchCustomers(page) {
  console.log(`  GET /customers?page=${page}`);
  await new Promise(resolve => setTimeout(resolve, 20));
  const start = (page - 1) * 3;
  const items = customers.slice(start, start + 3);
  return { items, nextPage: start + 3 < customers.length ? page + 1 : null };
}
async function* allCustomers() {
  let page = 1;
  try {
    while (page !== null) {
      const data = await fetchCustomers(page);
      yield* data.items;
      page = data.nextPage;
    }
  } finally {
    console.log(`  stopped after page ${page}`);
  }
}

for await (const name of allCustomers()) {
  if (name.startsWith("D")) {
    console.log("Found:", name);
    break;
  }
}

  GET /customers?page=1
  GET /customers?page=2
Found: Dan
  stopped after page 2

Page 3 was never requested. When the loop breaks, the generator's finally block runs, so this is where you'd close a connection or release a cursor.

Cursor-based APIs

Many APIs use a cursor instead of a page number. The shape is the same:

async function* allOrders(baseUrl) {
  let url = `${baseUrl}/orders?limit=100`;
  while (url) {
    const response = await fetch(url);
    if (!response.ok) throw new Error(`HTTP ${response.status} for ${url}`);
    const data = await response.json();
    yield* data.items;
    url = data.next;
  }
}

If a page fails, the generator throws, and the error comes out of the caller's for await loop, where a normal try/catch handles it.

Collect when you really need everything

Sometimes you do need the full list. Then collect it on purpose:

const everyone = await Array.fromAsync(allCustomers());

Array.fromAsync works in Node 22 and later and in current browsers. With a big API, be sure you want all of it in memory first.

Your report needs the first 50 customers who joined this year, and the API returns customers sorted by join date, 100 per page. How many pages does the generator version fetch? How many would "load everything, then filter" fetch?