Reading a Paginated API Without Holding the Whole Thing in Memory

Your API hands out 50 records at a time across 400 pages. You need all of them. You do not need them all at once.

Here's a very familiar situation that shows up constantly on the backend. Some API returns data in pages, 50 or 100 records at a time, and you need to walk every page: sync them to your database, export them to a file, run a report. The endpoint gives you a cursor or a page number and you keep asking until there's nothing left.

The way most of us write it the first time looks like this:

async function getAllRecords() {

  const all = [];

  let cursor = 0;

  while (cursor !== null) {

    const { records, nextCursor } = await fetchPage(cursor);

    all.push(...records);

    cursor = nextCursor;

  }

  return all;

}



const everything = await getAllRecords();

for (const record of everything) {

  process(record);

}

It works. At four hundred records it's fine. The trouble starts when the dataset grows, and it has three separate problems hiding in it.

It holds the entire dataset in memory before you touch a single record. It's all or nothing: if page 380 fails, you've thrown away the 19,000 records you already fetched. And it's eager. You can't start processing record one until the very last page has landed, even if all you wanted was the first ten.

, we pulled rows out of a huge file one at a time with a generator, so the file never fully loaded into memory. Lazy. Pull-based. You ask for the next row, you get the next row, nothing more.

In

One fetch. Not 400. When you break, the generator is paused at a yield, and breaking out of the loop means nobody ever asks it for record eleven. So it never runs the loop body again. It never fetches page two. The laziness here is the entire advantage: you only use the compute for the pages you actually process.

What this does to memory

The eager version's real cost is that it keeps every record alive at once. The streaming version holds about one page at a time. To show the gap more accurately, I measured peak heap growth for both, at three dataset sizes, with the same chunky records:

dataset       collect-all peak     stream peak

10,000 rows        4.0 MB             3.7 MB

100,000 rows      36.1 MB            12.4 MB

500,000 rows     161.1 MB            15.8 MB

The next time an API hands you data 50 rows at a time, you don't have to choose between holding all of it and writing a tangle of cursor bookkeeping. You write a loop that looks eager and runs lazy. The paging hides itself, and you only ever pay for the pages you actually walk through.

Cheers :)