I needed to ZIP 1,000+ files totaling 10 GB stored in Cloudflare R2. The catch: I had to do it on Cloudflare Workers -- 128 MB memory, no filesystem, no long-running processes. Every existing solution I tried failed. So I built my own streaming ZIP64 archiver from scratch.
This is the story of how that archiver became eazip.io, and the technical decisions behind it.
The Problem
I run a file-sharing service. Users upload files to Cloudflare R2, and sometimes they want to download hundreds or thousands of files at once. The obvious answer is a ZIP file. The not-so-obvious part is where to create it.
The constraints were brutal:
128 MB memory limit per Worker invocation
No filesystem -- Workers run in V8 isolates, not containers
Subrequest limits -- everyfetch(), R2 read, and database call counts toward a per-invocation cap
CPU time limits -- you get seconds, not minutes
Files up to 4 GB+ -- meaning ZIP64 is mandatory, not optional
And one more thing: I wanted to avoid egress fees entirely. If I sent R2 data through AWS ECS to build the ZIP, I'd pay AWS egress on every byte. Cloudflare Workers talking to Cloudflare R2 costs nothing in bandwidth. That economic constraint shaped the entire architecture.
Why Existing Solutions Failed
I evaluated every ZIP library I could find for the Workers environment.
zip.js -- Almost Perfect
zip.js supports ZIP64 and streaming. It looked ideal. But internally it chains TransformStream instances using pipeTo(), and Cloudflare's workerd runtime doesn't implement inter-TransformStream pipeTo(). I tried shimming globalThis.TransformStream with IdentityTransformStream and manually implementing pipeTo() in a wrapper. Memory usage exploded. The fundamental issue is that Workers' TransformStream lacks backpressure propagation across chained transforms.
JSZip -- Wrong Model
JSZip buffers the entire archive in memory before producing output. With a 128 MB limit and 10 GB of input files, that's a non-starter. It also doesn't support ZIP64 writes.
fflate -- No ZIP64
fflate is fast and Workers-compatible, but it doesn't support ZIP64. Any archive over 4 GB (or with entries over 4 GB) is out.
The Decision
None of them worked. I needed to write a ZIP64 archiver from scratch, designed specifically for the Workers constraint model.
Architecture: Streaming ZIP64 Without a Filesystem
The key insight is that the ZIP format is sequential enough to stream -- if you make the right tradeoffs.
How ZIP Files Work (The Short Version)
A ZIP file is structured like this:
[Local File Header + File Data] x N
[Central Directory]
[End of Central Directory]
The Central Directory at the end is an index referencing all files. This is why ZIP files can be read without scanning the entire archive -- readers jump to the end first.
For our purposes, this structure is actually a gift: we can stream Local File Headers and file data sequentially, accumulate metadata in memory, then write the Central Directory at the end. The metadata per file is small (filename, offset, CRC32, sizes), so even 5,000 files fit comfortably in memory.
Data Descriptors: Write Now, Fill In Later
Normally, a Local File Header must contain the CRC32 and compressed size before the file data. That means you'd need to read the entire file first to compute these values, then write the header. With a 4 GB file, that's obviously impossible in 128 MB of RAM.
The solution: Data Descriptors (GPBF bit 3). This ZIP feature lets you write placeholder values in the header and append the real CRC32 and sizes after the file data:
[Local File Header (CRC=0, size=0)] -- write immediately
[File Data stream] -- stream through
[Data Descriptor (real CRC, size)] -- write after streaming
Here's the conceptual flow:
async function* streamZipEntry(
filename: string,
sourceStream: ReadableStream<Uint8Array>
): AsyncGenerator<Uint8Array> {
// Write local file header with zeros for CRC/size
// GPBF bit 3 signals "data descriptor follows"
yield buildLocalFileHeader(filename, {
gpbf: 0x0008,
crc32: 0,
compressedSize: 0,
uncompressedSize: 0,
});
// Stream file data, computing CRC32 on the fly
let crc = 0;
let size = 0n;
const reader = sourceStream.getReader();
while (true) {
const { value, done } = await reader.read();
if (done) break;
crc = updateCrc32(crc, value);
size += BigInt(value.byteLength);
yield value; // Pass through without buffering
}
// Write ZIP64 data descriptor with real values
yield buildZip64DataDescriptor(crc, size, size);
}
We use STORE mode (no compression). The data passes through the Worker byte-by-byte with zero buffering beyond the current chunk. Peak memory stays well under 128 MB regardless of file size.
CRC32 Mid-Computation Serialization
CRC32 is computed incrementally as data flows through. But what happens when a Worker hits its CPU time limit mid-file?
CRC32 state is just a 32-bit integer. We serialize it along with the byte offset and resume from exactly where we left off:
interface ZipCheckpoint {
// ZIP state
currentFileIndex: number;
currentFileOffset: bigint;
crc32State: number; // Just a uint32
entryMetadata: EntryMeta[];
// R2 multipart state
uploadId: string;
uploadedParts: { partNumber: number; etag: string }[];
tailBuffer: Uint8Array; // < 5 MB leftover
}
This checkpoint gets serialized to R2 as JSON (with the tail buffer stored separately). When a new Worker invocation picks up the job, it deserializes the checkpoint and resumes streaming from the exact byte where the previous invocation stopped.
5 MB Boundary Buffering for R2 Multipart
R2 (and S3) multipart uploads require each part to be at least 5 MB (except the last one). But streaming ZIP data doesn't naturally align to 5 MB boundaries. A Local File Header might be 120 bytes. A tiny file might be 2 KB.
The solution: a tail buffer. We accumulate bytes until we hit 5 MB, then flush a part:
async function flushToR2(
chunk: Uint8Array,
state: MultipartState
): Promise<void> {
// Append to tail buffer
state.tail = concat(state.tail, chunk);
// Flush when we have enough for an R2 part
while (state.tail.byteLength >= PART_SIZE_MIN) {
const part = state.tail.slice(0, PART_SIZE_MIN);
state.tail = state.tail.slice(PART_SIZE_MIN);
const uploaded = await r2.uploadPart(
state.uploadId,
state.nextPartNumber,
part
);
state.uploadedParts.push({
partNumber: state.nextPartNumber,
etag: uploaded.etag,
});
state.nextPartNumber++;
}
}
The tail buffer is always under 5 MB, so memory stays bounded.
Checkpoint/Resume: State Serialization to R2
Workers can die at any point -- CPU limit, subrequest limit, or infrastructure issues. The system is designed around this assumption.
The "Checkpoint Authoritative" model:
- Stream data, flush R2 parts, accumulate metadata
- At a safe stopping point (e.g., every 128 MB of data processed), serialize the full state to R2
- Update the database with the checkpoint reference
- If the Worker dies before a checkpoint, we replay from the last checkpoint
The checkpoint is the single source of truth. Any R2 parts uploaded after the last checkpoint are treated as non-existent on resume. This makes the system crash-safe without distributed transactions.
Worker Instance 1:
[stream 128MB] → checkpoint A → [stream 128MB] → checkpoint B → [dies]
Worker Instance 2 (resume):
[load checkpoint B] → [stream from where B left off] → ...
A Cron-based monitor watches for stalled jobs (Worker died without updating status) and spawns new Worker instances to resume them.
Central Directory: The Finale
After all files are streamed, we write the Central Directory and ZIP64 End of Central Directory records using the metadata we accumulated:
async function* finalize(
entries: EntryMeta[]
): AsyncGenerator<Uint8Array> {
const cdOffset = currentOffset;
for (const entry of entries) {
yield buildCentralDirectoryHeader(entry);
}
const cdSize = currentOffset - cdOffset;
// ZIP64 End of Central Directory
yield buildZip64EndOfCentralDirectory(entries.length, cdSize, cdOffset);
yield buildZip64Locator(cdOffset + cdSize);
yield buildEndOfCentralDirectory(entries.length, cdSize, cdOffset);
}
The whole ZIP never exists in memory at once. It flows from source URLs, through the Worker, into R2 parts, and gets assembled into a complete file -- all without ever exceeding a few megabytes of RAM.
Results
This architecture has been running in production as the backend for eazip.io:
- 550K+ files processed
- 10 TB+ total data archived
- Up to 5,000 files and 50 GB per job
Zero egress fees (Workers + R2 = no bandwidth charges)
Peak memory well under 128 MB regardless of job size
The checkpoint/resume mechanism means jobs survive Worker restarts gracefully. A 50 GB ZIP that takes many Worker invocations to complete will checkpoint and resume automatically, with no manual intervention.
What I Learned
The ZIP format is more streaming-friendly than it looks. Data Descriptors were designed in 1993 for tape drives, but they solve exactly the same problem we have in serverless: you can't seek backward.
Serializable state is the key to serverless resilience. If you can serialize your entire computation state to a few kilobytes of JSON plus a small buffer, you can resume anywhere, on any instance, after any failure.
Constraints breed creativity. I never would have built this architecture if I had a 16 GB VM with a filesystem. The Workers limitations forced a design that's actually more resilient and cost-efficient than the traditional approach.
Egress fees are a hidden tax on cloud architectures. Keeping compute and storage on the same provider's network isn't just a performance optimization -- it's an economic one. A 10 GB ZIP downloaded 100 times would cost roughly $90 in AWS egress. On Workers + R2, it costs $0.
Try It
If you need to create ZIP files from remote URLs without managing servers, temp files, or cleanup jobs, eazip.io wraps all of this into a single API call:
curl -X POST https://api.eazip.io/jobs \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"files": [
{"url": "https://cdn.example.com/report-q1.pdf"},
{"url": "https://cdn.example.com/report-q2.pdf", "filename": "Q2.pdf"}
]
}'
There's a free tier (no credit card required) and the documentation covers the full API.
Have questions about the ZIP format internals or the Workers architecture? Drop a comment -- happy to go deeper on any part of this.
SOCIAL SHARE CARD GENERATOR