Building a Leak-Safe gRPC Frame Decoder on Reactor Netty

This is the second article in my explains why I chose to build the runtime directly on Reactor Netty and where its compatibility boundary sits. This article moves one layer down into the Stage 1 protocol implementation: the frame decoder that every RPC shape relies on.

gRPC protobuf messages are not written directly as raw bytes into HTTP/2 DATA frames. Every message starts with a five-byte envelope:

byte 0      bit 0 indicates compression; bits 1-7 must be zero

bytes 1-4   unsigned big-endian payload length

byte 5..n   protobuf message, or its compressed representation

Encoding this envelope is straightforward. The difficult part is decoding it without assuming that one input buffer contains one complete frame. HTTP/2, TCP, and Reactor Netty do not promise that buffer boundaries will line up with gRPC message boundaries.

This post describes the Stage 1 protocol layer. The project has since progressed beyond it, but the ownership and bounded-decoding rules introduced here remain the foundation for the later transport stages.

Encoding Must Define Ownership

The contract of for the executable cases.

Cancellation Must Release Undelivered Data

Suppose one source ByteBuf contains three messages: one, two, and three. The downstream requests two messages and then cancels. The test must assert not only the values it received, but also that the source buffer was released:

StepVerifier.create(GrpcFrameCodec.decode(Flux.just(source)), 0)

        .thenRequest(1)

        .assertNext(message -> assertMessage(message, "one"))

        .thenRequest(1)

        .assertNext(message -> assertMessage(message, "two"))

        .thenCancel()

        .verify();



assertEquals(0, source.refCnt());

That assertion is more important than a happy-path content check. Network code often behaves correctly under normal completion; leaks tend to appear during cancellation, size-limit failures, truncated frames, or competing terminal signals.

Cancellation can also arrive before a complete message exists. In that case there is no decoded value for the subscriber to release, so the decoder itself must release the partially accumulated source buffer:

@Test

void releasesPartialFrameInputWhenCancelled() {

    ByteBuf partial = Unpooled.buffer(8)

            .writeByte(0)

            .writeInt(16)

            .writeBytes(new byte[]{1, 2, 3});



    StepVerifier.create(

                    GrpcFrameCodec.decode(

                            Flux.just(partial).concatWith(Flux.never())),

                    0)

            .thenRequest(1)

            .thenAwait(java.time.Duration.ofMillis(10))

            .thenCancel()

            .verify();



    assertEquals(0, partial.refCnt());

}

The full executable case is , each with a specific meaning for client error handling and future retry policies. GrpcStatus is a record containing a code and a human-readable message:

public record GrpcStatus(Code code, String message) {

    public enum Code {

        OK(0), CANCELLED(1), UNKNOWN(2), INVALID_ARGUMENT(3),

        DEADLINE_EXCEEDED(4), NOT_FOUND(5), ALREADY_EXISTS(6),

        PERMISSION_DENIED(7), RESOURCE_EXHAUSTED(8),

        FAILED_PRECONDITION(9), ABORTED(10), OUT_OF_RANGE(11),

        UNIMPLEMENTED(12), INTERNAL(13), UNAVAILABLE(14),

        DATA_LOSS(15), UNAUTHENTICATED(16);

    }

}

Several details matter in practice:

• 
  DEADLINE_EXCEEDED may be returned even after the operation completed successfully. If the successful response crosses the deadline in transit, the client can still observe a timeout.


• 
  UNAVAILABLE indicates a transient failure for which a client may later retry safely; INTERNAL generally describes a server-side bug and should not be blindly retried.


• 
  UNIMPLEMENTED carries the semantic meaning of an unsupported method, commonly surfaced through an HTTP 404 response at the protocol boundary.

Text in the grpc-message trailer uses percent-encoding: printable ASCII characters other than % can pass through, while other bytes become %HH. This allows UTF-8 error descriptions to travel through ASCII HTTP/2 headers safely.

Unknown numeric status codes are mapped to UNKNOWN instead of causing a parse failure. That preserves forward compatibility when a peer adopts a newer gRPC specification.

Timeout: Eight Digits and a Unit

The gRPC grpc-timeout header carries a relative duration, not an absolute timestamp. By the time the server receives a request, part of the caller's original time budget has already been consumed by transport latency.

The wire format is compact: at most eight decimal digits followed by a unit suffix:

100m       -> 100 milliseconds

2S         -> 2 seconds

99999999H  -> roughly 11,415 years (the maximum value)

The six units are H (hours), M (minutes), S (seconds), m (milliseconds), u (microseconds), and n (nanoseconds).

When formatting a Duration, the implementation rounds upward using ceiling division. The encoded deadline must never be shorter than the caller's requested duration:

BigInteger amount = nanos.add(unitNanos.subtract(BigInteger.ONE))

        .divide(unitNanos); // ceiling division

BigInteger avoids overflow during nanosecond arithmetic. The formatter scans from nanoseconds upward and selects the first unit whose value fits within 99,999,999.

Compression: Identity by Default, Gzip Built In

GrpcCompression manages codec registration and negotiation:

GrpcCompression.Registry registry = GrpcCompression.Registry.builder()

        .add(GrpcCompression.GZIP)

        .build();



// Produces grpc-accept-encoding: gzip

String advertised = registry.advertisedEncodings();

identity is always implicit and appears first in the registry. The codec interface has only three operations: a name, byte-array compression, and bounded byte-array decompression.

Gzip decompression reads in 8 KiB chunks and uses Math.addExact() while accumulating the output size. It can stop immediately after exceeding maxDecompressedSize, and arithmetic overflow cannot silently wrap the counter. A 100-byte gzip payload can expand to gigabytes, so decompression-bomb protection is part of the protocol contract rather than an optional optimization.

GrpcMethod: One Description for Four Cardinalities

Each RPC is described by one GrpcMethod record:

var method = new GrpcMethod<>(

        "testing.InteropTestService", // full service name

        "Unary",                      // method name

        GrpcMethod.Cardinality.UNARY,

        new ProtobufMarshaller<>(TestRequest.parser()),

        new ProtobufMarshaller<>(TestResponse.parser()));



method.path();           // /testing.InteropTestService/Unary

method.fullMethodName(); // testing.InteropTestService/Unary

The Cardinality enum exposes singleRequest() and singleResponse(). The transport uses those flags to insert single() at the API boundary, turning cardinality violations into explicit errors instead of silently dropping values.

ProtobufMarshaller: Serialization Does Not Own the Input

ProtobufMarshaller wraps a protobuf Parser<T>:

public ByteBuf serialize(ByteBufAllocator allocator, T value) {

    var result = allocator.buffer(value.getSerializedSize());

    return result.writeBytes(value.toByteArray());

}



public T deserialize(ByteBuf message) {

    ByteBuffer bytes = message.nioBuffer(message.readerIndex(), message.readableBytes());

    return parser.parseFrom(bytes);

}

The important contract is that deserialize reads through a NIO ByteBuffer view. It does not move the input reader index and does not release the input. Ownership remains with the caller, allowing the frame decoder's doFinally(release) to manage the buffer lifecycle uniformly whether parsing succeeds or fails.

Stage 1 Exit Criteria

The protocol module does not need Reactor Netty on its classpath. That dependency boundary is itself part of the verification. The Stage 1 exit criteria are:

• every header and payload split point decodes correctly;


• cancellation leaves no unreleased ByteBuf (refCnt assertions);


• metadata preserves order, supports binary values, and enforces its size limit;


• unknown status codes do not throw;


• all timeout units parse correctly, formatting rounds upward, and arithmetic is overflow-safe;


• gzip decompression limits reject bombs;


• the marshaller does not change the input buffer state.

These tests use JUnit 5 @TestFactory and DynamicTest for parameterization, together with Reactor Test's StepVerifier for asynchronous behavior. The protocol layer is the reason the later transport stages can focus on HTTP/2 lifecycle instead of rediscovering framing and ownership rules.

The complete implementation is in , , .