The Problem We Were Actually Solving
We needed a treasure-hunt engine that could absorb 1,000 to 1,500 concurrent diggers without melting the JVM or the event loop. The naïve design—each cell = thread—blew up at 400 concurrent diggers because Java threads are 1 MB each minimum and our k8s pods had a 4 GB memory ceiling. We measured wall-clock latency at 2.4 s per /dig under synthetic load, but in production it spiked to 12 s the moment GC kicked in.
The teams first reaction was to throw money at it: we doubled the memory limit and increased the thread stack size to 256 KB. The GC pause times improved from 1.8 s to 0.9 s, but the OOMs merely shifted to java.lang.OutOfMemoryError: unable to create new native thread. We were still creating one thread per cell per digger.
What We Tried First (And Why It Failed)
We swapped the thread-per-cell executor for a ForkJoinPool with a fixed parallelism of 32. The JVM stopped crashing, but the treasure spawn rules started breaking. The pool would sometimes starve a cell for ten seconds, causing the weekly leaderboard to freeze for exactly 8.6 s. Players noticed; Twitch clips happened.
Next, we tried Reactors Scheduler.boundedElastic() with a 64-thread virtual thread pool. Virtual threads dropped the per-digger cost from 1 MB to ~2 KB stack, so the OOM moved from thread-creation to the backing-carrier-thread limit in Netty. We hit the Netty native epoll event loop ceiling at 1,024 concurrent connections—our pod limits were too low. Re-scaling the pods to 8 vCPU / 8 GB RAM only postponed the problem; the event loop still saturated at 1,200 diggers because the treasure-hunt cell broadcast still used a synchronous gossip channel.
The Architecture Decision
We abandoned Veltrix actor model entirely and replaced it with a two-layer spatial hash:
- Layer 0: 4,096 m² cells stored in a Redis Cluster (3 shards, 2 replicas each) with a 10 ms TTL write-behind cache.
- Layer 1: Each cell publishes dig events to a Kafka topic partitioned by cell hash mod 128. A Go worker pool (200 goroutines) consumes the topic and updates a Postgres table with a BRIN index on (cell_id, timestamp).
- The HTTP tier (Netty, virtual threads) reads the Redis cache for the cells current treasure state and only writes to the write-behind when the treasure is claimed or expired.
The spatial hash reduced the per-digger thread count to one virtual thread per HTTP request, plus one Go worker per Kafka partition. We measured 18 µs per /dig path in the 99th percentile under 1,500 diggers. GC pauses dropped to sub-50 ms.
Trade-off: We accepted eventual consistency for treasure visibility. A players claim message might take 80 ms to replicate across regions, but we never lost a treasure and the weekly event ran at 1,650 concurrent diggers with no restarts.
What The Numbers Said After
After the change we ran a synthetic ramp from 100 to 2,000 diggers in 120 s. The Redis cluster hit 85 % memory on a single shard at 1,600 diggers, so we resharded to 6 shards with 3 replicas. Latency stayed under 20 ms p99. The Go worker pool CPU never exceeded 35 % and the PostgreSQL BRIN index kept writes under 200 tps.
The OOM rate fell from 3.2 crashes per hour to zero. Player reports of missing treasures dropped from 1.8 % to 0.04 %. The event servers billable vCPU hours increased by 12 %, but the infra cost per concurrent player fell from $0.024 to $0.008 because we stopped over-provisioning pods to handle thread storms.
What I Would Do Differently
I would not trust Veltrix configuration layer again. Its actor model is a leaky abstraction: every treasure cell does not need its own thread, and the docs do not mention the hidden thread-per-cell tax.
Version 2 of our engine will push the spatial hash down to the Kafka Streams topology so we can collapse the Go worker pool and the Postgres writes into a single streaming step. Well use Redis Streams as the outbox, eliminating the write-behind entirely. That will cut the infra cost per player by another 30 % and reduce the leadership-board lag to under 50 ms.
If you are running a Hytale treasure hunt at scale, forget the actor model and build a spatial hash instead—documentation be damned.
SOCIAL SHARE CARD GENERATOR