Fast and Durable: how Redis and Postgres split the work in a game backend

Design
1

I wrote up the persistence layer of a multiplayer, turn-based game backend I’ve been building in Scala (Pekko, Cats Effect, Doobie, redis4cats), evolving the design one problem at a time: everything-in-memory → Postgres as the single source of truth → Redis taking on four separate jobs (a write-through cache, a lobby/chat store, TTL-based expiring stores for tokens and rate limits, and a pub/sub analytics bus).

It’s written for a general backend audience, with some Scala/Cats-Effect detail along the way — the write-through cache as a repository decorator, and using pipeToSelf to bridge an IO save into a single-threaded actor without blocking it or racing on its state. The whole service is open source.

Happy to answer questions or hear where you’d have made different calls!

4 Likes
2

That’s a nice article, I enjoyed reading it.

I took a similar approach with two projects in the past - one was some proprietary code that started as a purely in-memory implementation in order to get something out into production, get the delivery pipeline and CI set up and understand what a working system would look like.

It then used Hazelcast in an ad-hoc cluster (initially just one node; again, the architecture was rolled out incrementally) for serving things quickly - in effect that was playing the same role as Redis in your project. I used Hazelcast because it was there in the company stack and did just what I needed it to do, although a bunch of other technologies would have been just as good.

We had a backend durable persistence layer that was a wrapper around Cassandra. No, this wasn’t big data, but again, Cassandra was a Corporate Approved Thing and it has easier to get set up then say, SQLServer or whatever. It was a thoroughgoing PITA to work with and my learning point from that would be to use SQLServer, or Oracle, or CockroachDB, or Postgres, or anything else (bar HBase which I’ve had ‘fun’ with too in another life).

Overall, it was a success. Yay!

The other project was an open source project for bitemporal CQRS, again, that started with a purely in-memory implementation which was kept unchanged as a reference implementation, with a second implementation based on Redis (and a lot of shared code). That in turn begat a much more efficient in-memory implementation that sought to get past the problems with traditional CQRS event replaying, and that led to an attempt to add a durable data storage backend to that efficient implementation as a fourth implementation!

In the end I deemed that fourth implementation a failure - it worked, passed its tests, but had problems with performance due to the approach used for persisting data that were never entirely resolved (the storage was completely opaque regarding data maintenence too, unlike your average SQL approach).

I probably could write a fifth implementation layered on top of some SQL database, but I’m retired now. :cat_with_wry_smile:

It’s still out there if anyone wants the glory!