HOT-STATE Social Gaming · Scale 2016 — 2019

Five Million Players on a Part-Time Budget

A social game backend that stayed boring under sustained load, run by one engineer.

ROLE // Backend architecture and full AWS DevOps

5M+ Registered players
80K Daily actives
1 Engineer on the backend
STACK // Node.jsPythonRedisDynamoDBAWS Auto-ScalingSQSNginx

⬢ Client-anonymised by design. No employer, product or customer is named, and no proprietary code appears here — only the system shape and the reasoning behind it.

Separate what must be correct from what must be fast, then scale them differently. CLIENT EDGE SERVICES STATE ASYNC Mobile Client optimistic UI Load Balancer Stateless API Tier auto-scaling group Session & Progress Social Graph fan-out on read Leaderboards sorted sets Economy transactional Hot State Redis System of Record RDS High-Write Store DynamoDB Work Queue SQS Workers & Cron events, rewards, digests
interface service state async external Separate what must be correct from what must be fast, then scale them differently.

The shape of game traffic

A social game’s load curve is not a web app’s load curve. It is flat, then a push notification goes out and it is eleven times higher for four minutes, then it is flat again. It has a daily reset that produces a thundering herd on a schedule you chose yourself. And its worst-case day is its best-case day — a feature going well is the outage.

Standing up a backend for that at 5M+ registered players and 80K daily actives, as essentially one backend engineer also running the AWS estate, forces a specific kind of discipline: every component you add is a component you will be paged for.

Constraints that shaped the design

Auto-scaling is slower than the spike. By the time an instance boots, joins the group and passes health checks, the notification burst is over. Reactive scaling alone cannot catch it.

Social features change the read shape. The moment players can see friends’ progress, a single profile view becomes a fan-out. Naively implemented, the friend list of your most popular player becomes a self-inflicted denial of service.

Not all data deserves the same guarantees. A leaderboard that is two seconds stale is fine. A currency balance that is two seconds stale is a support ticket, and possibly a refund.

Decisions and trade-offs

Split the state by guarantee, not by feature

This is the decision the whole system rests on. Hot, high-churn, tolerant-of-loss state (sessions, live progress, leaderboards) went to Redis, where sorted sets make a leaderboard a data-structure choice rather than a query problem. High-write, key-addressed, append-shaped data went to DynamoDB. The economy — balances, purchases, anything where being wrong costs money — stayed in a transactional relational store.

The cost is three stores to operate instead of one, and a genuine consistency seam between them. The benefit is that you can scale the fast parts aggressively without ever putting the correct parts at risk. Trying to make one store do all three jobs is how teams end up either slow or wrong.

Keep the API tier stateless and boring

Every instance interchangeable, no session affinity, no local state, health checks that actually test dependencies. This is unfashionable advice precisely because it is not clever. It’s also what lets a scaling group be a real safety mechanism instead of a decorative one.

Scale on the schedule you control, then react to the rest

Since the biggest spikes were self-inflicted — pushes, daily resets, events — the group was scaled ahead of them on schedule, with reactive policies handling only genuine surprise. Paying for warm capacity for twenty minutes is dramatically cheaper than the alternative, and it converts the scariest part of the load curve into a non-event.

Push everything you can behind a queue

Reward grants, event processing, digests, anything a player doesn’t need synchronously — into SQS, out to workers. This does two things: it flattens the spike, because the queue absorbs what the API tier would otherwise have to survive, and it makes the failure mode “delayed” instead of “500”. Players forgive delayed. They don’t forgive missing.

Fan-out on read, with hard bounds

Precomputing every player’s social feed on write is elegant until someone with 4,000 friends does something. Reads were fanned out with hard limits and cached aggressively, which is less pure and considerably more predictable. Bounding the worst case matters more than optimising the average one.

Optimistic client, authoritative server

The client shows the result immediately and reconciles. This hides normal latency completely and means a slow round trip degrades into a brief correction rather than a frozen UI. The rule that keeps it honest: the server is always authoritative, and the client never gets to decide anything that touches the economy.

What I’d do differently in 2026

I’d move the leaderboard and social read paths behind a purpose-built read model earlier — I got a long way on Redis primitives and cache discipline, but the fan-out bounds were tuned reactively rather than designed. And I’d instrument the queue depth as the primary health signal from day one; it is by far the earliest honest indicator that something downstream is unwell, and I arrived at that later than I should have.