The arena has to outlive the process
For the first month of growordie, every time I shipped a fix I committed a tiny act of vandalism. The deploy would kill the server, and with it the entire arena: everyone mid-run got booted, and worse, their run never passed through the death screen, so the score they'd fought for was never even saved. I was punishing my most engaged players for the crime of being online while I was improving the game. This is how I stopped doing that, and why the fix is a boring one built out of parts I already had.
The short version: the live state of the world now survives the code I deploy on top of it. It takes a quick trip through Postgres between the old process and the new one, and your browser quietly reconnects and picks its snake back up. If you want the wider tour of the server this all lives inside, there's a write-up on the whole architecture. This one zooms in on the deploy.
- 01 The problem: one machine, all of it in RAM
- 02 The insight: state has to outlive the code
- 03 The sequence, step by step
- 04 The reconnect: rebind, don't respawn
- 05 The honest asterisks
- 06 What real zero-downtime looks like
01 · One machine, all of it in RAM
growordie is an authoritative server, which is the honest way to build a game where people can lie to you. Your browser doesn't decide anything; it just sends "I want to turn left," and the server runs the real simulation 30 times a second and streams the world back. I like this design. But it has a consequence that took me a while to feel in my gut: the truth of the game, every snake's position, length, body, kills, all of it, lives in plain JavaScript objects, in the RAM of exactly one process, on exactly one small machine. There is no second copy. The Map of snakes is the world.
That's completely fine while the process is alive. It's a catastrophe the instant the process dies. And a deploy is me killing the process on purpose. Fly sends the old machine a signal, the machine stops, that Map evaporates, and everyone connected to it is standing on a floor that just vanished. Two separate things break at once, and for a while I only noticed the loud one.
The loud break is obvious: the arena resets. Everyone's snake is gone, the socket drops, you're staring at a dead screen. Annoying, but you could imagine shrugging it off. The quiet break is the one that actually made me feel bad. A run's score only gets written to the leaderboard when the snake dies, through the normal death path. A deploy doesn't kill your snake, it kills the whole server out from under it, so your run never reaches that code at all. You didn't die. You were disappeared. Your 60-meter run, the best one you'd had all day, simply never happened as far as the database was concerned.
02 · State has to outlive the code
Once I framed it as "the world dies when the process dies," the shape of the answer was almost embarrassing. The world can't be allowed to depend on the process staying alive. It has to be able to step outside the process I'm about to kill, wait for the new one, and step back in. The code is the disposable part. The state is not.
I did briefly go down the fun rabbit hole. The proper answer to "keep game state alive across a code swap" is often the BEAM: Elixir or Erlang, with the world living in a supervised process or an ETS table that a hot code reload can slide underneath. I genuinely considered a rewrite. I talked myself out of it for two honest reasons. First, my ceiling in this game isn't CPU, it's bandwidth, the cost of streaming everyone the world, and switching runtimes does nothing for that. Second, BEAM hot code reloading is a beautiful trick that is also famously fiddly to get exactly right in production, and "fiddly, in production, on the thing that must not break" is not a trade I wanted. I don't need the world to survive a reload in place. I just need it to survive being written down and read back.
And I already had the place to write it down. Postgres is right there, holding accounts and the leaderboard. It's durable, it's shared, and crucially it's a thing that keeps existing while both the old machine and the new machine are alive at the same moment. So that's the whole trick: at shutdown the old process serializes the human snakes into one row in Postgres, and at boot the new process reads that row back. Postgres is the stepping stone the arena hops across, from the dying process to the fresh one.
the world steps outside the process· serialize · store · rehydrate
The arena is just objects in RAM, so on its own it dies the moment the process does. The fix is to let it hop out first: the old process writes the live human snakes into a single arena_snapshot row, exits, and the new process reads that row back before it opens the doors. Postgres is already wired up and it happily outlives both machines, so it makes a perfect stepping stone. No new runtime, no hot reload, just write it down and read it back.
The snapshot is deliberately small and deliberately human: bots aren't in it, they just respawn fresh on the new machine. The only thing worth carrying across a deploy is the state a person is emotionally invested in, their own run.
03 · The sequence, step by step
Here's the whole handoff, in order. It starts the instant Fly wants the old machine gone. My fly.toml asks Fly to send SIGINT (then SIGTERM) and, importantly, to wait before pulling the plug for real:
kill_signal = "SIGINT"
kill_timeout = "30s" # give the flush room to finish before SIGKILLThe default kill window is only a few seconds, which is not enough time to safely write everyone's run to a database. Thirty seconds is comfortably more than I ever need, and it's the outer wall around an inner safety valve I'll get to. When that signal lands, the process does five things, in this exact sequence:
1. Freeze the simulation. A single flag, shuttingDown = true, flips on. The 30 Hz step loop checks it and stops advancing the world; new joins and new spawns are turned away. Freezing first means nothing can change underneath me while I'm trying to write it all down. The world I snapshot is the world as it was the moment the signal arrived.
2. Bank every live run, in one transaction. Every human currently playing gets their run finalized exactly the way a normal death would finalize it: floored length, seconds alive, kills, distance. The naive way to save a full arena of them is a loop of awaited writes, which is slow and, worse, might not finish inside the window. Instead it's one batched transaction, bankRunsBatch, a single multi-row insert plus a single set-based update. A flush of a whole arena of humans costs one commit, not a thousand round-trips.
3. Snapshot the humans. Those same human snakes get serialized, head position, angle, length, the whole body as rounded integer points, plus the cosmetic and account bits, into one JSON blob written to the arena_snapshot table. This is the copy that will become live snakes again on the other side.
4. Tell the clients. Every connected browser gets a tiny {t:'updating'} message so it can show a calm "updating the arena" note instead of a scary error when the socket drops a beat later.
5. Exit clean. The process closes the server and exits 0. And wrapped around all of this is the safety valve: if the flush ever wedged on a stuck database, a 20-second timer forces the exit anyway, so a bad night for Postgres can never hang the whole deploy. Correctness first, but never at the cost of getting stuck.
Meanwhile Fly has already booted the new machine. Before it accepts a single join, it runs rehydrateFromSnapshot: it reads that snapshot row, and if it's fresh (younger than 90 seconds, so a snapshot left over from an old crash can't resurrect a stale arena), it rebuilds each human as a detached snake, then deletes the row. Detached is the clever little state in the middle of all this, and it's worth being precise about: the snake's body is fully back in the world and back in the collision grid, so it's a real, solid obstacle you can crash into. But it has no socket attached (ws = null) and the step loop skips right over it, so it doesn't steer, move, or grow. It's a frozen statue of a snake, waiting for its owner to come back and bring it to life.
the deploy handoff, in order· freeze · bank · snapshot · rehydrate · rebind
Reading left to right: the old machine catches the signal, freezes so nothing shifts under it, banks every run in one transaction, writes the snapshot, tells the clients, and exits. The state travels down into Postgres and back up into the new machine, which has already booted and rehydrates the humans as detached snakes before it opens for joins. Your browser, meanwhile, saw the socket close, put up a quiet overlay, and is retrying every 1.5 seconds. The moment the new machine is accepting joins, your reconnect lands and you rebind.
The part I find satisfying is the overlap in the middle. The new machine is booting and reading the snapshot while the old one is still exiting. Postgres holds the state steady in the gap between them, so there's a handoff instead of a hole.
04 · The reconnect: rebind, don't respawn
Now the other half, the one that happens in your browser. The client has one job here and it does it without you: when the socket closes, it waits 1.5 seconds and reconnects, on a loop, until it gets through. While it's waiting it shows that soft "updating the arena, reconnecting" note instead of a crash. You don't press anything. As far as you can tell, the game hiccuped for a couple of seconds and came back.
When the reconnect lands, your browser rejoins with the name and token it's had in local storage all along. And this is where the whole thing either pays off or falls apart, in one branch on the server. When someone joins with a name that's already live in the arena, the naive move is to reject it (name taken) or, worse, spawn them a brand new baby snake. Either one throws your run away. Instead, the server looks closer: if the snake holding that name is detached (no socket, owner mid-deploy) and the token checks out, then this isn't a stranger, it's you, coming home. So it re-attaches your socket to the snake that's already sitting there, clears the detached flag, and you are back in your own body, exact size, exact position, exact trail. No respawn. A rebind.
a match is you coming home· handleJoin · reconnect rebind
The frozen detached snake is sitting in the arena the whole time, a lethal obstacle to everyone else but doing nothing itself. When your reconnect arrives with the right token, the server proves it's really you and just plugs your socket back into that exact snake. You resume at 250 meters, not at 14 segments. The faded, crossed-out row is the alternative every wipe used to hand you: a shiny new nothing.
A wrong token gets none of this, of course. It's rejected as a name clash, and the detached snake stays put, waiting for its real owner. The rebind is the one door, and the token is the only key.
And if you never come back? Say you were on the subway and closed the tab. The detached snake can't sit there forever pretending to be a wall. A reaper sweeps every few seconds and, after about 55 seconds with no owner, quietly removes it, frees the name, and clears its body from the grid. There's no death screen and no kill in the feed, because there's nothing to report: your score was already banked back at the shutdown, before the snake was ever detached. The reaper is just tidying up an empty shell.
05 · The honest asterisks
I called this a zero-downtime deploy and I want to be straight about the asterisks, because there are a few and they're the interesting part.
It is not literally zero downtime. It's a brief freeze, call it 2 to 3 seconds, then you're back. The window is however long it takes the new machine to boot and rehydrate plus the time for your client's next 1.5-second reconnect to land. What it is not is a wipe. You keep your run and your place in the world across a blink, rather than losing both across a reset. I'd rather describe that honestly than call it something it isn't.
Your score is frozen at the moment of the deploy. This one is a deliberate trade and it's worth understanding. When I bank your run at shutdown, that number is locked in. The snake you rehydrate into is flagged as already-banked, so it can never be counted a second time, not when you reconnect, not when you eventually die, not when the reaper takes it. That guarantee, never double-count a run, is the one I care most about, because a leaderboard that inflates is a broken leaderboard. The cost is that if you keep growing after a deploy and beat your own record in that same continued run, the new peak won't reach the board. Your run continues for the fun of playing it; your score was settled at the deploy. I think that's the right side to err on.
Two deploys inside 90 seconds is handled; beyond that, the snapshot is ignored. The snapshot carries a timestamp and the boot only trusts one younger than 90 seconds. If I ship twice in quick succession, the detached snakes get re-snapshotted at each shutdown (already-banked and all), so they ride through both. But if a snapshot is ever stale, left over from a machine that died an hour ago, the new boot refuses to resurrect a ghost of an arena from another era. And critically, even in that worst case, nobody loses a score: the runs were banked at shutdown regardless. The snapshot is the nice-to-have; the bank is the guarantee.
None of these are bugs I'm hiding. They're the shape of the compromise, and the compromise buys me the thing that actually mattered: I can fix a bug at 9pm on a busy night without robbing the people who showed up to play.
06 · What real zero-downtime looks like
The freeze bugs me a little, and I know where the honest fix lives. It isn't a runtime rewrite. It's sharding: more than one arena room, each on its own machine. The instant there's more than one room, a deploy becomes a rolling thing you get almost for free, drain and update one room while the others carry the crowd, and there's never a moment when everyone is offline together. The leaderboard is already global in Postgres, so it doesn't care which room you played in; it never did. Sharding is the natural next chapter of this exact same design, not a repudiation of it.
And I'm staying in Node. The BEAM was tempting and it's genuinely the right tool for some shape of this problem, but my bottleneck is bandwidth and my state fits comfortably in one small machine's RAM with room to spare. The snapshot-and-rebind dance I built is simple, it leans on infrastructure I already run, and I can reason about every step of it at 9pm when it counts. Boring, on the thing that must not break, is a feature.
If you want to see the world this deploy is so careful not to drop, the architecture write-up is the place to start, and honestly the best way to test it is to go play and hope you don't notice the next time I ship.
The growordie team