The Gap Between Working and Robustness Is Bigger than I Thought
Making netplay for a turnbased game in Godot using its built in networking features. This is also a great opportunity to rubber duck at an audience
So the easiest way to do something online with Godot by far is to simply establish a multiplayer connection (pretty straightforward if you're using a VPN like Radmin or are capable of forwarding ports, establishing connections otherwise is a bit more work), synchronize the states of the two games, and then directly call each peer's logic directly via RPC (That is, Remote Procedure Calls). I recently built the capability for users to play Puzzle Touchdown in this manner over the internet and it works well enough that in low-latency environments, you can play games end-to-end without a problem, which is pretty huge for me, as this kind of task is one I am doing entirely on my own, for the first time.
However, while it does work in ideal conditions, players often won't be in those. In fact, players will likely expect the game to function in pretty severe network conditions even at the cost of diminishing the play experience. This would normally still be fine for a turn based game like Puzzle Touchdown, but unfortunately, there are downstream consequences of operating game logic directly via RPC, because (in theory) they could be called in a context other than the state function they were generated from.
By default, Node.rpc will let you know if your connection is still good at the beginning of your function call, and then send it off to the receiving peer. However, the window for a "good connection" is 8 seconds, so you could in theory be blasting various RPCs in a row without any particular knowledge of your opponent's state. The "reliable" transfer mode for RPCs ensures that there are many retry attempts across those 8 seconds, and will also ensure that the receiving peer performs reliable RPCs in the order that they were sent, so the programmer in this case does not have to sweat with creating loops to retry sending RPCs, or worry that their RPCs will be performed out of order.
However, and this is a huge however, RPCs are consumed immediately as they arrive, and just because they were generated with respect to state boundaries from the host, they may not be performed with respect to state boundaries by the client. So, consider a program flow where the host performs a game action and RPCs that action to its peer. The host awaits on an animation to finish, then enters a state where it can perform another game action and also RPCs that out. With enough latency and instability, the RPCs arrive at the client peer much closer together than they were generated by the host, and what was a conventional program flow on the host becomes a potential race condition for every client.
For Puzzle Touchdown in particular, the game possesses a few sharp state boundaries where play is suspended for a brief minigame. With enough latency, it would not be impossible for a player to get through the big play screen and begin making moves on the board while the animations are still going on the other players screen. However, because some non-interactive minigames like penalties and the big play wheel access the RNG during their animation rather than being accessed directly via RPC, the order of RNG access could be different from the order of RPCs being resolved, creating a desynchronization of game state.
Detecting this desynchronization and repairing the situation would inevitably look like a really shitty hitch or glitch. It would also entail more work and require the two games to communicate even more information over the wire, which may slow down the match. Moreover, as good practice, we try to write software that produces the desired behavior, rather than build a program that does ??? and then guard against everything but our desired behavior.
So if we can't drive game functions directly because we cannot guarantee the nature of our client peer's state or the way that they are going consume the RPCs they receive, what could we do instead? Well, we'll simply have to create a messaging system, where the host (or more accurately, whoever's turn it is) sends the client messages that the client consumes at the appropriate times, so that way functions are only called within the states that they're supposed to, preventing any undesired behavior.
In order to achieve this, we have a few options: we can continue using RPCs, as a turnbased game is inherently low volume and Godot manages many aspects of communication automatically. It also allows us to call any function with basically any kind and amount of data (within reason), which makes it inherently powerful. We can send messages directly using PacketPeer.put_var, or PacketPeer.put_packet, as our multiplayer API inherits directly from that class. If we allow full objects, we could do a hell of a lot, at the cost of potentially opening up remote code execution (scary!) for users.
The main advantage of using methods other than RPC is that they're much faster in exchange for losing the connection checking, message ordering, and automatic retries of RPCs. In a real-time action environment, that might be the only way to enhance performance in poor network conditions, but for our game, small delays are going to appear to the client as their opponent thinking rather than a network issue. We'll probably continue using RPCs for the time being.
So rather than calling game functions directly, we're going to be creating new functions that store messages in the order they're received (I suppose a mailbox is a totally fine metaphor). When a state function is called, if it is not the turn of the current host, the state function will instead look into the mailbox to see if there's a function call or data waiting for it that matches the state function. If there is, it will consume the message and run that function/use that data, and if not, it will simply no-op in the same manner as if your opponent hadn't performed any action. Consuming messages in this manner ensures state boundaries are respected, because the game mutates its state according to the messages it consumes, in the same manner as the player had originally created them, rather than simply performing arbitrary function calls without respecting the client's state. The mailbox itself could be as simple as a dictionary of arrays in the networking global singleton I've written to manage connecting to opponents, with state functions as keys and the arrays containing messages in the order they've been received.
Some aspects of the current system may have to be maintained; because the networking window is 8 seconds, and turns can elapse because time ran out, that means the client could end the host's turn before a message containing the host's move arrived. Therefore, the host still needs to be in charge of their own timer, and they'll let the client know when that time is elapsed rather than the other way around.
That...Should...Do it! I know this is probably a multiday project and I want to know all the details before I commit to it, which is my primary motivation for blogging about it. I hope you enjoyed reading; I usually save these sorts of posts for internal discussion at NEST, especially since I don't have any fun pictures for you today, but I think that discussing this more openly might be really helpful for everyone to understand what my day-to-day actually looks like. Have a good one : )