desync_head Apologies for missing last month’s update – this was partly due to a personal hardship, lack of time due to day jobs and other obligations and – to be quite honest – a lack of things to write about. We have been refining designs for some time now and while it can certainly be interesting to look at individual puzzles and see how we changed them in response to playtest feedback, doing so now would be premature as we are not done with them. Also, it would spoil the puzzles themselves.

This month, I am going to tell you an anecdote about a bug hunt that was somewhat amusing as it illustrates the cascading effects small changes can have on a codebase.

As previously mentioned, we do remote playtests and look at the recorded results in order to learn about how well the design works. In order to keep the size of these playbacks small, we record player input and play it back in the same application rather than record videos. This way we get an exact playback at a minimal size, as long as the game itself is completely predictive – for the latest playtest we found that this was not always the case and some playbacks would intermittently desync.

oh_nooo

Having playbacks desync does not make them entirely useless as the system used to detect desyncs shows the real player position so we will at least get some idea of what the player was doing, but it does mean the amount of information in the playback is severely reduced so it needed to be fixed before subsequent playtests.

As we mentioned in the playback post, the game needs to be completely deterministic to recreate a session from an input stream – when debugging issues with playback desync I tend to start with recording player position in a normal game and a replay until I get a replay that desyncs, then try to log more information and work my way backward to see why the results differ. After a few hours of doing it (this was a particularly hard-to-reproduce desync) it became clear that randomly generated numbers were different in the original game and playback. For most cases this does not matter much since very little of the gameplay is randomized, but the paintbrush itself has a random noise pattern on the edge which can cause physics to end up in  the wrong world. Should this happen so that the player only just manages to pass an obstacle in the original playthrough but hit it in the playback, the player may have kept walking but the playback assumes she was just pressing against the obstacle and the error compounds.

I briefly covered this in the playback post, but random numbers in computers are commonly generated by starting with a random-like seed, like the current time, and applying an algorithm to that seed in order to create a pseudorandom number. To create another number, the algorithm is applied on the first number and so on for each new random value desired – this means that as long as the seed is the same and the exact same random numbers are generated in the same order, they will be deterministic.

(At this point I should mention that this is an oversimplification that only really works if you are using random for seeming noise and not for applications like dice rolls or cryptography where it is important that the numbers generated are unpredictable and evenly distributed. My colleague Ben Deane wrote about modern ways of using randomness in C++ a while back, this is a more thorough analysis)

Anyway, what I finally figured out was that the player avatar’s idle animation was the culprit. A while back, I added a system to have it blink when standing still – after doing so, it would wait a random amount of time (up to 6 seconds) and blink again. This system was not reset on level start so the playback would not know when the first blink occurred – the difference between the real time and the playback time could be up to 6 seconds and if any other random number was requested during that interval (for instance, if the player was painting) it would receive a different number in the random list than it had in the original game, thus causing all the above problems.

This was easily solved by simply resetting the idle animation handler when the new level was loaded, but hitting it gave me pause – most random numbers were for noise in visual effects and it was completely irrelevant whether they looked the same in the playback or not. Conversely, the few random numbers that were actually relevant to the game (such as the initial direction of a randomly moving game object) probably should not be completely random since that would make the game more frustrating for potential players trying to speedrun it. What I ended up doing was create an “Unsafe” random generator for effects and another one for gameplay – this other one would always be initialized with the same seed for a given level so the level acts the same every time you enter it.

… The lesson, for those who skipped to the end, is be very careful about how you are generating random numbers if your game is deterministic, and to consider if you really want randomness or if noisy but predictable would not be the better choice.