Put simply, playtesting is to design what QA is to engineering. It is not there so the testers can tell you what to do (although collecting feedback is a side benefit) but so that you can observe how your design and implementation performs with real data. Much has been said about how to do playtesting properly already – actually, Juha covered our own process a while back – so instead I will go into the technical details behind our own system. Much of it will be obvious, but hopefully there will be something to learn for someone.
For Backworlds, we use a simple form of input journaling to record data as players play through the game. Simply put, we capture all input data in a stream and make the input handler grab the data from that stream instead of mouse and keyboard when we want to play it back. Some of you may follow the development of the Witness and might have read the post on playtest data in which Jonathan Blow argues against input journaling. He makes a good point and it is important to consider what kind of data you need before you decide on a playtest system – in our case we need to closely monitor how well players get through the trivial platforming parts of the level, so a full playback makes sense.
Our playback data is stored per level and consists of a header and a command stream. The header mainly consists of information about the starting state of the player – position, direction, velocity and some other data. It also contains a hash of the level data so we can determine if the level itself changed from when the playback was recorded, and a random seed generated from the current time every time a level is started – originally this was put in a Mersenne Twister but I swapped it to a Multiply-With-Carry since it was simpler. There usually isn’t any need to write your own random number generator though. The important thing is, of course, that it needs to be completely deterministic so every call that affects the game needs to get pseudorandom numbers from the same place, and for simplicity’s sake only called from one thread. Finally, the playback header contains a small chunk of system information for reference – CPU type, GPU type, game settings and the like. Not strictly necessary to replay the playback, but helpful in case it is broken.
Besides pseudorandom numbers the other thing requiring tight control is time. We have a global object to manage time – in a normal case this will simply use the performance timer and give you the absolute time, but when recording playbacks it will increment the time with 1/60th second every frame to keep it deterministic. This means that the game also has to run in 60 fps to give a realistic experience – note that it is not enough to simply enable VSync when creating the rendercontext to do this as GPU drivers have the option to force it to always off. For Backworlds, we turn on VSync and time each frame – if 1/60th of a second has not passed between frames, we induce the delay manually.
When recording, we record the message stream as a set of 16-bit unsigned integers – the high four bits represent message types and the low 12 the identifier – such as key type or mouse position. For simplicity’s sake we record mouse position every frame and let that signify the start of a new frame – we then add a message every time a key or mouse button is pressed or released. In addition, we have messages for silent keydowns – these cause the key to be registered as pressed without creating a KeyPressed event and we use them to set up the initial state of the keyboard. If, for instance, the player was holding down the ‘walk left’ key when entering a level it would not register an event when recording it, but we still want the player avatar to be walking left in the recording. We also store player position every 300 frames to make sure that the playback did not desync due to some bug.
Speaking of bugs, it is important to test playbacks thoroughly before sending test builds to people – if someone tries your game for hours and you then have nothing to show for it, you have wasted their time. Some things to remember;
- Always write the commandstream directly to file and always flush the file after every write. Should the game crash, you will still have the data leading up to it.
- Be very careful to initialize all data so there are no unknowns. If you tend to be sloppy with this, consider using a third-party tool to help you detect problems – I have no idea about free tools, sadly, but I have used and liked Coverity tools.
- Make sure that you can create the playback file – error-check writes and do not let the player start the game if they fail. Again, you would be wasting their time.
… That’s all for now. Next time I’ll go over how the playback ends up in our hands.