A few weeks ago we started initial playtesting for a big bunch of new levels. So I am going to write a bit about how we go about with testing and how we use the observations gathered.

First off, Ron Carmel at 2D Boy has a nice writeup about rules for playtesting that anyone interested in these things should check out.

The How

During the development of the demo, we only did in person testing. We sat the player down at a computer and let them play without us instructing, only quietly taking notes. If they got stuck for a long time we would however give them hints, but our goal was, and is, to not have any kind of hint system so those occasions made it quite clear we needed to redesign the puzzle.

For the new puzzles, we actually break one of Ron’s rules, the one which states that you should always perform tests in person. Even though we do agree that in person is the optimal way, we wanted to make it easier to gather a larger amount of data. So Anders created a very handy feature for the game that can record someone playing the game and then automatically upload it to a server for us to watch.

We get this replay data per level and not per play session. The advantage of this is that the player does not need to play in one sitting, but can return any time to play some more just as they would do with any other game. Another advantage is that they do not get the stress of having someone watching over their shoulder. I have found in previous in person tests that some players get performance anxiety as I watch them.

Now, the big disadvantage is that we can not observe the actual player. You can usually notice boredom or frustration quite easily (or enjoyment) while watching someone. Also you cannot ask them questions, like when you want to know what they are thinking about a certain puzzle.

Processing

When observing playtests there are generally three types of issues we see; bugs, level design issues and game design issues. We usually write these down as observations while watching someone play or seeing a replay. Bugs are put into our issue tracking on GitHub.

Bugs are just straight up things that do not work as intended. The player getting stuck in geometry, a button that doesn’t work and so on. There isn’t much discussion regarding these, except if the way to fix the bug somehow affects other things.

Level design issues are usually specific to only a certain puzzle, level or possibly to the world that level belongs to. One example from the second level in the demo was that players did not quite see where to proceed at the top part so I moved down the ice shelf so that it was clearly visible even if you did not jump.

In general, we do not consult each other as much when fixing level design issues as these do not affect other things.

A game design issue is pretty much anything that is not specific to a certain part of the game. One example is that after the development of our prototype and additional test levels, we decided that the game should be focused mainly on puzzles and not platforming. Basically this meant that we removed challenges that were based on your ability to navigate rather than figuring out a puzzle. Another more simple example could just be the jump height or movement speed of the player character.

For game design issues we always have a discussion, since the effect these changes have on the level design are usually quite substantial. Sometimes we fight about it for a bit, but so far have reached agreement by good arguments for our standpoint, since we are both equally involved in the game. In my experience, people in charge at places with hierarchy do not always care about giving good arguments for their decisions.

If our methods are indeed good or not, is yet to be proven by the finished game.