Testing and Validation
To justify the feature of the guidebook and ensure it is needed for the game, user-tested was completed on the final result from sprint 2. The testing that occurred was observational testing, which meant that I let the users play the game and recorded what they did and where they got stuck. I tested a total of five people to help establish what is intuitive in the game and what is not.
I started the game for the users, and let them play through the game while I watched. I took notes of what they did, where they got stuck and what they said while playing the game to help establish emotion.
After they finished working through the game and said that they don't think there are any more aspects to explore, I explain how to actually use the game.
From testing five users, only two figured out how to make the main character move, and attack. No users figured out how to or even that you could place buildings. Many of the users just kept on clicking around the screen expecting something to happen. One user went into the shop five times trying to find something useful to help the player move.
From this many of the users did not figure out what the point of the game was, thus, the game is not intuitive. This means that the game does need instructions and a guide to know how to play and win the game.
To best present the information in a meaningful and helpful manner, we conducted user testing on our low fidelity guidebook screen wireframes to understand which format was the most legible, captivating and suitable for the target audience. The following testing plan outlines the tests undertaken and how we measured success to produce the best outcome for users. See this link to view low-fidelity prototypes and more about wireframes. From left to right --> Options 1, 2 and 3 respectively.
The testing undertaken was customer preference testing. Whereby users are given three options to choose from and must rank in order of which they like the most, with 1 being their first preference and 3 being their last preference. After scoring the options, the user must respond with an explanation for their ranking. Preference testing is fundamental to grasping users' dislikes and likes early on.
-
Preference: 2,3 & 1.
Reasoning: "I really like the idea of creating a book as the layout adds character to the aesthetic and ties in the idea of the oceanic adventures out on the sea with those old leather sailing books. So that’s why I chose 2 and 3 as my top preferences and made 1 last as it doesn’t have as much of a special effect." -
Preference: 2,1 & 3. Reasoning: "Just by looking at option 3, I have never seen anything done like that, and I’d say it would be because one it’s hard to produce and two people would prefer to read a flat book than a sentence or a paragraph that ascends upwards. Although it’s creative, I think for game instructions I just want it plainly laid out for me, like given directly. So that’s why I went with 2 as you still get that cool book design but, it’s easier to quickly read the instructions."
-
Preference: 2,1 & 3. Reasoning: "Option two appeals to me the most as it is similar to how I read on my iPad. Which is for those who don’t know a flat top-down view of the text. I guess it works the best as there is no curvature or effects that make it difficult to read, and that is probably why I put 3 last. I don’t mind option 1, but it is a bit bland and I personally appreciate games that make the effort to intertwine their game theme throughout all interfaces."
-
Preference: 1,2 & 3. Reasoning: "I strongly dislike the angled book layout (option 3) as it is hard to read text angled like that. If I were playing a game, I’d find it a pain and almost nauseating to have to tilt my head to read text. For that reason, I prefer the 2D format of both 1 and 2."
-
Preference: 2,1 & 3 Reasoning: "I like the middle option as the layout is the most intuitive. What I mean by this is that I can easily find the exit button, see how to flick through pages and the content on the right-hand side is presented neatly. I ranked the first option second as I also like the simple layout of that option. But I just didn’t enjoy the buttons “next and prev”, I prefer the arrows so I went with the middle one."
After gathering the scores, we found that there was a strong favour towards option 2. Option 2 has a flat view and is made to appear like an old-fashioned novel. The users leaned towards this option as they found it was the perfect mix of integration of the game and legibility. There was a common preference to be able to quickly read the information and return to the game, and not make the process a hassle. Additionally, the option still captured the theme nicely with a lot of the users after the testing saying it would also look great when produced with the pixel design. Therefore, customer preference testing was very beneficial in grasping how users would like the guidebook to be organised and depicted.
A big task for the guidebook was arranging the topics put forward by our studio members in a logical order that a user would expect to view the information. There is no better way than hearing what users themselves have to say by doing some testing. Therefore, we asked users to arrange the topics in how they'd like to view the content which is relative to their first experiences playing the game. Since the topics are subject to the studio's opinions, we gave the user the ability to transform the topics by adding or removing topics too.
Please list in order (1 being the first option, 2 being the second option, and so on…) the order you would like to view the game information from the guidebook. Some motives for ranking could be the importance you associate with the topic or logical timeline or sequence of the game. We also have the option to ‘not include’ a topic; do so by placing this topic name in the red cell. If excluded from the guidebook, please give a valid reason why. And if you identify any elements you wish to add to the topics, please add them in the green cell and provide a reason why and where you’d place them.
See the image below which outlines the feedback from user testing.
The user testing brought to light the irrelevance of certain topics and a common understanding of what users like to know about in a particular order. The majority of users didn't think cut scenes, camera-related adjustments, and NPCs were important enough to be outlined in the guidebook. More specifically, users were perplexed as to why cut scene controls exist and said just to give users the option to skip. And for camera-related adjustments, they should just be found in settings and don't really require a full-on explanation. In terms of the generally agreed-upon ordering of the table of contents for the guidebook, it seems as though users would like to know about the crystal and main character, then who they have to protect themselves from (enemies) when to expect the enemies (night cycle), what to do in the day cycle (buildings, shop etc). A final version of the list will be presented below. However, we have also manipulated the table of contents topics to include suggestions from users, which were generic how-to-play instructions, currency and the final boss level. These were added based on frustrations and pain points experienced by the users when playing the game and not having enough information.
Table of Contents - Revised:
- How to Play
- Crystal
- Main Player (their movements, health, lives, and objectives)
- Day and Night cycles
- Enemies (Generic Enemies and hint to boss level)
- Types of Buildings
- Resource Buildings
- Currency
- Shop resources
- Inventory
To ensure the guidebook contained the correct information and was in line with our fellow studio members, we created a draft guidebook google document which was shared among all teams. Teams were able to manipulate the draft and a change log was created to record the cross-team collaboration.
See the following google doc guidebook or the images below of the document:
For this navigational test, a mixture between the original game and the high-fidelity wireframe was used to increase the realistic feeling of the finished game. The aim was to see how well users found out skeleton layout.
The user must complete the following tasks without any guidance or assistance. The condition that must be satisfied is that the users must not have any prior knowledge regarding the game.
- Try to play the game without a manual or guidance
- Find the help that is available on the Main Game screen
- Search for the game topic that they are confused
- Let users wander the game; test whether the game is already playable without any need of further guidance.
- To know whether the Guidebook icon that is on the Main Game screen is recognizable as a place where users turn to when they are confused and are looking for clarification.
- To measure the clarity of the structured content of the Guidebook
The navigational testing was done on 5 people ranging from age 20 to 25. The results of each task are as follows.
From this task, 1 out of 5 users is struggling to play the game as is without any further clarification needed. The rest of the test participants said they definitely need help in playing and operating the game.
The results from this task are interesting. All of the users know where to get help; they go directly pressing the Guidebook page. They agreed that the button to access the Guidebook is clear and straightforward.
The users pressed into the topics that are available on the "Table of Contents" page. They feel that it is so convenient for them to be redirected after pressing the button. The content of the Guidebook is well-organized and the fact there is a picture that clarifies the description helps tremendously.
From the results of the given tasks, we achieved a 100% task completion rate when it comes to things related to the Guidebook. On the other hand, the game still needs a sort of tutorial, in the beginning, to guide the user on what is the game, how to play it, and how to win it since the test subjects are mainly confused regarding those topics.
To gauge how successful sprint three's end product we conducted user testing (navigational based) to measure how well the user interacts with the feature. The results are helpful in understanding how intuitive an action may be, and ultimately, measures the interface's success.
Throughout the course, team 10 have found navigational-based user testing very useful in understanding the effectiveness of their implementation. Therefore, navigational testing was used on the product of sprint 3 to measure the effectiveness of our implementation with usability. This was strictly an observational test to see if the product was intuitive and simple to use. The following were the tasks performed by users.
- Test 1: Find how to access the achievements page.
- Test 2: Find the number of enemies within the game.
- Test 3: List the currencies used in the game.
-
User 1: All three tests were achieved very quickly, and no issues were experienced. The first test took the most time as the user took more time to flick from page one to the achievements page which is at the end of the guidebook. The user had no issues identifying the number of enemies by navigating by flicking through the pages to the enemies’ page. Lastly, the third task was straightforward enough after attempting task 2, so the user knew to flick through the pages to check out the currency link.
-
User 2: The user did not spend a long time finding the achievements page in task 1. They clearly understood that you could flick back and forth using the arrows on the bottom of the pages, however, tried to use the table of contents as a hyperlink to get to the page. The user must have had a preference for efficiency and wanted to quickly jump to the page using the table of contents. The user easily found the header for enemies and currency respectively which allowed them to complete the testing by flicking through the pages.
-
User 3: In this case, the user did not experience any inconvenience or slower responses. The user sped through the first task, by identifying the arrows to flick through to the achievements page. Then the user swiftly answers tasks 2 and 3. They thoroughly enjoyed the flicking between pages animation and commented that "the guidebook is so engaging; I am a big fan". They also mentioned they recommended placing images above the text across all pages as it looks a bit messy.
From the testing, we have gained insight that two options to access pages within the guidebook should be implemented, these being the arrows and table of contents, to enable flexibility of the users’ preferences concerning navigation ability. Additionally, we will clean up the layout of images and text. In sprint 4, we can make the game juicier by adding a hovering effect on the table of contents to make it more apparent that it is a linking attribute. Overall, all three users quickly completed the testing with ease and had a positive interaction with the end product feature.
JUnit testing was aimed to be completed at the end of this sprint, to ensure that the code does work in the way we want it to. Classes that were tested were guidebookScreen, and guidebookAction.
Both these classes had tests to check if they were null as seen below:
boolean testNull() {
if (actions.getClass() != null) {
return true;
}
return false;
}
@Test
public void notNullTest() {
assertTrue(testNull());
}
From guidebookActions
In the guidebookAction, we also tested to check the page Number should be 0, as the pages were just initialized and should be 0.
@Test
public void pageNumberTest() {
int currentPage = actions.currentPage;
assertEquals(currentPage, 0);
}
Additional tests were also made, but mainly just to check that the event handler was created correctly in the class.