PX-MBT

PX-MBT is a framework for automated player experience testing with emotion pattern verification.

• PX-MBT utilizes a model-based testing approach for test suite generation
• It employs a computational model of emotions developed based on a psychological theory of emotions to model players' emotions during game-plays (more specifically, Ortony, Clore, and Collins' theory/OCC). The theory behind this computational model can be found: in this paper.
• It verifies emotion patterns given by game designers.

Architecture

The picture below shows the general architecture of PX-MBT. There are four main components: the Model-Based Testing (MBT) component for generating tests using e.g. EvoMBT, the Model of Emotions component implements the aforementioned computational model of emotions, an aplib Basic (test) Agent for controlling the in-game player-character, and a set of APIs and scripts labelled as the PX Testing Tool as an interface for a game designer towards the framework.

More information on these components can be found here.

PX-MBT requires some inputs that the game designers/developers need to provide inputs; these are ① in the picture.

px-mbt architecture

PX-MBT offers the following main functionalities:

• Model-based test suite generation. Given a model (in extended finite state machine/EFSM), the Model-based testing component ② generates a test suite consisting of abstract test cases to be executed on the game under test (GUT). It employs EvoMBT and Aplib which together offer the possibility of using various search algorithms (e.g. MOSA) and LTL model checking for generating test cases from a model.

• Agent-based execution. To execute an abstract test case that comes from a model, it needs to be interpreted to concrete interactions with the GUT. This step is also called concretization. PX-MBT uses aplib agents as the executor: the test case is first coverted to goals and tactics for an agent, which in turn converts tactics into primitive interactions with the GUT. Aplib is an agent programming framework specifically for testing computer games, e.g. it comes with a programming model which is suitable for that purpose. Using aplib eases the effort in implementing the concretization layer.

• Emotion trace generation. Attaching the Model of Emotions component to an aplib basic test agent creates an emotional test agent ③, which is able to simulate emotions based on incoming events. Via an 'interface' (also called Environment) the emotional test agent is connected to the GUT. Each (abstract) test case of the test suite is then given to the agent for execution. The agent computes its emotional state upon observing events and records it in a trace file.

• Emotion pattern verification. After a whole test suite is executed on the GUT, functions from the PX Testing Tool can be used to analyze the produced emotion traces, e.g. to verify emotional requirements and to make heat-maps and timeline graphs of the dynamic of the emotions that occur during the tests.

• Data Export. The framework exports a range of artifacts. Test cases are exported in various forms: as text, graphically through dot syntax, in serialized binary format, and in custom CSV formats. Emotion trace files are exported in CSV format, while heat-maps and timeline graphs are saved in PNG format.

Examples/demos

Some examples/demos are provided below. To run them you first need to build PX-MBT (see How to build below).

• A quick example (with a small 2D game MiniDungeon)
• A simple example with a game called Lab Recruits

How to import PX-MBT into your project

PX-MBT is distributed as a software package. In particular, it is not an end application. The typical use case is to import PX-MBT into your own Java project, which would then give you access to functionalities/APIs of PX-MBT.

As PX-MBT is organized as a Maven project, the expected use case is to import it from another Maven project (or a project that uses a build system that can import a Maven project, such as Gradle).

To import PX-MBT into your Maven project:

1. First, add this Jitpack repository in your pom.xml. PX-MBT is distributed from Jitpack. Adding Jitpack repository will make your project to automatically download and install PX-MBT in your own Maven local repository, which is convenient.

    <repositories>
      <repository>
          <id>jitpack.io</id>
          <url>https://jitpack.io</url>
      </repository>
    </repositories>

2. Add the dependency to PX-MBT in your pom.xml, with the right version. In the snippet below the version is 2.0.2.

    <dependency>
        <groupId>com.github.iv4xr-project</groupId>
        <artifactId>eplaytesting-pipeline</artifactId>
        <version>2.0.2</version>
    </dependency>

The version number must be a git-tag in PX-MBT Github repository.

How to build

Building is needed if you want to run the demos, or if you clone PX-MBT to extend it for your own purpose. PX-MBT is written in Java. It requires Java-11 or higher. The project is Maven-based.

• Clone or download a zip of this project and unzip it. Then, go to the project root.
• To build just do mvn compile from the project's root. This will automatically download all its dependencies and then compile the project.
• If you want to install the project into your local Maven repository, do mvn install -DskipTests.

Some of the packages used by PX-MBT are distributed from jitpack. If some fails to download (e.g. maybe jitpack site is down) you can install it manually. See the instructions here.

What is system requirement of PX-MBT?

PX-MBT classes and methods would run on any OS (Linux, Windows, Mac, etc) that runs Java-11 or higher. To run PX-MBT demos: the Lab Recruits demo needs either Windows or Mac. The MiniDungeon demo can run on any OS.

The target game (the game on which you want to use PX-MBT for your own project) will have its own system requirement to run, but this is not related to the requirement of PX-MBT itself.

What game-architectures can be targeted?

Games written in Java would be easiest to target, as PX-MBT is also written in Java. Other architectures can be targeted through a socket connection. This can be achieved by adding server-side program at the game-side that accepts command from a PX-MBT test agent and translates the command to e.g. player's keyboard command. The server-side program should also able to send back observation on the game state back to the agent. More on this is explained in the Documentation section below, on the topic Building the interface to connect PX-MBT to your own computer game.

Documentation

The document below explains the general idea of model-based testing of a game, and the main APIs to do it:

• Modelling a game with an EFSM, and model-based testing.

To use PX-MBT, an 'interface' connecting it to your game under test is needed. The documents below provide guidelines on how to implement such an interface. This would take some effort; but it is a one-off effort, after which you will be able to use the interface for further automated testing the game.

• Building the interface to connect PX-MBT to your own computer game.
• Concretization layer

The documents below explain to generate tests, to run them, and then to perform PX verification on the resulting execution traces.

• Preparing PX testing
• Running the test cases (and producing emotion traces)
• PX verification and other analyses

FASE-23 Study

This is a study conducted to assess the feasibility and performance of PX-MBT approach an a 3D game called Lab Recruits. The artifact of this study is available, containing its dataset and a replication package:

• PX-MBT FASE-23 experiment replication-package and dataset. A copy of the dataset is provided in this project as well, in ./FASE23Dataset.
• Instructions
• The paper describing the experiments: Model-based Player Experience Testing with Emotion Pattern Verification

Components

PX-MBT makes use of a number of main components. Some of them are provided by other open source projects which are included in PX-MBT as dependencies (this means that you also have access to these projects classes and methods from PX-MBT). Components marked with 💡 correspond to the GREEN components in the above picture of PX-MBT architecture.

• 💡 The PX Testing Tool component. More precisely it is a package, containing classes for generating test cases from a model, for reducing the test suite through sampling, for running the test cases on the game under test, and for performing PX evaluation on the generated traces. This component is implemented in PX-MBT itself, in the package eu.iv4xr.ux.pxtesting. The key classes in these package are: TestSuiteGenerator, PXTestAgentRunner, and EmotionPattern.

• 💡 The Model of Emotions component is implemented in the project jocc. More information on how this model works can be found in jocc Github site.

• 💡 Emotion Pattern is implemented in PX-MBT. It is implemented over LTL (Linear Temporal Logic). The implementation of LTL and 💡 LTL model checking in put in the aplib project, as these are useful for other purposes beyond PX-testing. aplib in included in PX-MBT as dependency. More information about LTL and LTL model checking can be found here.

• EFSM is provided by the project iv4xr-mbt aka Evo-MBT. It also provides the search-based testing (SBT) algorithms used by PX-MBT for its model-based test generation. Test generation using LTL model checking is also supported, using the afore mentioned LTL model checker.

• The Basic Agent for executing test cases is providing by the project aplib.

Papers

• Saba Gholizadeh Ansari, I. S. W. B. Prasetya, Davide Prandi, Fitsum Meshesha Kifetew, Frank Dignum, Mehdi Dastani, Gabriele Keller, Model-based Player Experience Testing with Emotion Pattern Verification, in the 26th International Conference on Fundamental Approaches to Software Engineering (FASE), 2023.

• An Appraisal Transition System for Event-Driven Emotions in Agent-Based Player Experience Testing, Ansari, Prasetya, Dastani, Dignum, Keller. In International Workshop on Engineering Multi-Agent Systems (EMAS), 2021.

License: Apache 2.0