Game hitches before lipsync (worse on HDD) and general performance issues

[drojf]

To my understanding, the game engine does not preload textures or audio. This means that every time a sprite is displayed, or a sound is played, the file is loaded from disk.

This is hidden in some cases, as:

• I think the OS provides some caching of file accesses anyway
• the loading is done on a different thread so you don't directly see the fps drop, but if you're on a HDD which hasn't yet spun up (or is taking a while), the game will definitely freeze until it's loaded the sprites/audio needed to continue.

In the past, someone's added some debug logging which records how long it takes to load audio files and images. You can enable this logging by showing the flag menu by pressing SHIFT-F10 (which toggles LFlagMonitor), then checking the output_log.txt. I think the game loads the audio on a different thread from the sprites, so the total time is not the sum, but the max of the texture load time and the audio load time.

The worst case is when lipsync is enabled, as it has to load 3 sprites, one for each mouth position, plus the voice file to be played. Here is an example from the log (cleaned up):

Loaded portrait/rim_komaru_0 in 102ms
Applied filter to portrait/rim_komaru_0 in 36ms
Loaded portrait/rim_komaru_1 in 103ms
Applied filter to portrait/rim_komaru_1 in 39ms
Loaded portrait/rim_komaru_2 in 105ms
Applied filter to portrait/rim_komaru_2 in 37ms
Loaded audio ps3/s05/27/100500036.ogg in 468 ms

EDIT: due to some changes I made to the DLL, I think the last .ogg file timing wouldn't have been correct - it would have been the sum of loading all the lipsync textures + the voice file. I've just crossed it out for now

(the filter is not a disk access - we apply tints and other effects at runtime to avoid needing to pre-generate all combinations of sprite + filter)

It would take at least 468ms before the game can begin to play the audio/do lipsync. I think the game doesn't begin displaying text (like animate the text being drawn across the screen) until the audio is loaded (but I'm not sure). I'm not sure why the .ogg file takes so long to load in this case.

While the issue isn't that bad for most users, even on a HDD, but having the game hitch randomly is quite annoying. Other programs running or Windows Update could also cause the game to lag, if they were using all the disk I/O.

If you're on an SSD, I don't think this would be an issue for most users.

In terms of user feedback, we occasionally get reports of this, but it's rare all things considered (I would expect to get more reports than we do).

---

I probably won't actually do it, but preloading textures/audio before reaching them would fix the issue. You would need to pre-parse the scripts, check for any file paths, and load them in advance to do this.

We already tell users to install the game on an SSD instead of a HDD if possible on our FAQ.

---

Edit: I've been playing some other games installed on my computer's HDD, and they also hitch when playing. I was under the impression that most games preload to hide these sorts of problems, but perhaps it's not as common as I thought.

[drojf]

After doing a little testing and thinking about this issue, the easiest way to fix it would be to fix how lipsync textures is stored on disk.

Currently, the lipsync textures are loaded in a blocking manner (the same way all textures in the game are loaded). Every time lipsync occurs (when a voice file is played with the corresponding character on the screen), our mod will currently re-load all 3 textures used for lipsync. Our sprites are currently 2560 x 1440, which isn't trivial size on disk or to be loaded into graphics memory.

To fix this, we could instead store and display just the mouth part of the sprite, like most games do. This has been brought up before, but wasn't considered as critical to the mod, and seemed too much work, but after thinking about it I don't think it's that bad to do (and I've done something similar in the past anyway so it's not foreign to me).

This would involve:

1. Diffing the base sprite (the '0' sprite) for each pose against the other mouth sprites (the '1' and '2' sprite)
2. Saving the mouth sprites for '1' and '2' (the '0' pose is just the base sprite)
3. Determining the mouth cutout region, and saving its top left corner position to a text file
4. Making the game:
   • load the textfile/database of mouth positions
   • Making the game display the mouth sprites for '1' and '2' at the position specified in the text file
5. Publishing the new DLL
6. Publishing new graphics archives with the new mouth sprites (it would be possible to distribute it as a separate archive until we're sure this works correctly, as the file would be very small)

---

Something I also thought about was that we actually already know in advance which lipsync textures to load, because the character who is about to lipsync should already be on the screen by the time lipsync happens. So we could load the lipsync textures at the same time we load the base texture for the character. However with the above changes, this is probably unnecessary, as the lipsync textures will be very small already.

---

You might be wondering how the game runs seemingly OK even though all textures are loaded in a blocking manner - this is because the load times are hidden as they mostly occur at scene transitions. But lipsync occurs immediately after you click, so it's easier to notice (if it blocks, it will block the text from appearing on the screen).

Audio files are loaded and played in a non-blocking manner (the legacy Unity WWW is used for loading the .ogg files in a coroutine), so I'm not going to worry about that for now.

---

Update: Just realized that there is a problem. Currently the lipsync repeatedly re-uses the layer where the sprite is located to draw the mouths. I imaginined overlaying the mouth sprite ontop of the character, but for that to work, I need an additional layer to draw the mouth on (I'm not sure if multiple characters talk at the same time, in which case I would need one layer per each character talking at the same time). This is probably the most annoying bit to handle of all this.

---

Another side note - if we do implement this, we will know for sure whether the stuck sprite issue has to do with lipsync (see Idealpersona's previous investigation into the stuck sprite bug: #45).

If the stuck sprite bug still occurs after this, it will be just the mouth which is stuck to the screen, not the whole sprite.

[drojf]

This should be "fixed" now, see

• Lipsync without spectrum and lipsync caching #82

While separating the sprite mouths from the sprite body has not been implemented, all sprite loads occur just before (or during?) scene transitions, so you cannot notice them. In addition, sprite loads are now cached if the same sprite is used multiple times, avoiding the issue previously where every time a voice was played, the images would be reloaded.