Implement a "Safe Mode" for recovering crashing projects during initialization

[rsubtil]

Implements godotengine/godot-proposals#2628

Note

This description is updated whenever any significant changes are made, and should always reflect the current state of this PR

Overview

This adds a new special "Safe Mode" that, as described, disables some engine features that can make a project fail to launch during startup. When this mode is launched, the following features are completely disabled:

• Tool scripts (GDScript and C#)
• Editor plugins
• GDExtension addons
• Scene restoring

This is displayed to the user as soon as the project is opened in this mode, through both a popup, and a notification:

Because some of these features may be necessary for a project to work properly, this mode may put the project in an unusable/unworkable state. Since this is a temporary mode meant to allow only for some basic troubleshooting, a few other changes were done as well to reflect this:

• All launching functionality is disabled, with the run bar modified to reflect it
  
• The asset library is disabled, since it's pointless to use during this mode
  
• Tool scripts are displayed differently to clarify that they're not being run during this mode
  
• The plugin configuration panel has a warning to clarify plugins do not run during this mode
  

Usage

Automatic detection

When launching a project, Godot creates a lock file at user://.safe_mode_lock. This file persists during the engine initialization, and is removed when the editor starts scanning the project files, after one second.

If any crash/hang happens during initialization, this file is not removed. So, if this file exists the next time Godot attempts to open the project, then the engine knows the project failed to open last time, thus prompting the user to open it in this mode.

CLI

This mode can be manually triggered by opening a project with the new --safe-mode flag:

$ godot -e -path <...> --safe-mode

$ godot --help

Godot Engine v4.3.beta.mono.custom_build.865a09e0c (2024-05-27 18:48:52 UTC) - https://godotengine.org
...
Run options:
...
  --safe-mode                       E  Start the editor in safe mode, which disables typical features that can cause startup crashes, such as tool scripts, editor plugins, GDExtension addons, and others.
...

UI

When a project is opened, the project manager searches for any existing lock file; if that's the case, it opens a popup to suggest using this mode. It explains what this mode does and its intended usage.

It is also possible to use this mode manually from the new options button next to the Edit button:

Test projects

To test this functionality, I've made a test suite of projects that crash the engine on initialization with typical user scenarios:

• GDScript tool script: - Project with a bad tool script, which is attached to a node on the main scene.
• C# tool script: - Same as above, but with a C# script, which requires rebuilding the project when the script is changed.
• Editor plugin: - Project with a bad editor plugin, which internally has bad GDScript code.
• GDExtension addon: - Project with an external error in the GDExtension addon.

All of these test cases fail to open on a normal Godot session and require manual intervention to be recovered. But under safe mode, every test must open successfully.

	GDScript tool script	C# tool script	Editor plugin	GDExtension addon
Crash	tool_gdscript_crash.zip	tool_csharp_crash.zip	plugin_crash.zip	gdextension_crash.zip
Hang	tool_gdscript_hang.zip	tool_csharp_hang.zip	plugin_hang.zip	gdextension_hang.zip

• Crash: This project has some bad code that immediately crashes the engine when opened.
• Hang: This project has some bad code that loops infinitely, which hangs the engine and leaves it frozen until forcibly closed.

Future work

This is a basic implementation so far, and the usefulness of this mode should be improved with other additional features:

• An automatic mechanism to detect failing scenarios and automatically prompt users to enable this mode, as described in Add a way to disable all plugins when opening a project (safe mode) godot-proposals#2628 (edit: implemented in this PR too)
• Being able to open/edit scenes when some dependencies are missing, since these can originate from temporarily disabled editor/GDExtension plugins.
• Adding other problematic features for safe mode that I didn't cover (e.g. importing assets)

[YeldhamDev]

Is the yellow border for the safe mode dialog necessary? It won't even be seen by most since windows aren't embedded by default in the editor.

[rsubtil]

Is the yellow border for the safe mode dialog necessary? It won't even be seen by most since windows aren't embedded by default in the editor.

Good point, I don't use the default setting of separated windows 😅

It was more of an experiment, but I believe the yellow banner at the run bar is already noticeable enough 👍

[Calinou]

Note that we could automatically prompt the user to edit a project in safe mode after an editor (not project) crash. To do so, we need to write a file in the project's .godot/ folder when the editor crashes. This can be accomplished in the crash handler when Engine::get_singleton()->is_editor_hint() is true.

When opening any project, the editor checks for the presence of this file. If the file is present, a dialog is spawned to prompt the user to open the project in safe mode. Regardless of the user's answer, the file is removed afterwards when the editor is done initializing and rescanning things.

When opening a project from the command line, the project opens as usual, but a warning message is printed to warn the user that they can use --safe-mode if desired (this is done to avoid breaking automation). The file is then removed after the editor is done initializing and rescanning things.

After doing this, we probably won't need to have the Edit (Safe Mode) button anymore in the project manager (which I feel is too prominent right now). We could make it manually accessible by shift-clicking, but I think the automatic prompt will suit the 90% use case.

A similar approach could be used to write a "lock file" in .godot/ to warn the user when opening the Godot editor multiple times at once on the same project. (This is sometimes desired, but it comes with some edge cases so it probably makes sense to warn about it.)

[rsubtil]

@Calinou yes, I think this an important mechanism to complement this feature. I hadn't included it yet because, actually, I wanted to confirm this detail that you've mentioned, before proceeding:

This can be accomplished in the crash handler when Engine::get_singleton()->is_editor_hint() is true.

Do we really want this mode to be triggered on any editor crash? I think it would generate more false positives (e.g. internal engine crashes) and generate user confusion because this mode would implicitly be triggered for all types of crashes, when it can only help in troubleshooting a subset of them.

I'm more concerned in ensuring we avoid false positives, and limiting this mode to only trigger on initialization crashes helps to reduce that. The usefulness of this mode, in my opinion, is tied to situations where a user lost the ability to edit the project at all and requires some form of external intervention to recover them.

[KoBeWi]

Maybe we could create some "init" file when starting editor and delete it once the editor loads. If it doesn't load it means it crashed.
Though I suppose there are some edge cases like exporting from CLI etc. so not sure if it can be done reliably.

[rsubtil]

@KoBeWi should I add the auto-detect crash functionality we discussed already in this PR, or do it in another PR after this one is merged?

I think @Calinou brings a valid point in doing it right now; if this functionality is implemented, there isn't much need to add the new "Edit in Safe Mode" button to the project manager.

I don't expect this functionality to take much effort to implement, but considering my current availability, it might still take around one week to implement.

[KoBeWi]

Well, this is not going to be merged before 4.4, so you have some time to improve it.

[rsubtil]

I've finish up the auto-detection mechanism and rework to the project manager interface. Will update the original PR description with these new changes.

EDIT: Done. The gist of the new changes are essentially:

• The auto-detection menchanism was implemented with a lock file at each project's user://.safe_mode_lock. It is removed if the engine opens successfully, and if not, used in the next launch to detect a failure.
• The project manager's Edit button was reworked to have an additional options button next to it, where it is possible to manually trigger the safe mode for any project.

[rsubtil]

Fixed a merge conflict that appeared in the meantime. But, since master updated to 4.4, this revealed a bug with the auto-detection mechanism: if a warning existed when opening a project (e.g. "Do you wanna upgrade the project to Godot 4.4"), this would bypass the check for safe mode.

I've changed the code to divide this in two steps: the check_safe_mode exclusively to check the safe mode, and the check_warnings for the current warnings. And then modified the code to call the correct instances depending on the current scenario. Now, if a project has both warning types, it first displays the safe mode popup, then the warnings popup before launching.

[fire]

Remember headless mode has no prompts and needs to work with integration and deployment

[rsubtil]

Remember headless mode has no prompts and needs to work with integration and deployment

The detection mechanism is implemented only on the project manager; to trigger this mode from CLI, one has to pass the --safe-mode flag explicitly.

[akien-mga]

Just a quick update on the review status: I'd like to do a proper review myself, that's a feature I've wanted for a while so I want to make sure it works well.

Will also need a rebase before merge.

[rsubtil]

Thanks, I didn't notice another merge conflict arose in the meantime; I'll fix it, possibly today

[rsubtil]

Merge conflict has now been fixed, but it wasn't as simple as I hoped; it came from #96861, @KoBeWi please have a look to ensure I didn't break anything 😅

[KoBeWi]

You renamed _open_selected_projects_with_migration method

[rsubtil]

You renamed _open_selected_projects_with_migration method

Yes, that was intended; I renamed _open_selected_project_ask to _open_selected_project_check_warnings to not only make it's functionality clearer, but also because there's now multiple possible steps when opening a project (check warnings, check safe mode).

Since you're doing something similar, I saw fit to also rename it to _open_selected_project_check_migration.

[KoBeWi]

Since you're doing something similar, I saw fit to also rename it to _open_selected_project_check_migration.

But the function performs the migration. The checks are technically done before the info dialog appears.

[rsubtil]

But the function performs the migration. The checks are technically done before the info dialog appears.

Ok yeah, you're right, I've reverted this name change.