[3.x] Add support for single compilation unit builds

[lawnjelly]

AKA Unity / Jumbo builds.

Adds support for simple SCU build. This speeds up compilation by compiling multiple cpp files within a single translation unit.

This is an optional extra build that occurs when you set use_scu=yes in Scons.

Supersedes #69405
3.x version of #77170 , see that PR for details.

This PR now removes the third party acceleration so that can be reviewed as a separate PR.

Some benchmarks

(Intel i5-7500T, 2.7ghz x 4, Linux Mint 20, SSD. i.e. low end CPU)

Note: These figures were including third party acceleration, so the PR will now be slightly less efficient.

Full rebuild (DEV)

Normal build 7mins 53
SCU build 3mins 19

Touching node.h (DEV)

Normal build 4mins 10
SCU build 1min 33

Full rebuild (release)

Normal build 18mins 10
SCU build 12mins 16

[Calinou]

I tested this PR on an Intel Core i9-13900K on Fedora 38:

scons -j32 platform=x11 linker=mold optimize=none fast_unsafe=yes progress=no debug_symbols=yes builtin_embree=no builtin_enet=no builtin_freetype=no builtin_graphite=no builtin_harfbuzz=no builtin_libogg=no builtin_libpng=no builtin_libtheora=no builtin_libvorbis=no builtin_libwebp=no builtin_mbedtls=no builtin_miniupnpc=no builtin_pcre2=no builtin_zlib=no builtin_zstd=no

• With scu_build=no, modifying scene/main/node.h then recompiling: 4.3 seconds on average
• With scu_build=yes, modifying scene/main/node.h then recompiling: 2.9 seconds on average

The gain isn't as pronounced as in the master PR, but that's expected as we're starting to hit linking and SCons limitations here. I wonder if I could get sub-second incremental builds with full dynamic linking… 🙂

[lawnjelly]

Master already has the "lowest hanging fruit" converted, which is most of core, in order to accelerate touching node.h, which is the most common iterative bottleneck for contributors.

This PR, has quite a bit more in the SCU build than master. It accelerates core like master, but a lot of the added areas are in third party / modules, which isn't wholly recompiled on touching node.h - so these additional changes are primarily to improve full rebuild speeds. Depending on @akien-mga 's preferences, we can easily leave the additional changes out and just accelerate core as in master.

Converting thirdparty for full rebuild hasn't tended to be as dramatic because a lot of it is more difficult to accelerate. There are a lot of small separate libraries (these could potentially be amalgamated), and also global namespace collisions which prevent adding to the SCU build, so only a small fraction is accelerated currently. The option of modifying the code to prevent namespace collisions is also less attractive for third party, as we want to easily be able to update their source code, so we are relying on the library authors to write "clean" code.

There is in short a balance to be made for third party between the gains, and any maintenance requirements / cognitive load. We could completely go to town, but if that makes the build hard to maintain, then the costs may exceed the benefits.

Timings before / after SCU build

The overall benefit is quite a complex relationship, and to a certain extent this depends on how efficient the build was before. Master is exceptionally inefficient at compiling (thus more gains are on the table). If you switch on the "show includes" compiler setting for master you can see why - huge dependency chains.

The other thing is as you say, on your (super fast) hardware you may be hitting linking bottlenecks quicker than in master. On my slower hardware it's closer to a third of the time for touching node.h, which is comparable to master. Bear in mind on my PC the build takes 1 min 33, and yours takes 2.9 seconds, so the relative bottlenecks are likely to be different.

[Calinou]

Bear in mind on my PC the build takes 1 min 33, and yours takes 2.9 seconds, so the relative bottlenecks are likely to be different.

To clarify, I also have ccache installed, which means a lot of the files from previous builds are reused. This was also the case when benchmarking the master PR, and is effective both with and without SCU builds. Without ccache, it'd probably take twice as long on average for those incremental builds.

[akien-mga]

I think accelerating third-party more is great! However the diff between this PR and the master one I reviewed make it a bit hard to easily approve it without spending a lot of time separating both.

It would be easier for me to merge first the equivalent changes to what was done in master, and then review a follow up PR accelerating the rest (which can then be forward ported too).

But that's more work for you to separate things again, so it's not necessarily a good option either. Let me know what you prefer, I can also try to find the time to review this one as is.

[lawnjelly]

It would be easier for me to merge first the equivalent changes to what was done in master, and then review a follow up PR accelerating the rest (which can then be forward ported too).

Sure that's not a big deal my end, and it does make sense, easier to review is always good. I'll have a go at this. 👍

[lawnjelly]

PR is now modified to remove third party changes and be as similar as possible to the master PR.

[akien-mga]

Should be platform/*/export/*.cpp, no?

This didn't seem needed in the master version, what's the difference here?

[lawnjelly]

Slight difference here:
In master it compiles each export platform individually. In 3.x, particularly as the export platforms are fairly finalized, these are amalgamated as a single SCU file which is stored in platform/.scu/ folder. (The path is correct, but it will be confusing as this is the first time you have seen this pattern.)

This makes more sense when read with the scu_builders.py file for this:

    process_folder(
        [
            "platform",
            "android/export",
            "iphone/export",
            "javascript/export",
            "osx/export",
            "uwp/export",
            "windows/export",
            "x11/export",
        ]
    )

This is adding all the various platform exporter folders all into a single uber-SCU file which is stored off the platform folder.

To explain further, the array argument, the first string is the "main" folder, and the following folders are considered sub-folders to be added to the same SCU file.

I'm happy to simplify if preferred to the same method as master, but will lose out on acceleration, but it is worth bearing in mind the exporters are quite significant in the incremental rebuilds. In 3.x, each export directory typically only contains a single file, so there is no accelerating unless they are amalgamated.

[akien-mga]

Thanks!