[3.x] Add a project setting to enable 3D shadow dithering

[Calinou]

This improves overall shadow quality at the cost of a grainy appearance when up close. Dithering is disabled by default to preserve the current appearance in existing projects, but it will be enabled by default in Godot 4.0.

This setting works in both GLES3 and GLES2 and with any shadow filter mode (even Disabled).

This PR is effectively the opposite of #53961, but for 3.x 🙂

This effectively resurrects some of the ideas behind #7204 as well.

Note: GLES3 rendering is currently broken (it's all white without any shader errors). This is because more than 32 conditionals are being used, like in #54355.
Edit (June 2023): This seems to be fixed in the latest 3.x branch now, likely by incindental changes made for the ubershader implementation.

Preview

Don't pay attention to the peter-panning - it's unrelated to this pull request.

GLES3

Shadow Filter Mode: Disabled

No dithering	With dithering

Shadow Filter Mode: PCF5 (default)

No dithering	With dithering

Shadow Filter Mode: PCF13

No dithering	With dithering

GLES2

Shadow Filter Mode: Disabled

The SpotLight rendering breaks when the filter mode is Disabled for some reason (even without this pull request).

No dithering	With dithering

Shadow Filter Mode: PCF5 (default)

No dithering	With dithering

Shadow Filter Mode: PCF13

No dithering	With dithering

[Calinou]

Rebased and tested again, it works as expected (including GLES3 now). I'm marking this as ready for review accordingly.

[lawnjelly]

This reminds me I spent an evening doing a little experiment with dithering in 3.x a couple of months ago. This screenshot was just pushing things to get as soft shadows as possible:

But it does illustrate a couple of things:

• We may end up wanting to support more than 1 dithering method. Would an enum be better for this from the get go (rather than having an on / off and an enum)?
• For dithering, having a variable range is probably important, so the user can tune how they want things to look.

The option to varying the dither pattern by time is worth thinking about if the PR doesn't do this already. Also the dither pattern in the screenshot seems to look very regular, maybe we will end up wanting to have different dither pattern options.

As resolutions have increased (even on mobile) using dither to better trade off quality / performance becomes more viable option.

[Calinou]

• We may end up wanting to support more than 1 dithering method. Would an enum be better for this from the get go (rather than having an on / off and an enum)?

I'm not sure if it's worth the added complexity, as I don't think a lot of users will rush to toggle this option (it's disabled by default after all). The fact that shadows have visible noise in 4.x is already a controversial point, so I think a boolean is good enough.

• For dithering, having a variable range is probably important, so the user can tune how they want things to look.

This makes sense too, but I guess it should piggyback on the Shadow Blur property added by #60243 if that PR is merged. This is how it works in 4.x 🙂

The option to varying the dither pattern by time is worth thinking about if the PR doesn't do this already.

See #61834 which does this for master. I think this should be implemented in a separate PR to make the logic easier to follow.

Note that 3.x does not have TAA, so the usefulness of jittering the shadow dithering pattern is significantly decreased. You need a high refresh rate monitor (and a high framerate) for it to look good, unless your monitor has very slow response times.

[lawnjelly]

I don't know if you can do this with defines in GLSL, but it might be worth testing if something like this is possible:

#ifdef SHADOW_USE_DITHERING
#define PLUS_DITHER +((-vec2(0.5) + quick_hash(gl_FragCoord.xy)) * shadow_pixel_size)
#else
#define PLUS_DITHER 
#endif
...
avg += textureProj(shadow, vec4(pos PLUS_DITHER + vec2(-shadow_pixel_size.x, 0.0), depth, 1.0));