Improve shadow filters

[x2048]

With the introduction of new SM texture distortion in #12166, PCF filter for shadows was outdated and produced wrong results.

What's done:

• Use information available in the new vertex shaders from Implement shadow offsets for the new SM distortion function #12191 (perspective_factor), instead of trying to reverse-calculate it from the final position.
• Use fewer filter samples, as it is no longer necessary with the new, more accurate distortion, the same applies to penumbra calculation.
• Optimize the filter convolution loops to perform few repetitive calculations.

Benefits:

• Filtering and less 'dancing' of shadows when texture precision permits that.
• Soft shadows work again.
• It's faster than the previous implementation.

To do

This PR is a Work in Progress and depends on #12191 being merged, but review is welcome.

• Rebase when Implement shadow offsets for the new SM distortion function #12191 is merged.
• Fix artifacts for certain combinations of overlapping distant and close shadows.
  - So far it has been solved by limiting the soft shadow radius

How to test

1. Enable dynamic shadows at your favorite quality level.
2. Optional: You can also set the soft shadow radius to something more than 1.
3. Start a compatible game
4. Enjoy the filtered/soft shadows

[lhofhansl]

Is there any way to make the shadow boundaries softer?

This is with view distance 300 and shadow texture size 1024:

Or this:

Would be nice if that can look smoother.

[lhofhansl]

Looks like the moire pattern are worse now:

This is the Hidroplane mode. (https://content.minetest.net/packages/apercy/hidroplane/)

[x2048]

The moire pattern on objects might get worse due to changes in offset biases that need more tuning (I spotted some artifacts too). Latest changes to combat peter-panning might also contribute to more visible moire/self-shadowing.

[lhofhansl]

Fix artifacts for certain combinations of overlapping distant and close shadows.

I noticed that with soft-shadows. When a close shadows overlaps a distance soft-shadow the far (softer) shadow "wins". Is that what you mean?

[lhofhansl]

Filtering with latest change:

[lhofhansl]

Peter panning almost gone:

Traded for no close filtering I guess:

[x2048]

Fix artifacts for certain combinations of overlapping distant and close shadows.

I noticed that with soft-shadows. When a close shadows overlaps a distance soft-shadow the far (softer) shadow "wins". Is that what you mean?

There was a more specific problem that I would call a "tree on a cliff" problem. When a high tree projected a shadow on a cliff, its silhouette would appear in the shadow of the cliff, because those pixels had much longer shadow depth and penumbra radius. The problem should be fixed now by translating depth to penumbra using a linear function (opposed to quadratic function previously).

The problem of softer shadows taking over the harder one is due to storing only one depth value per texel, and I don't see how it can be solved without complicating the SM texture generation significantly.

[x2048]

Peter panning almost gone:
Traded for no close filtering I guess:

Yes, that's the tradeoff I chose to do for near-zero shadow depth.

[x2048]

Is there any way to make the shadow boundaries softer?

Should be done in the latest commits. Filter radius is now at least 2 texels, which together with adaptive quality reduces effect of the sharp edges / dancing shadows.

[lhofhansl]

So far I think it still looks better without this change. :(

[x2048]

Can you share examples?

[x2048]

@lhofhansl I've tried to find the balance between player's feet and peter panning. Looks much better on my machine at least.

[lhofhansl]

Thanks for looking!

The fragment shaders do not compile for me. I had to make the following changes:

-	int n = 0;
+	float n = 0;
...
-	depth *= world_to_texture / max(n, 1);
+	depth *= world_to_texture / max(n, 1.0);

and

-	samples = clamp(int(pow(2 * radius + 1, 2.0)), 1, samples);
+	samples = int(clamp(int(pow(2 * radius + 1, 2.0)), 1, samples));

At the relevant places on the object and nodes fragment shaders.

Also the player shadow looks still "jaggedy":

[x2048]

@lhofhansl I've applied the fixes, thank you.

Also the player shadow looks still "jaggedy":

I don't get the same effect as you do.

[lhofhansl]

Hmm... I think my view distance is larger (300) and shadow map size is set 4096 (and max shadow distance is 450)

[lhofhansl]

Thanks for sticking with this @x2048. I'll try out the latest change.

[lhofhansl]

Closeby shadows look very pixelate still. (shadow map size 2048, viewing_range 300, max shadow distance 450).

I always think that more filtering looks better and immerses better, since you notice the artifacts less.

[lhofhansl]

Don't give up on us @x2048 :)

[x2048]

Still here. Keeping silence since there's not a lot I can share now

[x2048]

@lhofhansl OK, I have something interesting now.

1. Fixed a problem with OpenGL filtering screwing depth values and leading to artifacts at shadow edges.
2. Changed penumbra radius calculation to O(1) so now soft shadow radius can be adjusted without performance penalty.
3. Edgy feet are still here but not that edgy. I prefer to avoid peter panning more than proper shadow for own feet.

[lhofhansl]

👍

With soft-shadows set to 10 is looks nice now! (Still the very close edges with the player shadows, but I think that is cool)
Might want to default soft-shadows to 5 or something, and also allow larger values.

[x2048]

Agree. I've increased the maximum blur to 30 now, changed default to 5 and copied the the changes to object shaders.

[x2048]

Some screenshots:

• Duck plane:
  
• Colored windows
  

[kilbith]

@x2048 Can you also change the default value of shadow_update_frames to 1 because I really can stand this "laggy" feel with shadows when the map gets changed. It really does look bad.

[x2048]

@kilbith I'll change the defaults in a different PR

[Zughy]

@x2048 is this still WIP?

[x2048]

@Zughy Thanks for pointing out, it's approved now, so not a WIP