Addons: TSL PostProcessing - Fix Pixelation Pass Output

[cmhhelgeson]

Description

The current version of PixelationPassNode outputs incorrect normal values, causing the background of the screen to brighten when the normalEdgeStrength is increased.

Normal Edge Strength: 0

Normal Edge Strength: 2

This PR fixes the issue by outputting the correct normals and aligning the normalEdgeStrength behavior with the original shader. It also adds more views to the inspector.

https://raw.githack.com/cmhhelgeson/three.js/fix_pixelation_pass/examples/webgpu_postprocessing_pixel.html

[Mugen87]

I don't understand why using directionToColor() fixes the issue. This should only be used to decrease the bandwidth since it allows to use a RGBA8 texture for the normals instead of RGBA16. Why does it fix normals?

[cmhhelgeson]

Here is a comparison of WebGPU with normalView on the left versus WebGL pixel processing on the right. Despite the same settings, the outputs are different ( notice the lack of normal edges on the left facing sides of each cube on the back )

WebGPU normalView Pixelation vs WebGL Pixelation

As compared to when directionToColor( normalView ) is applied.

WebGPU directionToColor( normalView ) Pixelation vs WebGL Pixelation

Here is a comparison of the normals output by directionToColor( normalView) and normalView. The directionToColor( normalView ) normals are intended to replicate those output by the override material produced by the MeshNormalMaterial pass in RenderPixelatedPass, thus producing an effect that matches the behavior of the original.

directionToColor( normalView ) Inspector vs normalView Inspector

When looking at the codebase, I noted that in SVGRenderer, the output color performs a similar operation to the directionToColor functionality when .isMeshNormalMaterial is set to true.

} else if ( material.isMeshNormalMaterial ) {

     _normal.copy( element.normalModel ).applyMatrix3( _normalViewMatrix ).normalize();

     _color.setRGB( _normal.x, _normal.y, _normal.z ).multiplyScalar( 0.5 ).addScalar( 0.5 );

}

I couldn't find the relevant lines of code within the MeshNormalMaterial logic, but given this information as well as the similarities between directionToColor( normalView ) and the visual output of the normalMaterial within the RenderPixelatedPass, I simply intuited that the current approach was the most likely one to produce a visual result similar to the original. Granted, there are probably a bevy of considerations I am not taking into account, and the original effect is likely working around some limitations of the WebGLRenderer that have been obviated by the WebGPURenderer.

[cmhhelgeson]

Output of WebGL Pixelated Pass Normals Output

TLDR: I believe the original pass is using a transformed normalView, something that is suggested by the behavior in SVGRenderer and the agreement between the visual output of RenderPixelatedPass's normal pass and directionToColor( normalView ). I am trying to investigate the codebase further to see if there is a more accurate solution, but this replicates the original effect's output more accurately without the attendant problems with the background.

[Mugen87]

How about using something like this at the beginning of pixelation() to avoid side effects to the background:

const depth = sampleDepth( 0, 0 ).toVar();
depth.greaterThanEqual( 1.0 ).discard();

[cmhhelgeson]

How about using something like this at the beginning of pixelation() to avoid side effects to the background:

const depth = sampleDepth( 0, 0 ).toVar();
depth.greaterThanEqual( 1.0 ).discard();

This prevents the background from properly being rendered, so it would need to be different.

[Mugen87]

The current version of PixelationPassNode outputs incorrect normal values, causing the background of the screen to brighten when the normalEdgeStrength is increased.

I had the time for a closer look now and I can't reproduce this issue on latest dev. For comparison:

Normal strength: 0

Normal strength: 2

The background color is the same. Do I miss something?

[cmhhelgeson]

The background color is the same. Do I miss something?

I'm not sure, I've tried on multiple Windows computers with Chrome and the background color changes. I'm not sure if a separate browser and/or OS would produce a different effect.

https://raw.githack.com/mrdoob/three.js/dev/examples/index.html?q=pixel#webgpu_postprocessing_pixel

[cmhhelgeson]

@Mugen87 What browser/OS or what branch did you test on? I know that's likely not the issue but I'm trying to reconcile what may have caused the difference in results we both saw despite both testing the example on the current dev branch.

[Mugen87]

I'm testing with latest dev on macOS/Chrome. The behavior is equal across all browsers (Chrome, Firefox and Safari) and WebGL/WebGPU.

I'll try to test on a Window device when I have the chance.

[cmhhelgeson]

I'm testing with latest dev on macOS/Chrome. The behavior is equal across all browsers (Chrome, Firefox and Safari) and WebGL/WebGPU.

I'll try to test on a Window device when I have the chance.

Weird. Can you show the visual result on Mac when you output the normal of the pixelationPass to the inspector? Maybe that will surface the difference between the normals on Mac and normals on Windows if there is one.

[Mugen87]

It looks like so:

To be clear, this is from the current example on dev not from the PR.

[cmhhelgeson]

It looks like so:

To be clear, this is from the current example on dev not from the PR.

Pretty much the same thing I was seeing on unmodified code/dev.

[cmhhelgeson]

On Mac's WebGPU example does the effect of the normalPass correlate with its effect in the WebGL sample (i.e the lines produced by the normalEdgeStrength are of equivalent thickness and look the same?)