[glsl-out] Incorrect rounding for max-float constant expression

[Wumpf]

In v0.12 (prior to gfx-rs/naga#2266), this wgsl constant
const f32max = 0x1.fffffep+127;
would inline in glsl as 3.4028234663852886e38.

However, in v0.13 this results into a glsl constant
const float f32max = 3.4028235e38;
When testing a provided max-float value (read from a texture in my case) for >=f32max this will now yield false in a WebGL setup.
Interestingly, this scenario works fine on Metal where this yields constant float f32max = 340282350000000000000000000000000000000.0;. I suspect that glsl applies different rounding here.

More backends and situations might be affected by this, but I was able to verify it only so far for glsl-out and rule it out for msl-out. It would be worth exploring the effect on other backends as well and find out if this has any other accuracy problems.

[fornwall]

The difference between 3.4028234663852886e38 and 3.4028235e38 comes from casting f32 to f64 or not before formatting f32max using {:?}:

let f32max = 3.4028235e38_f32;
println!("f64: {:?}", f32max as f64); // What glsl-out effectively did before gfx-rs/naga#2266, outputs "3.4028234663852886e38"
println!("f32: {:?}", f32max); // What glsl-out does now, outputs "3.4028235e38"
println!("Display: {}.0", f32max);  // What metal-out does, outputs "340282350000000000000000000000000000000.0"

But while different in string format, parsed as 32-bit floating point numbers by rustc, the three numbers are identical. Check https://evanw.github.io/float-toy/ or the below snippet:

let a = 3.4028234663852886e38_f32;
let b = 3.4028235e38_f32;
let c = 340282350000000000000000000000000000000.0_f32;
println!("a: {:0>32b}", a.to_bits()); // "01111111011111111111111111111111"
println!("b: {:0>32b}", b.to_bits()); // "01111111011111111111111111111111"
println!("c: {:0>32b}", c.to_bits()); // "01111111011111111111111111111111"
println!("{}, {}", a == b, b == c); // "true, true"

Here the bits 01111111011111111111111111111111 are indeed the correct bits for f32max:

0     11111110  11111111111111111111111
^     ^         ^
sign  exponent  fraction

sign: 0, positive number
exponent: 127, maximum value
fraction: 11111111111111111111111, all bits set

Since we've found an observable change in behaviour, we still need to understand what is happening.

@Wumpf To determine if the format makes any difference, could you try wgpu = { git = "https://github.com/fornwall/wgpu", rev = "25934854" }? This is wgpu 0.17 with a single modification to output const float f32max = 3.4028234663852886e38 instead. Does that make any difference? Otherwise it might be the change from inlining to a constant variable which causes the difference in behaviour.

[Wumpf]

thanks for the detailed analysis and providing a test case @fornwall!
I tried it against my repro case and it worked with your tweak! This would imply that glsl reads the string interpretation incorrectly? 🤔

I didn't build a simple repro case on my side, so the instructions to see the thing are somewhat.. involved. For reference, this is the commit I tried against rerun-io/rerun@a395e5a
But it should be possible to build something very minimal: What happens in my case is that I read a f32max from a texture and then check it here https://github.com/rerun-io/rerun/blob/andreas/naga-glsl-constant-fix/crates/re_renderer/shader/utils/size.wgsl#L13 which ends up determining the size of a bunch of points (quads) being drawn.
➡️ Meaning the whole thing boils down to comparing a float constant to a float that comes in via a read.

[fornwall]

This would imply that glsl reads the string interpretation incorrectly? 🤔

A stackoverflow question shows that there a WebGL2 implementation computed a value as the equivalent 64-bit floating point code in C++ would have, which seems to be allowed behaviour.

Related: gpuweb/gpuweb#3431, wgsl: add way to get special values for a given numeric type (e.g. max, min, lowest).

As I understand it the previous as f64 formatting happened more to work "by chance" (?), so I don't know what a good way forward here is unfortunately.

[Wumpf]

If the problem is that webgl uses 64bit floats that would imply in this case that...

• the max constant is in actuality a 64bit float, meaning it has a higher value than expected
• the comparison operation may compare two f64 or a f64 with a f32 one, with the potential f32 one having max-f32 loaded -> this works fine if the f64 was correctly set

So that would mean that if we always set the 64bit representation of floats for constants in Glsl as it does on the branch you provided we should be at least better off? I mean the problem boils down to 3.4028234663852886e38_f64 != 3.4028235e38_f64, right? Wouldn't say it worked by chance since all f32 can be represented in f64. If the hypothesis is correct, using f64 literals always would solve the issue consistently I believe 🤔

Agreed that having limit constants in wgsl would be preferable in any case, but arguably this is an orthogonal problem to the one of "constant gets the wrong value assigned"

[fornwall]

In a webgl2 shader, it seems like indeed 3.4028235e38 != 3.4028234663852886e38 with precision highp float: https://fornwall.net/webgl2-float-comparison (at least on all browser and os setups I've tested with)

As said above, this is in contrast to treating the numbers are 32-bit floats, for instance in rust: assert_eq!(3.4028235e38_f32, 3.4028234663852886e38_f32);.

So the effective behaviour is like the shader uses f64 in computations:

let value_from_texture = 3.4028235e38_f32 as f64;
let f32max: f64 = 3.4028235e38_f32;

which causes value_from_texture >= f32max to be false.

Using f64 literals would indeed solve this problem. Are there any potential drawbacks? Let's create a PR to raise visibility and perhaps get more eyes on it (I can do it later today).

[teoxoy]

I think we are running into rust-lang/rust#63171 whereas previously we were passing an f64 for formatting which was causing the fmt function to give us more digits.

The full value of f32::MAX is 340282346638528859811704183484516925440 (ref https://stackoverflow.com/a/60224754). So, if not enough digits are written, we are relying on the rounding behavior of each shading language's float parser.

[teoxoy]

In my case 3.4028235e38 seems to get rounded to NaN (1.0/0.0) on https://fornwall.net/webgl2-float-comparison.

Also found https://float.exposed/0x7f7fffff which is a nice tool for looking at representations of floats.

Not sure what a proper solution for this is, maybe we should just cast to f64 before writing although once we fully support f64 we'd need f128 for those. Or come up with an algo for formatting floats properly but that sounds involved.

[fornwall]

Created gfx-rs/naga#2443, adding the as f64 which was confirmed resolving the issue mentioned here.