Add coverage measurement for Clang and increase test coverage

[goerch]

I've seen others adding coverage as different build types. Not sure if we should work in that direction.

Commands to process the profile output (run in build):

# Windows
llvm-profdata merge -sparse tests/*.profraw -o ggml.profdata
llvm-cov show ./bin/Release/test-grad0.exe -instr-profile=ggml.profdata
llvm-cov report ./bin/Release/test-grad0.exe -instr-profile=ggml.profdata

# Linux
llvm-profdata merge -sparse tests/*.profraw -o ggml.profdata
llvm-cov show ./bin/test-grad0 -instr-profile=ggml.profdata
llvm-cov report ./bin/test-grad0 -instr-profile=ggml.profdata

# Mac OS
xcrun llvm-profdata merge -sparse tests/*.profraw -o ggml.profdata
xcrun llvm-cov show ./bin/test-grad0 -instr-profile=ggml.profdata
xcrun llvm-cov report ./bin/test-grad0 -instr-profile=ggml.profdata

I'm sure terrible stuff will happen on non clang environments.

[goerch]

Looks like I'm missing a library at link time. This implicitly works on Windows with ClangCL. Grateful for any hints!

[ggerganov]

The GGML_TEST_COVERAGE cmake option is now disabled by default. Need to explicitly enable it to generate coverage.
I've added an example in the Github Actions CI for Ubuntu and Mac OS

This seems to work now:

Can we add some Github bot to report the coverage stats on each PR?
Or somehow generate HTML page with the reports?

[goerch]

@ggerganov : thanks for helping out!

Can we add some Github bot to report the coverage stats on each PR?

That would be nice, but I'm out of my depth there.

Or somehow generate HTML page with the reports?

The documentation for llvm-cov mentions a format option which seems to support HTML.

In the meantime I tried to change test-grad0 to support F16 and F32, but I'm not sure if this makes sense, because there are only a couple of type specific functions

• ggml_compute_forward_dup_f16
• ggml_compute_forward_add_f16_f32
• ggml_compute_forward_add_f16_f16
• ggml_compute_forward_alibi_f16
• ggml_compute_forward_rope_f16

but not a complete infrastructure. What do you intend us to do with these?

[goerch]

I think I found a reasonable way to add half precision tests for dup, add and rope. When testing I noticed the probably unrelated messages

test-grad0: iter:0/4
cross_entropy_loss: ndims=2, i=0, k=0, x0=0.602405, xm=0.502405, xp=0.702405, f0=3.546007, f1=3.546007, g0=0.000000, g1=-0.096169, eps=0.100000, error_abs=0.096169, error_rel=0.000000
cross_entropy_loss: ndims=3, i=0, k=0, x0=0.512436, xm=0.412436, xp=0.612436, f0=1.799026, f1=1.799026, g0=0.000000, g1=-0.104886, eps=0.100000, error_abs=0.104886, error_rel=0.000000
test-grad0: iter:1/4
cross_entropy_loss: ndims=2, i=0, k=0, x0=-0.776666, xm=-0.876666, xp=-0.676666, f0=1.353190, f1=1.353190, g0=0.000000, g1=0.284413, eps=0.100000, error_abs=0.284413, error_rel=0.000000
cross_entropy_loss: ndims=3, i=0, k=0, x0=0.797418, xm=0.697418, xp=0.897418, f0=1.246108, f1=1.246108, g0=0.000000, g1=0.576844, eps=0.100000, error_abs=0.576844, error_rel=0.000000
test-grad0: iter:2/4
cross_entropy_loss: ndims=2, i=0, k=0, x0=-0.838496, xm=-0.938496, xp=-0.738496, f0=0.417900, f1=0.417900, g0=0.000000, g1=-0.258858, eps=0.100000, error_abs=0.258858, error_rel=0.000000
cross_entropy_loss: ndims=3, i=0, k=0, x0=0.645314, xm=0.545314, xp=0.745314, f0=0.307985, f1=0.307985, g0=0.000000, g1=0.075483, eps=0.100000, error_abs=0.075483, error_rel=0.000000
flash_attn f32: ndims=3, i=0, k=39, x0=-0.089866, xm=-0.090016, xp=-0.089716, f0=0.179296, f1=0.179296, g0=-0.000348, g1=0.001414, eps=0.000150, error_abs=0.001762, error_rel=5.067115
flash_attn f32: ndims=4, i=0, k=23, x0=0.111595, xm=0.111445, xp=0.111745, f0=0.437807, f1=0.437807, g0=0.000695, g1=-0.001808, eps=0.000150, error_abs=0.002503, error_rel=3.599792
test-grad0: iter:3/4

I also noticed that we have ggml_compute_forward_flash_attn_f16, which I just tried to add a test for, but the corresponding ggml_compute_forward_flash_attn_back_f16 seems to be missing. Should I try to add it?

Edit: one more observation: it seems a couple of operations like relu are not covered by test-grad0. Are these good candidates for more tests?

[goerch]

@ggerganov : the test for relu needed an unrealistic large relative error. I believe this was a bug, fixed in my last commit. Do you want me to add more to this PR?

[ggerganov]

Is this goto forgotten?

[goerch]

Yes, sorry! Had already changed that, but forgot to push the commit.

[ggerganov]

Great work! Thanks