Add NVTX instrumentation

[simonbyrne]

Allow profiling with Nsight Systems.

I've added the headers to the tree until I can figure out how the deps stuff works.

To try it out, add

WITH_NVTX=1

to Make.user

[simonbyrne]

Currently failing due to a linker error:

/home/simonbyrne/src/julia/contrib/install.sh 644 julia_version.h /home/simonbyrne/src/julia/usr/include/julia/
 g++ -march=native -mtune=native -m64 -shared  -pipe -fPIC -fno-rtti -std=c++17 -fno-gnu-unique -pedantic -DUSE_NVTX -D_FILE_OFFSET_BITS=64    -O3 -ggdb2 -falign-functions -momit-leaf-frame-pointer -D_GNU_SOURCE -I. -I/home/simonbyrne/src/julia/src -I/home/simonbyrne/src/julia/src/flisp -I/home/simonbyrne/src/julia/src/support -I/home/simonbyrne/src/julia/src/gc -I/home/simonbyrne/src/julia/usr/include -I/home/simonbyrne/src/julia/usr/include -I/home/simonbyrne/src/julia/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wno-comment -Wpointer-arith -Wundef -Wno-unused-result -DJL_BUILD_ARCH='"x86_64"' -DJL_BUILD_UNAME='"Linux"' -I/home/simonbyrne/src/julia/usr/include -DLLVM_SHLIB "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys.so\"" "-DJL_LIBJULIA_SONAME=\"libjulia.so.1.12\"" ./jltypes.o ./gf.o ./typemap.o ./smallintset.o ./ast.o ./builtins.o ./module.o ./interpreter.o ./symbol.o ./dlload.o ./sys.o ./init.o ./task.o ./array.o ./genericmemory.o ./staticdata.o ./toplevel.o ./jl_uv.o ./datatype.o ./simplevector.o ./runtime_intrinsics.o ./precompile.o ./jloptions.o ./mtarraylist.o ./threading.o ./scheduler.o ./stackwalk.o ./method.o ./gc-alloc-profiler.o ./gc-heap-snapshot.o ./jlapi.o ./signal-handling.o ./safepoint.o ./timing.o ./subtype.o ./rtutils.o ./crc32c.o ./APInt-C.o ./processor.o ./ircode.o ./opaque_closure.o ./codegen-stubs.o ./coverage.o ./runtime_ccall.o -Wl,-rpath,'$ORIGIN' -Wl,-z,origin -Wl,--enable-new-dtags -o /home/simonbyrne/src/julia/usr/lib/libjulia-internal.so.1.12.0 -Wl,-z,relro -Wl,-Bdynamic -Wl,-no-undefined   -Wl,-Bsymbolic-functions -Wl,--enable-new-dtags -Wl,-z,noexecstack -L/home/simonbyrne/src/julia/usr/lib -L/home/simonbyrne/src/julia/usr/lib -Wl,--whole-archive ./flisp/libflisp.a -Wl,--whole-archive ./support/libsupport.a -Wl,--whole-archive ./gc/libgc.a -ljulia -Wl,--whole-archive /home/simonbyrne/src/julia/usr/lib/libuv.a -Wl,--whole-archive /home/simonbyrne/src/julia/usr/lib/libutf8proc.a -Wl,--no-whole-archive -lunwind  -L/home/simonbyrne/src/julia/usr/lib  -lLLVMTargetParser -lLLVMSupport -lLLVMDemangle -lrt -ldl -lpthread -lm -lz -Wl,--no-as-needed -ldl -lrt -lpthread -latomic -Wl,--export-dynamic,--as-needed,--no-whole-archive -Wl,--version-script=/home/simonbyrne/src/julia/src/julia.expmap     -Wl,-soname=libjulia-internal.so.1.12
/usr/bin/ld: ./gf.o: in function `nvtxInitializeInjectionLibrary_v3':
/home/simonbyrne/src/julia/src/./nvtx3/nvtxDetail/nvtxInit.h:216: undefined reference to `dlopen'
/usr/bin/ld: /home/simonbyrne/src/julia/src/./nvtx3/nvtxDetail/nvtxInit.h:225: undefined reference to `dlsym'
/usr/bin/ld: /home/simonbyrne/src/julia/src/./nvtx3/nvtxDetail/nvtxInit.h:265: undefined reference to `dlclose'
/usr/bin/ld: /home/simonbyrne/src/julia/src/./nvtx3/nvtxDetail/nvtxInit.h:228: undefined reference to `dlclose'
...

[simonbyrne]

This is now working thanks to @giordano!

[giordano]

How can I test this? I need to use https://github.com/JuliaGPU/NVTX.jl and install Nsight Systems?

[simonbyrne]

You don't need NVTX.jl, just Nsight Systems (the .run file is pretty easy to install).

A simple test case

nsys profile --trace=nvtx julia -e 'GC.gc()'

[ViralBShah]

cc @maleadt (in case you hadn't already seen it)

[giordano]

It took me a while to get a usable Nsight Systems on the Grace Hopper machine (@fspiga suggested me to simply use rpm2cpio nsight-systems-2024.4.1-2024.4.1.61_3431596-0.aarch64.rpm | cpio -idmv as non-root user to unpack the rpm), but I eventually got a trace

I need to practice more to get used to the interface, but it looks cool!

[giordano]

@simonbyrne do I understand correctly this doesn't support JIT-ed functions? When calling a user function in the list of events I see loads of names like anon!0xffff25630044. For that I'd still need to use NVTX.jl to instrument the code?

[simonbyrne]

Yes, this just instruments the compiler and runtime.

[ViralBShah]

Should we merge this?

[maleadt]

I haven't tested this, but LGTM.