[xpu] ERROR: Failed building wheel for triton when USE_XPU=1 make triton

[ZhaoqiongZ]

🐛 Describe the bug

According to this https://pytorch.org/docs/main/notes/get_start_xpu.html
When I make triton with USE_XPU=1, I encountered the following error

Building wheels for collected packages: triton
  Building wheel for triton (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for triton (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [189 lines of output]
      copy /root/.triton/nvidia/ptxas/bin/ptxas to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/bin/ptxas ...
      copy /root/.triton/nvidia/cuobjdump/bin/cuobjdump to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/bin/cuobjdump ...
      copy /root/.triton/nvidia/nvdisasm/bin/nvdisasm to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/bin/nvdisasm ...
      copy /root/.triton/nvidia/cudacrt/include to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/include ...
      copy /root/.triton/nvidia/cudart/include to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/include ...
      copy /root/.triton/nvidia/cupti/include to /tmp/pip-req-build-goyt6fp4/python/../third_party/nvidia/backend/include ...
      running bdist_wheel
      running build
      running build_py
      running build_ext
      CMake Deprecation Warning at CMakeLists.txt:6 (cmake_policy):
        The OLD behavior for policy CMP0116 will be removed from a future version
        of CMake.

        The cmake-policies(7) manual explains that the OLD behaviors of all
        policies are deprecated and that a policy should be set to OLD only under
        specific short-term circumstances.  Projects should be ported to the NEW
        behavior and not rely on setting a policy to OLD.


      -- The C compiler identification is GNU 11.4.0
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
      -- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
      -- Linker detection: GNU ld
      -- SPIRV-LLVM location is not specified. Will try to download
                        SPIRVToLLVMTranslator from https://github.com/KhronosGroup/SPIRV-LLVM-Translator into
                        /tmp/pip-req-build-goyt6fp4/python/build/cmake.linux-x86_64-cpython-3.10/lib/Target/SPIRV/SPIRVToLLVMTranslator
      CMake Warning (dev) at /tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
        The package name passed to `find_package_handle_standard_args` (PkgConfig)
        does not match the name of the calling package (SPIRVToLLVMTranslator).
        This can lead to problems in calling code that expects `find_package`
        result variables (e.g., `_FOUND`) to follow a certain pattern.
      Call Stack (most recent call first):
        /tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.29/Modules/FindPkgConfig.cmake:114 (find_package_handle_standard_args)
        python/build/cmake.linux-x86_64-cpython-3.10/lib/Target/SPIRV/SPIRVToLLVMTranslator/CMakeLists.txt:9 (include)
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2")
      -- SPIR-V Headers location is not specified. Will try to download
                spirv.hpp from https://github.com/KhronosGroup/SPIRV-Headers into
                /tmp/pip-req-build-goyt6fp4/python/build/cmake.linux-x86_64-cpython-3.10/_deps/spirv-llvm-translator-build/SPIRV-Headers
      -- Checking for one of the modules 'SPIRV-Tools'
      CMake Warning at python/build/cmake.linux-x86_64-cpython-3.10/lib/Target/SPIRV/SPIRVToLLVMTranslator/CMakeLists.txt:119 (find_package):
        By not providing "FindSPIRV-Tools.cmake" in CMAKE_MODULE_PATH this project
        has asked CMake to find a package configuration file provided by
        "SPIRV-Tools", but CMake did not find one.

        Could not find a package configuration file provided by "SPIRV-Tools" with
        any of the following names:

          SPIRV-ToolsConfig.cmake
          spirv-tools-config.cmake

        Add the installation prefix of "SPIRV-Tools" to CMAKE_PREFIX_PATH or set
        "SPIRV-Tools_DIR" to a directory containing one of the above files.  If
        "SPIRV-Tools" provides a separate development package or SDK, be sure it
        has been installed.


      CMake Warning at python/build/cmake.linux-x86_64-cpython-3.10/lib/Target/SPIRV/SPIRVToLLVMTranslator/CMakeLists.txt:120 (find_package):
        By not providing "FindSPIRV-Tools-tools.cmake" in CMAKE_MODULE_PATH this
        project has asked CMake to find a package configuration file provided by
        "SPIRV-Tools-tools", but CMake did not find one.

        Could not find a package configuration file provided by "SPIRV-Tools-tools"
        with any of the following names:

          SPIRV-Tools-toolsConfig.cmake
          spirv-tools-tools-config.cmake

        Add the installation prefix of "SPIRV-Tools-tools" to CMAKE_PREFIX_PATH or
        set "SPIRV-Tools-tools_DIR" to a directory containing one of the above
        files.  If "SPIRV-Tools-tools" provides a separate development package or
        SDK, be sure it has been installed.


      -- SPIRV-Tools not found; project will be built without --spirv-tools-dis support.
      -- Found SPIRVToLLVMTranslator: /tmp/pip-req-build-goyt6fp4/python/build/cmake.linux-x86_64-cpython-3.10/lib/Target/SPIRV/SPIRVToLLVMTranslator
      -- Adding Python module
      -- Triton backends tuple: intel,nvidia,amd
      CMake Warning at /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/llvm/AddLLVM.cmake:1935 (message):
        LLVM_EXTERNAL_LIT set to
        /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/bin/llvm-lit, but the path does
        not exist.
      Call Stack (most recent call first):
        /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/llvm/AddLLVM.cmake:1980 (get_llvm_lit_path)
        /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/llvm/AddLLVM.cmake:2070 (add_lit_target)
        test/CMakeLists.txt:21 (add_lit_testsuite)


      -- Found Python: /root/miniforge3/envs/py310/bin/python3.10 (found version "3.10.14") found components: Interpreter
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      -- Configuring done (9.7s)
      CMake Error at /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/llvm/LLVMExports.cmake:64 (set_target_properties):
        The link interface of target "LLVMSupport" contains:

          ZLIB::ZLIB

        but the target was not found.  Possible reasons include:

          * There is a typo in the target name.
          * A find_package call is missing for an IMPORTED target.
          * An ALIAS target is missing.

      Call Stack (most recent call first):
        /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/llvm/LLVMConfig.cmake:263 (include)
        /root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib/cmake/mlir/MLIRConfig.cmake:10 (find_package)
        CMakeLists.txt:61 (find_package)


      -- Generating done (0.3s)
      CMake Warning:
        Manually-specified variables were not used by the project:

          CMAKE_VERBOSE_MAKEFILE
          LLVM_ENABLE_WERROR


      CMake Generate step failed.  Build files cannot be regenerated correctly.
      Traceback (most recent call last):
        File "/root/miniforge3/envs/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/miniforge3/envs/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/miniforge3/envs/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 583, in <module>
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 530, in run
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 300, in run
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-e15n7t9s/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 337, in run
        File "<string>", line 426, in build_extension
        File "/root/miniforge3/envs/py310/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-goyt6fp4', '-G', 'Ninja', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-e15n7t9s/overlay/bin/ninja', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DLLVM_ENABLE_WERROR=ON', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-goyt6fp4/python/build/lib.linux-x86_64-cpython-310/triton/_C', '-DTRITON_BUILD_TUTORIALS=OFF', '-DTRITON_BUILD_PYTHON_MODULE=ON', '-DPython3_EXECUTABLE:FILEPATH=/root/miniforge3/envs/py310/bin/python3.10', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DPYTHON_INCLUDE_DIRS=/root/miniforge3/envs/py310/include/python3.10', '-DTRITON_CODEGEN_BACKENDS=intel;nvidia;amd', '-DTRITON_PLUGIN_DIRS=', '-DPYBIND11_INCLUDE_DIR=/root/.triton/pybind11/pybind11-2.11.1/include', '-DLLVM_INCLUDE_DIRS=/root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/include', '-DLLVM_LIBRARY_DIR=/root/.triton/llvm/llvm-10dc3a8e-ubuntu-x64/lib', '-DCMAKE_BUILD_TYPE=TritonRelBuildWithAsserts', '-DJSON_INCLUDE_DIR=/root/.triton/json//include', '-DPYBIND11_INCLUDE_DIR=/root/.triton/pybind11/pybind11-2.11.1/include', '-DCUPTI_INCLUDE_DIR=/tmp/pip-req-build-goyt6fp4/third_party/nvidia/backend/include', '-DROCTRACER_INCLUDE_DIR=/tmp/pip-req-build-goyt6fp4/third_party/amd/backend/include']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for triton
Failed to build triton
ERROR: Could not build wheels for triton, which is required to install pyproject.toml-based projects
make: *** [Makefile:37: triton] Error 1

Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.6
Libc version: glibc-2.35

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-100-generic-x86_64-with-glibc2.35
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             224
On-line CPU(s) list:                0-223
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8480+
CPU family:                         6
Model:                              143
Thread(s) per core:                 2
Core(s) per socket:                 56
Socket(s):                          2
Stepping:                           6
Frequency boost:                    enabled
CPU max MHz:                        2001.0000
CPU min MHz:                        800.0000
BogoMIPS:                           4000.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          5.3 MiB (112 instances)
L1i cache:                          3.5 MiB (112 instances)
L2 cache:                           224 MiB (112 instances)
L3 cache:                           210 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-55,112-167
NUMA node1 CPU(s):                  56-111,168-223
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==2.0.0
[pip3] optree==0.11.0
[pip3] torch==2.5.0a0+gitdf85f34
[pip3] torchaudio==2.4.0a0+b829e93
[pip3] torchvision==0.20.0a0+bf01bab
[conda] mkl-include               2024.1.0              intel_691    intel
[conda] mkl-static                2024.1.0              intel_691    intel
[conda] numpy                     2.0.0                    pypi_0    pypi
[conda] optree                    0.11.0                   pypi_0    pypi
[conda] torch                     2.5.0a0+gitdf85f34           dev_0    <develop>
[conda] torchaudio                2.4.0a0+b829e93           dev_0    <develop>
[conda] torchvision               0.20.0a0+bf01bab           dev_0    <develop>

cc @gujinghui @EikanWang @fengyuan14 @guangyey

[Stonepia]

This is because ZLIB is missing in your system. You may need to install it using apt-get install zlib1g-dev. A similar issue is on triton-lang/triton#2687

[Stonepia]

To be specific, this issue is because the Triton source build process needs the ZLIB (for llvm). Thus, if you need to build Triton from the source, then you have to install zlib. If you install the triton by whl, then this is not needed.

Currently, when USE_XPU=1 is enabled, it will automatically build the triton from the source. Thus, the zlib should be included as a dependency.

We are WIP to enhance this logic. In the future, we will provide the nightly wheel package. By that time, when you call USE_XPU=1 make triton, it will install the triton whl from the nightly packages instead of building it from the source. That would leverage this problem.