-
Notifications
You must be signed in to change notification settings - Fork 504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
INT4 quantization not working on MI210 #154
Comments
Yeah, int4 quantization doesn't work on AMD GPUs right now. |
datagero
pushed a commit
to datagero/pytorch
that referenced
this issue
Jul 10, 2024
- Add AMD support for int4 kernel - Only supports CDNA2 and CDNA3 gpus for now - Uses `mfma_f32_16x16x16bf16` instruction for matrix multiply - Uses `v_and_or_b32` instruction and `__hfma2` instrinsic for unpacking bf16 values - Enable hipify for `__nv_bfloat16` and `__nv_bfloat162` data types - Enable int4 unit tests for CDNA2 and CDNA3 AMD gpus - Fix torchscript issues due to hipify for `__nv_bfloat16` type - TorchScript has its own implementation for bfloat16 type - Implemented in `__nv_bloat16` structure at [resource_strings.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/jit/codegen/fuser/cuda/resource_strings.h) - So, we shouldn't hipify any reference of `__nv_bfloat16` in the torchscript implementation - Hence moved the `__nv_bfloat16` direct references in `codegen.cpp` and `cuda_codegen.cpp` to `resource_strings.h` which is already exempted from hipify Fixes pytorch#124699 Fixes pytorch-labs/gpt-fast/issues/154 Co-authored-by: Nikita Shulga <[email protected]> Pull Request resolved: pytorch#129710 Approved by: https://github.com/malfet
PR merged. This issue can be closed now. |
xuhancn
pushed a commit
to xuhancn/pytorch
that referenced
this issue
Jul 25, 2024
- Add AMD support for int4 kernel - Only supports CDNA2 and CDNA3 gpus for now - Uses `mfma_f32_16x16x16bf16` instruction for matrix multiply - Uses `v_and_or_b32` instruction and `__hfma2` instrinsic for unpacking bf16 values - Enable hipify for `__nv_bfloat16` and `__nv_bfloat162` data types - Enable int4 unit tests for CDNA2 and CDNA3 AMD gpus - Fix torchscript issues due to hipify for `__nv_bfloat16` type - TorchScript has its own implementation for bfloat16 type - Implemented in `__nv_bloat16` structure at [resource_strings.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/jit/codegen/fuser/cuda/resource_strings.h) - So, we shouldn't hipify any reference of `__nv_bfloat16` in the torchscript implementation - Hence moved the `__nv_bfloat16` direct references in `codegen.cpp` and `cuda_codegen.cpp` to `resource_strings.h` which is already exempted from hipify Fixes pytorch#124699 Fixes pytorch-labs/gpt-fast/issues/154 Co-authored-by: Nikita Shulga <[email protected]> Pull Request resolved: pytorch#129710 Approved by: https://github.com/malfet
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
INT8 quantization works fine, but INT4 does not work.
![Capture](https://private-user-images.githubusercontent.com/106262476/320603101-ac10df53-860e-4da9-b51e-1ad17e3fe3c4.PNG?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzMDk2NTYsIm5iZiI6MTcyNTMwOTM1NiwicGF0aCI6Ii8xMDYyNjI0NzYvMzIwNjAzMTAxLWFjMTBkZjUzLTg2MGUtNGRhOS1iNTFlLTFhZDE3ZTNmZTNjNC5QTkc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTAyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkwMlQyMDM1NTZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ZjYwYzBkMGVjNDk1NWM3MjE1Yzk3NzczN2NkMGQxODQ5MmE1NTA5N2Y1NDRmZTNhZWY2YmJiNGQwZmJhNDA0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.nXUwifQ05JjXkO4TSBZUD3LGgYHKuDvCwT1jhD3-62I)
The text was updated successfully, but these errors were encountered: