[WIP] refactor distributed code

[guangyey]

Stack from ghstack (oldest at bottom):

• -> [WIP] refactor distributed code #132371
• [WIP] Introduce a device-agnostic runtime API design #132204

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132371

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 21 New Failures, 1 Pending, 1 Unrelated Failure

As of commit a45a42c with merge base a147fa5 ():

NEW FAILURES - The following jobs have failed:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
  distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False
• inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
  distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False
• Lint / lintrunner-noclang / linux-job (gh)
  >>> Lint for torch/distributed/tensor/parallel/fsdp.py:
• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) (gh)
  distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group
• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) (gh)
  distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, amz2023.linux.12xlarge) (gh)
  /var/lib/jenkins/workspace/xla/torch_xla/csrc/BUILD:31:17: Compiling torch_xla/csrc/tensor_impl.cpp failed: (Exit 1): gcc failed: error executing command (from target //torch_xla/csrc:tensor) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 421 arguments skipped)
• pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.11-clang10 / test (default, 1, 4, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.11-clang10 / test (dynamo, 1, 3, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.12-clang10 / test (default, 1, 4, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.12-clang10 / test (dynamo, 1, 3, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 1, 3, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.8-clang10 / test (default, 1, 4, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 3, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, amz2023.linux.4xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 4, amz2023.linux.2xlarge) (gh)
  test_cpp_extensions_mtia_backend.py::TestCppExtensionMTIABackend::test_device_context
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, amz2023.linux.2xlarge) (gh)
  distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, amz2023.linux.2xlarge) (gh)
  distributed/_tensor/test_dtensor_ops.py::TestDTensorOpsCPU::test_dtensor_op_db_H_cpu_float32

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) (gh) (trunk failure)
  distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention

This comment was automatically generated by Dr. CI and updates every 15 minutes.