[Deprecated][inductor] switch CppCodeCache to new cpp_builder.

[xuhancn]

This PR is the implemention of #124245 step 2.1

1. Switch CppCodeCache to new CPP builder.
2. Switch AotCodeCompiler to new CPP builder.
3. Switch HalideCodeCache to new CPP builder.

TODO:
Current still need to optimize code structure and circle import. I will do this work in further PR. I have did the exploration: #128330

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128303

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (15 Unrelated Failures)

As of commit 033835f with merge base 5fba5d8 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• linux-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-build / build (gh) (matched linux rule in flaky-rules.json)
  Error response from daemon: Head "https://registry-1.docker.io/v2/pytorch/libtorch-cxx11-builder/manifests/cpu-main": received unexpected HTTP status: 503 Service Unavailable
• linux-binary-libtorch-pre-cxx11 / libtorch-cpu-shared-with-deps-pre-cxx11-build / build (gh) (matched linux rule in flaky-rules.json)
  Error response from daemon: Head "https://registry-1.docker.io/v2/pytorch/manylinux-builder/manifests/cpu-main": received unexpected HTTP status: 503 Service Unavailable
• linux-binary-manywheel / manywheel-py3_8-cuda11_8-build / build (gh) (matched linux rule in flaky-rules.json)
  Error response from daemon: Head "https://registry-1.docker.io/v2/pytorch/manylinux-builder/manifests/cuda11.8-main": received unexpected HTTP status: 503 Service Unavailable
• linux-binary-manywheel / manywheel-py3_8-cuda12_1-build / build (gh) (matched linux rule in flaky-rules.json)
  Error response from daemon: Head "https://registry-1.docker.io/v2/pytorch/manylinux-builder/manifests/cuda12.1-main": received unexpected HTTP status: 503 Service Unavailable
• linux-binary-manywheel / manywheel-py3_8-cuda12_4-build / build (gh) (matched linux rule in flaky-rules.json)
  Error response from daemon: Head "https://registry-1.docker.io/v2/pytorch/manylinux-builder/manifests/cuda12.4-main": received unexpected HTTP status: 503 Service Unavailable
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh) (similar failure)
  Process completed with exit code 1.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
  'Test'
• inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
  dynamo/test_inline_inbuilt_nn_modules.py::InlineInbuiltNNModulesMiscTests::test_outside_linear_module_free_inline_inbuilt_nn_modules

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128903)
  hf_BigBird
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128902)
  hf_BigBird
• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128901)
  hf_BigBird
• inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2, unstable) (gh) (#128871)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
• inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128929)
  hf_BigBird
• inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128931)
  hf_BigBird
• inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#128932)
  hf_BigBird

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[xuhancn]

@pytorchbot rebase

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Tried to rebase and push PR #128303, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

[xuhancn]

@pytorchbot rebase -b main

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

[pytorchmergebot]

Successfully rebased xu_switch_to_new_cpp_builder onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout xu_switch_to_new_cpp_builder && git pull --rebase)

[desertfire]

internal-external out of sync. Will need to wait for a while.

[facebook-github-bot]

@desertfire has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[xuhancn]

Hi @desertfire , could you please let me know the import status?

[jgong5]

Can you remove the unused or commented out code?

restructure code to solve it in next PR, let's land it firstly.

[facebook-github-bot]

@desertfire has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[xuhancn]

@desertfire Please help on validate new two fix in fb_code:

1. 'CUDA_HOME environment variable is not set.
2. error: no such file or directory: 'cd7r7ustk6uz5y6ctrjm4iuakbwa6ov7qiohl7qzqvx66nztzu2a.cpp'

[xuhancn]

@pytorchmergebot rebase -b main

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

[pytorchmergebot]

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/main pull/128303/head returned non-zero exit code 1

Rebasing (1/10)
Rebasing (2/10)
Rebasing (3/10)
Rebasing (4/10)
Auto-merging torch/_inductor/codecache.py
Auto-merging torch/_inductor/cpp_builder.py
CONFLICT (content): Merge conflict in torch/_inductor/cpp_builder.py
error: could not apply 891cf9e1b9... switch AotCodeCompiler library linker to new cpp_builder.
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 891cf9e1b9... switch AotCodeCompiler library linker to new cpp_builder.

Raised by https://github.com/pytorch/pytorch/actions/runs/9599403273

[xuhancn]

@desertfire this PR seems damaged, please move to new one: #129144