Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torch.compile graph break with unsupported LOAD_BUILD_CLASS #128942

Open
sujoysaraswati opened this issue Jun 18, 2024 · 0 comments
Open

torch.compile graph break with unsupported LOAD_BUILD_CLASS #128942

sujoysaraswati opened this issue Jun 18, 2024 · 0 comments
Labels
dynamo-must-fix These bugs affect TorchDynamo reliability. feature A request for a proper, new feature. module: dynamo module: graph breaks oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@sujoysaraswati
Copy link
Contributor

sujoysaraswati commented Jun 18, 2024 •
edited by pytorch-bot bot
Loading

🐛 Describe the bug

torch.compile produces a graph break when a class is defined within a method that is compiled. Example:

import torch
import torch._dynamo as dynamo
 
dynamo.reset()
 
def toy_example(inp):
    class A:
        def __init__():
            pass
        def something():
            pass
    return inp + 2
 
compiled_fn = torch.compile(toy_example, backend="inductor")

inp = torch.arange(2)
r1 = compiled_fn(inp)
print(f"r1 = {r1}")

This creates a graph break with the log:

V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0] torchdynamo start compiling toy_example /content/load_build_class.py:6, stack (elided 6 frames):
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]   File "/content/load_build_class.py", line 17, in <module>
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]     r1 = compiled_fn(inp)
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 433, in _fn
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]     return fn(*args, **kwargs)
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 1116, in __call__
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0]     return self._torchdynamo_orig_callable(
V0618 09:39:34.526000 136241583071872 torch/_dynamo/convert_frame.py:776] [0/0] 
I0618 09:39:34.552000 136241583071872 torch/_dynamo/logging.py:56] [0/0] Step 1: torchdynamo start tracing toy_example /content/load_build_class.py:6
V0618 09:39:34.552000 136241583071872 torch/fx/experimental/symbolic_shapes.py:2541] [0/0] create_env
V0618 09:39:34.568000 136241583071872 torch/_dynamo/symbolic_convert.py:786] [0/0] [__trace_source] TRACE starts_line /content/load_build_class.py:7 in toy_example (toy_example.A)
V0618 09:39:34.568000 136241583071872 torch/_dynamo/symbolic_convert.py:786] [0/0] [__trace_source]         class A:
V0618 09:39:34.576000 136241583071872 torch/_dynamo/symbolic_convert.py:809] [0/0] [__trace_bytecode] TRACE LOAD_BUILD_CLASS None []
V0618 09:39:34.576000 136241583071872 torch/_dynamo/symbolic_convert.py:825] [0/0] empty checkpoint
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks] Graph break: skip: from user code at:
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/content/load_build_class.py", line 7, in toy_example
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     class A:
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks] Traceback (most recent call last):
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 948, in __call__
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     result = self._inner_convert(
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 472, in __call__
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     return _compile(
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_utils_internal.py", line 84, in wrapper_function
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     return StrobelightCompileTimeProfiler.profile_compile_time(
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     return func(*args, **kwargs)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/lib/python3.10/contextlib.py", line 79, in inner
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     return func(*args, **kwds)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 817, in _compile
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     guarded_code = compile_inner(code, one_graph, hooks, transform)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 231, in time_wrapper
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     r = func(*args, **kwargs)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 636, in compile_inner
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     out_code = transform_code_object(code, transform)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/bytecode_transformation.py", line 1185, in transform_code_object
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     transformations(instructions, code_options)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 178, in _fn
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     return fn(*args, **kwargs)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 582, in transform
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     tracer.run()
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 2462, in run
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     super().run()
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 904, in run
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     while self.step():
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 816, in step
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     self.dispatch_table[inst.opcode](self, inst)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 641, in _missing
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     unimplemented(f"missing: {opname}")
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]   File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/exc.py", line 221, in unimplemented
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     raise Unsupported(msg)
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks] torch._dynamo.exc.Unsupported: missing: LOAD_BUILD_CLASS
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks] 
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks] from user code:
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]    File "/content/load_build_class.py", line 7, in toy_example
V0618 09:39:34.577000 136241583071872 torch/_dynamo/convert_frame.py:989] [0/0] [__graph_breaks]     class A:

Versions

PyTorch version: 2.5.0.dev20240617+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.27.9
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.1.85+-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU @ 2.00GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
Stepping: 3
BogoMIPS: 4000.39
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32 KiB (1 instance)
L1i cache: 32 KiB (1 instance)
L2 cache: 1 MiB (1 instance)
L3 cache: 38.5 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0,1
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Vulnerable; SMT Host state unknown
Vulnerability Meltdown: Vulnerable
Vulnerability Mmio stale data: Vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Vulnerable
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Vulnerable (Syscall hardening enabled)
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Vulnerable

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pytorch-triton==3.0.0+45fff310c8
[pip3] torch==2.5.0.dev20240617+cu121
[pip3] torchaudio==2.4.0.dev20240617+cu121
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.18.0
[pip3] torchvision==0.19.0.dev20240617+cu121
[pip3] triton==2.3.0
[conda] Could not collect

cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @amjames @rec @anijain2305
@masnesral masnesral added feature A request for a proper, new feature. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module module: dynamo module: graph breaks labels Jun 19, 2024
@jerrychenhf jerrychenhf mentioned this issue Aug 26, 2024
Closed
@XuehaiPan XuehaiPan added the dynamo-must-fix These bugs affect TorchDynamo reliability. label Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dynamo-must-fix These bugs affect TorchDynamo reliability. feature A request for a proper, new feature. module: dynamo module: graph breaks oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@malfet @XuehaiPan @sujoysaraswati @masnesral