Two missing header files in package list.

[cchheennhhaaoo]

Hi,

We have registered a backend of pytorch and maintain a framework based on it. When building our framework, we found 2 header files are missing which may cause building failed.

1. "include/torch/csrc/generic/*.h" is not in package list while "include/torch/csrc/*.h" is in. It may cause error when using "include/torch/csrc/Storage.h" since it requires "include/torch/csrc/generic/Storage.h".
2. "torch/csrc/distributed/rpc/*.h" is also not in package list, it may cause error when using "./torch/csrc/jit/python/pybind_utils.h" since this file includes "torch/csrc/distributed/rpc/py_rref.h".

Could you please add "include/torch/csrc/generic/*.h" and "torch/csrc/distributed/rpc/*.h" to package list in setup.py?

cc @malfet @seemethere @jbschlosser

[malfet]

Removing releng and module: build labels and adding "needs design"
"torch/csrc/Storage.h" depends on "torch/csrc/generic/Storage.h" at least since 1.4

So perhaps the better solution would be not export Storage.h, as it looks like an internal, rather than public header

[chengjunlu]

So perhaps the better solution would be not export Storage.h, as it looks like an internal, rather than public header

Even with the untyped storage #62030, the extension still need to register the inter-op for ByteStorage of different backend. Like: The THPInsertStorageCopyFunction API.

Or there is other way for the extension for the same purpose?

[chengjunlu]

@malfet
The latest pytorch has refactor the THP Storage to the untyped storage.
But the untyped storage (legacy ByteStorage) still has some coupled code in CUDA and CPU.
Do you think it is possible to decouple the code further and reuse the code for other backend extension out tree?

By doing so we can remove all the legacy things in generic/*

[cchheennhhaaoo]

Hi, @malfet, any suggestion about @chengjunlu's comments?

[cchheennhhaaoo]

@malfet @ejguan Could someone help with it?

[chengjunlu]

Hi all,
After investigate the latest pytorch. All the missed header issue has been solved. No longer need this requirements.
@cchheennhhaaoo Please close this.

Thanks
John

[access2rohit]

I am also facing this issues while linking using libtorch-1.12.1

torch/include/torch/csrc/jit/python/pybind_utils.h:29:10: fatal error: torch/csrc/distributed/rpc/py_rref.h: No such file or directory

Any updates on when this will be fixed ?

[chengjunlu]

I am also facing this issues while linking using libtorch-1.12.1

torch/include/torch/csrc/jit/python/pybind_utils.h:29:10: fatal error: torch/csrc/distributed/rpc/py_rref.h: No such file or directory

Any updates on when this will be fixed ?

This issue seems fixed in master pytorch. Maybe available in pytorch 1.13.