Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(EasyDict) #120219

Open
adonnini opened this issue Feb 10, 2024 · 77 comments
Assignees
Labels
dynamo-must-fix These bugs affect TorchDynamo reliability. export-triaged This tag is used to tag issues that have been looked by PT2 Export team and determined the next step oncall: export triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@adonnini
Copy link

adonnini commented Feb 10, 2024

Hi,
I am trying run for inference another model on an android device using executorch. Execution fails with a
torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(EasyDict)
error.
Below, you will find the relevant portion of the traceback log.
The model I am trying to process is largely based on this model
https://github.com/mie-lab/location-prediction/tree/main
I did not modify the modules involved in the execution error.

I re-ran the code using

TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1

The log and traceback log produced are ver long. Please let me know if you want me to post them.

Please let me know if you need additional information.
Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 569, in proxy_args_kwargs
    proxy_args = tuple(arg.as_proxy() for arg in args)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 569, in <genexpr>
    proxy_args = tuple(arg.as_proxy() for arg in args)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/base.py", line 273, in as_proxy
    raise NotImplementedError(str(self))
NotImplementedError: UserDefinedObjectVariable(EasyDict)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 67, in <module>
    res_single = single_run(train_loader, val_loader, test_loader, config, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 23, in single_run
    model, perf = get_trainedNets(config, model, train_loader, val_loader, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/utils.py", line 47, in get_trainedNets
    best_model, performance = trainNet(config, model, train_loader, val_loader, device, log_dir=log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/train.py", line 312, in trainNet
    pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (config, config.total_loc_num, device))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py", line 133, in capture_pre_autograd_graph
    m = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1268, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 417, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 580, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 384, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 643, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 247, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 524, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 489, in transform
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2098, in run
    super().run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/nn_module.py", line 329, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1231, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 301, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 255, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 84, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/nn_module.py", line 307, in call_function
    *proxy_args_kwargs(args, kwargs),
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 576, in proxy_args_kwargs
    raise unimplemented(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(EasyDict) 

from user code:
   File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/MHSA.py", line 36, in forward
    emb = self.Embedding(src, context_dict)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/embed.py", line 149, in forward
    emb = self.emb_loc(src)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4 @aakhundov

@kimishpatel
Copy link
Contributor

@angelayi can you help take a look?

@kimishpatel
Copy link
Contributor

So it seems that EasyDict is not dynamo traceable.

@kimishpatel
Copy link
Contributor

@angelayi I think EasyDIct is from here https://github.com/makinacorpus/easydict/blob/master/easydict/__init__.py, not sure why it is not traceable

@adonnini
Copy link
Author

adonnini commented Feb 14, 2024

Does the fact that EasyDict is not dynamo traceable make it impossible to diagnose the problem reported in the traceback log above?

I tried using AttrDict (https://pypi.org/project/attrdict/) instead of EasyDict

Execution failed with the same error

torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(AttrDict)

@iseeyuan iseeyuan added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 16, 2024
@adonnini
Copy link
Author

I also tried using

class AttrDict(dict):
    def __init__(self, *args, **kwargs):
        super(AttrDict, self).__init__(*args, **kwargs)
        self.__dict__ = self

i.e. not using any external library. Execution failed with the same error

torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(AttrDict) 

@kimishpatel kimishpatel transferred this issue from pytorch/executorch Feb 20, 2024
@kimishpatel
Copy link
Contributor

Assigned to export

@suo
Copy link
Member

suo commented Feb 20, 2024

pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (config, config.total_loc_num, device))

Slightly tangential: curious why you are using this API to capture the graph, as it is a private API (in torch._export).

@kimishpatel
Copy link
Contributor

pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (config, config.total_loc_num, device))

Slightly tangential: curious why you are using this API to capture the graph, as it is a private API (in torch._export).

@suo so on executorch side the export flow is, atm:

  1. capture_pre_autograd
  2. to_edge.

Before export was able to produce pre-dispatch IR, for the needs to quantization we needed pre-dispatch IR. For the reason 1 was used. In order to not have two different export workflows, for quant and executorch, it was decided that we will have two step API for edge specific export. Step 1 will provide pre-dispatch IR and passes, such as quant, that need pre-dispatch IR can work on it. Later edge will apply decomps/export to produce core-aten IR. 1 was is supposed to be replaced with equivalent export API but @angelayi might know the status. Then we can start replacing 1 with the new export API

@zhxchen17
Copy link
Contributor

cc @tugsbayasgalan who is working on migrating capture_pre_autograd_graph to torch.export.

@avikchaudhuri avikchaudhuri added the export-triage-review This tag is used to tag issues that have been looked by PT2 Export team and pending discussions. label Feb 20, 2024
@avikchaudhuri
Copy link
Contributor

@yanboliang offered to look into the the dynamo coverage issue. Meanwhile @kimishpatel any chance you can use non-strict (i.e., strict=False) to unblock?

@avikchaudhuri avikchaudhuri added export-triaged This tag is used to tag issues that have been looked by PT2 Export team and determined the next step and removed export-triage-review This tag is used to tag issues that have been looked by PT2 Export team and pending discussions. labels Feb 20, 2024
@kimishpatel
Copy link
Contributor

@avikchaudhuri it is coming from call to export from capture_pre_auto_grad. does it support nonstrict? @adonnini can you try that?

@adonnini
Copy link
Author

@kimishpatel I added strict=False to
export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
When I ran the code, it failed at this line
lowered_module = edge_program.to_backend(XnnpackPartitioner())
Telling me it could not find flatc

Then, I had the bright idea (not really) to update executorch installation to the latest nightly build (20240209).
Now, execution fails with a brand new error. Please see traceback log below.
I searched for information on this error. I did not come up with anything I found useful, not to say that there wasn't relevant information. I just could not decipher whether it was applicable to my case

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 21, in <module>
    from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/__init__.py", line 8, in <module>
    from .partition.xnnpack_partitioner import (
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/xnnpack_partitioner.py", line 82, in <module>
    class XnnpackOperatorSupport(OperatorSupportBase):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/xnnpack_partitioner.py", line 236, in XnnpackOperatorSupport
    @_constraint(exir_ops.edge.aten.mean.dim)
TypeError: 'staticmethod' object is not callable

@kimishpatel
Copy link
Contributor

@adonnini so at least with strict=False you are able to get around the export issue and fails elsewhere, right? @mcr229 for xnnpack

@adonnini
Copy link
Author

I hope that is the case. It should be. Should I wait to hear from @mcr229 with regards to this latest error? Please let me know if there is anything I can/should do.
Thanks

@kimishpatel
Copy link
Contributor

Sorry to ask you to create another issue, but we can go back to executorch repo and create issue there and mark this as "unblocked" since we dont have proper solution but have a workaround. If you dont mind doing that, tag @mcr229 on the issue

@adonnini
Copy link
Author

No problem at all. I created a new issue in executorch repo, pytorch/executorch#2025

@angelayi angelayi added module: dynamo dynamo-must-fix These bugs affect TorchDynamo reliability. labels Feb 22, 2024
@adonnini
Copy link
Author

adonnini commented Mar 9, 2024

Hi,

I cloned the main executorch branch (git clone --branch main https://github.com/pytorch/executorch.git).

Ran executorch installation/set-up following the updated instructions in
https://pytorch.org/executorch/stable/getting-started-setup.html

I set strict=False in
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (config, config.total_loc_num, device), strict=False)

then ran the training module for my model including the executorch code.

Execution failed once again with
NotImplementedError: UserDefinedObjectVariable(EasyDict)

Below you will find the tracebak log

Please let me know what I should do next to help resolve this problem.

Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 621, in proxy_args_kwargs
    proxy_args = tuple(arg.as_proxy() for arg in args)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 621, in <genexpr>
    proxy_args = tuple(arg.as_proxy() for arg in args)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 304, in as_proxy
    raise NotImplementedError(str(self))
NotImplementedError: UserDefinedObjectVariable(EasyDict)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 68, in <module>
    res_single = single_run(train_loader, val_loader, test_loader, config, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 23, in single_run
    model, perf = get_trainedNets(config, model, train_loader, val_loader, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/utils.py", line 47, in get_trainedNets
    best_model, performance = trainNet(config, model, train_loader, val_loader, device, log_dir=log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/train.py", line 316, in trainNet
    pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (config, config.total_loc_num, device))
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_export/__init__.py", line 168, in capture_pre_autograd_graph
    m = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1296, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1536, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 437, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1536, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 919, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 675, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 534, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 165, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 499, in transform
    tracer.run()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2175, in run
    super().run()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 831, in run
    and self.step()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 794, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1245, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 695, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 336, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 701, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2311, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2425, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 831, in run
    and self.step()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 794, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1286, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 695, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 334, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 288, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 89, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 701, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2311, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2425, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 831, in run
    and self.step()
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 794, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1245, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 695, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 314, in call_function
    *proxy_args_kwargs(args, kwargs),
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 628, in proxy_args_kwargs
    raise unimplemented(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 190, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function args: UserDefinedObjectVariable(EasyDict) 

from user code:
   File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/MHSA.py", line 36, in forward
    emb = self.Embedding(src, context_dict)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1536, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/embed.py", line 149, in forward
    emb = self.emb_loc(src)

@angelayi
Copy link
Contributor

angelayi commented May 7, 2024

Hmm, seems like a bug on our part, sorry about that! Sorry, this is my mistake -- looks like we don't support device/dtype inputs yet...
Actually, to unblock you for now, is it possible to not pass in the device as an input?

@adonnini
Copy link
Author

adonnini commented May 8, 2024

@angelayi I understand. I'll take a look. It should be possible. Is device/dtype support planned? Any idea about the timeline?
Thanks

@adonnini
Copy link
Author

adonnini commented May 8, 2024

@angelayi I temporarily removed the need to specify the device. Code execution fails producing the traceback log reported below.

The error should not have anything to do with the removal of the device as a parameter. Based on my (limited) experience with torchscript it might have to do with limitations in executorch support for models (this is just a guess).

What do you think?

Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 68, in <module>
    res_single = single_run(train_loader, val_loader, test_loader, config, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 23, in single_run
    model, perf = get_trainedNets(config, model, train_loader, val_loader, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/utils.py", line 47, in get_trainedNets
    best_model, performance = trainNet(config, model, train_loader, val_loader, device, log_dir=log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/train.py", line 386, in trainNet
    pre_autograd_aten_dialect = torch.export._trace._export(m, (config, total_loc_num), strict=False, pre_dispatch=True)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 833, in wrapper
    raise e
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 816, in wrapper
    ep = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/exported_program.py", line 85, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 994, in _export
    ep_non_strict = _export_non_strict(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 520, in _export_non_strict
    gm, graph_signature = transform(aot_export_module)(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 955, in _aot_export_non_strict
    gm, sig = aot_export(wrapped_mod, args, kwargs=kwargs, **flags)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1135, in aot_export_module
    fx_g, metadata, in_spec, out_spec = _aot_export_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1354, in _aot_export_function
    fx_g, meta = create_aot_dispatcher_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 558, in create_aot_dispatcher_function
    fw_metadata = run_functionalized_fw_and_collect_metadata(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/collect_metadata_analysis.py", line 151, in inner
    flat_f_outs = f(*flat_f_args)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 176, in flat_fn
    tree_out = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 709, in functional_call
    out = mod(*args[params_len:], **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 942, in forward
    tree_out = self._export_root(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/MHSA.py", line 39, in forward
    emb = self.Embedding(src, context_dict)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/embed.py", line 149, in forward
    emb = self.emb_loc(src)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward
    return F.embedding(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/functional.py", line 2235, in embedding
    return handle_torch_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/overrides.py", line 1626, in handle_torch_function
    result = mode.__torch_function__(public_api, types, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_safeguard.py", line 43, in __torch_function__
    return func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/functional.py", line 2266, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not dict

@angelayi
Copy link
Contributor

angelayi commented May 8, 2024

@adonnini Just checking, does your model work when you run it eagerly? It's erroring because it says you're passing in a dict as the second argument to torch.embedding when it expects a tensor.

@adonnini
Copy link
Author

adonnini commented May 8, 2024

@angelayi the model works fine when I run it eagerly. In fact, the argument passed in the model is a dict. The model takes in input dataset and a dict as the two input parameters. And, this works without problems when running eagerly.
Please let me know if you need any other information or need me to do something.
Thanks

@angelayi
Copy link
Contributor

angelayi commented May 8, 2024

@adonnini Sorry, it's not clear to me what's wrong just looking at the errors... Is it possible if you could send a mini repro?

@adonnini
Copy link
Author

adonnini commented May 9, 2024

@angelayi I'll do that. It may take a few days as I need to clean up some of the code and I am in the middle of a couple of other projects.
By the way, I am using two models to predict location. For the other one, I was able to produce the lowered model using executorch.
Thanks

@adonnini
Copy link
Author

adonnini commented May 10, 2024

@angelayi setting up a mini repo is not going to be as straightforward as I thought because I have a lot of stuff in progress in the code. Would it be possible to work off the original repo of the model at
https://github.com/mie-lab/location-prediction/tree/main
I can answer any questions you may have and help you out as needed.
Thanks

@adonnini
Copy link
Author

@angelayi I know you are very busy. I hope I am not bothering you (too much). Will you be able to take a look at the repo of the model I am using to see if you can gain a better understanding of why export expects a tensor when in fact the model passes a tensor and a dictionary to the forward method in question?

The code in question is this:

    def forward(self, src, context_dict) -> Tensor:
        # print("LocationPrediction - models - MHSA - about to run AllEmbedding --- ")
        emb = self.Embedding(src, context_dict)
        seq_len = context_dict["len"]

        # positional encoding, dropout performed inside
        src_mask = self._generate_square_subsequent_mask(src.shape[0])
        src_padding_mask = (src == 0).transpose(0, 1)
        # src_mask = self._generate_square_subsequent_mask(src.shape[0]).to(device)
        # src_padding_mask = (src == 0).transpose(0, 1).to(device)
        out = self.encoder(emb, mask=src_mask, src_key_padding_mask=src_padding_mask)

        # only take the last timestep
        out = out.gather(
            0,
            seq_len.view([1, -1, 1]).expand([1, out.shape[1], out.shape[-1]]) - 1,
        ).squeeze(0)

        return self.FC(out, context_dict["user"])

where

        self.FC = FullyConnected(self.d_input, config, if_residual_layer=True, total_loc_num=total_loc_num)

and

        # encoder
        encoder_layer = torch.nn.TransformerEncoderLayer(
            self.d_input, nhead=config.nhead, activation="gelu", dim_feedforward=config.dim_feedforward
        )
        encoder_norm = torch.nn.LayerNorm(self.d_input)
        self.encoder = torch.nn.TransformerEncoder(
            encoder_layer=encoder_layer,
            num_layers=config.num_encoder_layers,
            norm=encoder_norm,
        )

and

        self.Embedding = AllEmbedding(self.d_input, config, total_loc_num)

@angelayi
Copy link
Contributor

angelayi commented May 13, 2024

@adonnini Can you share the script you use to export this model? I've cloned the repo but I want to make sure our setup is the same.

@adonnini
Copy link
Author

@angelayi You will find the code below. I hope I understood your request correctly. Please let me know if you meant something different.
Thanks

from easydict import EasyDict as edict

import executorch
import executorch.exir as exir
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.exir import ExecutorchBackendConfig, EdgeProgramManager, to_edge
from torch._export import capture_pre_autograd_graph
from torch.export import export, ExportedProgram, dynamic_dim
import torch.export._trace

import logging
from models.MHSA import TransEncoder

config = load_config("./config/geolife/transformer.yml")

config = edict(config)

total_loc_num = config.total_loc_num

m = TransEncoder(config, total_loc_num).to(device)
m.eval()

pytree.register_pytree_node(edict,
    flatten_fn=_dict_flatten,
    unflatten_fn=_dict_unflatten,
    serialized_type_name="EasyDict",
    flatten_with_keys_fn=_dict_flatten_with_keys
    )

pre_autograd_aten_dialect = torch.export._trace._export(m, (config, total_loc_num), strict=False, pre_dispatch=True)
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (config, total_loc_num, device), strict=False)
edge_program: EdgeProgramManager = to_edge(aten_dialect)
to_be_lowered_module = edge_program.exported_program()

lowered_module = edge_program.to_backend(XnnpackPartitioner())

save_path = save_path = "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/loweredModels/tpt_delegate.pte"
with open(save_path, "wb") as f:
    f.write(lowered_module.to_executorch().buffer)


def load_config(path="./config/geolife/ind_transformer.yml"):
        """
        Loads config file:
        Args:
            path (str): path to the config file
        Returns:
            config (dict): dictionary of the configuration parameters, merge sub_dicts
        """

        print("LocationPrediction - utils - train.py - load_config - path - ",path)

        with open(path, "r") as f:
            cfg = yaml.safe_load(f)

        config = dict()

        print("LocationPrediction - utils - train.py - load_config - config - ",config)

        for _, value in cfg.items():
            for k, v in value.items():
                config[k] = v

        return config

def _dict_flatten(d):
    return list(d.values()), list(d.keys())

def _dict_flatten_with_keys(d):
    values, context = _dict_flatten(d)
    return [(pytree.MappingKey(k), v) for k, v in zip(context, values)], context

def _dict_unflatten(values, context):
    return dict(zip(context, values))

@adonnini
Copy link
Author

@angelayi Sorry to bother you. I know you are very busy. Did you get a chance to take a look at the repository for the model I am using and my latest note above?
Please let me know if you have any questions or need any additional information.
Thanks

@adonnini
Copy link
Author

@angelayi I was able to produce a lowered model using torchscript and load it successfully in my Android app using the pytorch mobile runtime. This is not a solution to this issue as executorch is the runtime environment I need/want to use. I did it to see if there was a problem with the model. I think this test shows that there probably isn't.
Please let me know if you need any other information in order to reproduce the issue.
Thanks

@adonnini
Copy link
Author

@angelayi did you get a chance to duplicate my set-up and try to reproduce the issue? Please let me know if you need additional information and I can help in any way. Thanks.

@adonnini
Copy link
Author

adonnini commented Jun 1, 2024

@angelayi

I set up a public repository with a copy of my set-up. Here is a link to the README.MD

https://github.com/adonnini/LocationPredictionContextQ/blob/main/README.md

Please let me know if you have problems accessing the repository or have any other problems.

As I say in the Notes section, please keep in mind that I come from the Java world and am a Python beginner.

I hope this helps.

Thanks

@adonnini
Copy link
Author

adonnini commented Jun 5, 2024

@angelayi Here I am bugging you again! Sorry. When do you think you will have a chance to reproduce this error directly using my set-up (in the repository I created)?
Please let me know if you encounter any issue when accessing the repository and running the code.
Thanks

@adonnini
Copy link
Author

@angelayi Could you please let me know if you will not be able to work on this issue any longer? Thanks

@angelayi
Copy link
Contributor

angelayi commented Jun 13, 2024

Hi @adonnini, getting the repro to work is taking longer than expected. If anyone from the ExecuTorch team has bandwidth to take a look and figure out a minimum repro, that would be great! @kimishpatel

@adonnini
Copy link
Author

@angelayi I am sorry to hear this. You are having a hard time running the code in my set-up?
Perhaps, I can help. What are the problems you are running into?
Are you not able to reproduce the error I have been running into by following these instructions?

1)    If there is a folder called temp in /data delete it.
 2)   In the working directory, execute the main.py config/geolife/transformer.yml script.

Thanks

@adonnini
Copy link
Author

@angelayi I re-read your message above. I am not sure I understand what you mean by "figure out a minimum repro". the only script you need to run is main.py following the instructions in the readme.md file in the repository. I think tat setting up executorch is necessary and unavoidable.
I am here to help in any way I can.
Thanks

@adonnini
Copy link
Author

@angelayi Were you able to make any progress in getting the repro to work? As you mentioned more than one month ago, you were having problems.
Please let me know if you have any questions, and I can help in any way.
Thanks

@adonnini
Copy link
Author

@anijain2305 Hi do the changes you made to the issue three days ago mean that nobody will be working on its resolution any longer?
Please let me know as this issue is blocking some of my work and I will need to plan accordingly.
Thanks

@anijain2305
Copy link
Contributor

@adonnini No, I was just doing a triage of open Dynamo issues. This is an export issue, so removed the Dynamo tags.

But I think you should look at this comment - #122340 (comment)

This other issue seems very relevant to this one.

@adonnini
Copy link
Author

@anijain2305 Thanks. I took a look at the comment you reference above.
Forgive me. I am not sure why you say it is very relevant to the issue I have run into.
Below, you will find my code. It's the result of many exchanges with your colleagues, lately mostly with @angelayi .

After making all the changes suggested by your colleagues, the traceback produced when I run the code has changed to the one you will find below.

Could you please let me know why you think the comment you referenced is relevant and what you think I should do next? Please let me know if you need anything from me.

Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 68, in <module>
    res_single = single_run(train_loader, val_loader, test_loader, config, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/main.py", line 23, in single_run
    model, perf = get_trainedNets(config, model, train_loader, val_loader, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/utils.py", line 47, in get_trainedNets
    best_model, performance = trainNet(config, model, train_loader, val_loader, device, log_dir=log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/utils/train.py", line 386, in trainNet
    pre_autograd_aten_dialect = torch.export._trace._export(m, (config, total_loc_num), strict=False, pre_dispatch=True)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 833, in wrapper
    raise e
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 816, in wrapper
    ep = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/exported_program.py", line 85, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 994, in _export
    ep_non_strict = _export_non_strict(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 520, in _export_non_strict
    gm, graph_signature = transform(aot_export_module)(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 955, in _aot_export_non_strict
    gm, sig = aot_export(wrapped_mod, args, kwargs=kwargs, **flags)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1135, in aot_export_module
    fx_g, metadata, in_spec, out_spec = _aot_export_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1354, in _aot_export_function
    fx_g, meta = create_aot_dispatcher_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 558, in create_aot_dispatcher_function
    fw_metadata = run_functionalized_fw_and_collect_metadata(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/collect_metadata_analysis.py", line 151, in inner
    flat_f_outs = f(*flat_f_args)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 176, in flat_fn
    tree_out = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 709, in functional_call
    out = mod(*args[params_len:], **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 942, in forward
    tree_out = self._export_root(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/MHSA.py", line 39, in forward
    emb = self.Embedding(src, context_dict)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/models/embed.py", line 149, in forward
    emb = self.emb_loc(src)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward
    return F.embedding(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/functional.py", line 2235, in embedding
    return handle_torch_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/overrides.py", line 1626, in handle_torch_function
    result = mode.__torch_function__(public_api, types, args, kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_safeguard.py", line 43, in __torch_function__
    return func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/functional.py", line 2266, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not dict

CODE

from easydict import EasyDict as edict

import executorch
import executorch.exir as exir
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.exir import ExecutorchBackendConfig, EdgeProgramManager, to_edge
from torch._export import capture_pre_autograd_graph
from torch.export import export, ExportedProgram, dynamic_dim
import torch.export._trace

import logging
from models.MHSA import TransEncoder

config = load_config("./config/geolife/transformer.yml")

config = edict(config)

total_loc_num = config.total_loc_num

m = TransEncoder(config, total_loc_num).to(device)
m.eval()

pytree.register_pytree_node(edict,
    flatten_fn=_dict_flatten,
    unflatten_fn=_dict_unflatten,
    serialized_type_name="EasyDict",
    flatten_with_keys_fn=_dict_flatten_with_keys
    )

pre_autograd_aten_dialect = torch.export._trace._export(m, (config, total_loc_num), strict=False, pre_dispatch=True)
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (config, total_loc_num, device), strict=False)
edge_program: EdgeProgramManager = to_edge(aten_dialect)
to_be_lowered_module = edge_program.exported_program()

lowered_module = edge_program.to_backend(XnnpackPartitioner())

save_path = save_path = "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPrediction/loweredModels/tpt_delegate.pte"
with open(save_path, "wb") as f:
    f.write(lowered_module.to_executorch().buffer)


def load_config(path="./config/geolife/ind_transformer.yml"):
        """
        Loads config file:
        Args:
            path (str): path to the config file
        Returns:
            config (dict): dictionary of the configuration parameters, merge sub_dicts
        """

        print("LocationPrediction - utils - train.py - load_config - path - ",path)

        with open(path, "r") as f:
            cfg = yaml.safe_load(f)

        config = dict()

        print("LocationPrediction - utils - train.py - load_config - config - ",config)

        for _, value in cfg.items():
            for k, v in value.items():
                config[k] = v

        return config

def _dict_flatten(d):
    return list(d.values()), list(d.keys())

def _dict_flatten_with_keys(d):
    values, context = _dict_flatten(d)
    return [(pytree.MappingKey(k), v) for k, v in zip(context, values)], context

def _dict_unflatten(values, context):
    return dict(zip(context, values))

@adonnini
Copy link
Author

adonnini commented Jul 30, 2024

@anijain2305 I changed the export statement from

pre_autograd_aten_dialect = torch.export._trace._export(m, (config, total_loc_num), strict=False, pre_dispatch=True)

to

pre_autograd_aten_dialect = torch.export.export(m, args=(config, total_loc_num), strict=False)

(I did not include the kwargs parameter as it is not relevant in my model)

export failed producing the traceback log reported below

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/main.py", line 68, in <module>
    res_single = single_run(train_loader, val_loader, test_loader, config, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/main.py", line 23, in single_run
    model, perf = get_trainedNets(config, model, train_loader, val_loader, device, log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/utils/utils.py", line 47, in get_trainedNets
    best_model, performance = trainNet(config, model, train_loader, val_loader, device, log_dir=log_dir)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/utils/train.py", line 412, in trainNet
    pre_autograd_aten_dialect = torch.export.export(m, args=(config, total_loc_num), strict=False)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/__init__.py", line 174, in export
    return _export(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 945, in wrapper
    raise e
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 928, in wrapper
    ep = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/exported_program.py", line 89, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 1455, in _export
    aten_export_artifact = export_func(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 1317, in _non_strict_export
    aten_export_artifact = _export_to_aten_ir(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 583, in _export_to_aten_ir
    gm, graph_signature = transform(aot_export_module)(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 1268, in _aot_export_non_strict
    gm, sig = aot_export(wrapped_mod, args, kwargs=kwargs, **flags)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1131, in aot_export_module
    fx_g, metadata, in_spec, out_spec = _aot_export_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1350, in _aot_export_function
    fx_g, meta = create_aot_dispatcher_function(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 231, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 562, in create_aot_dispatcher_function
    fw_metadata = run_functionalized_fw_and_collect_metadata(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/collect_metadata_analysis.py", line 163, in inner
    flat_f_outs = f(*flat_f_args)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 178, in flat_fn
    tree_out = fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 748, in functional_call
    out = mod(*args[params_len:], **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/export/_trace.py", line 1255, in forward
    tree_out = self._export_root(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/models/MHSA.py", line 40, in forward
    emb = self.Embedding(src, context_dict)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/LocationPredictionContextQ/models/embed.py", line 149, in forward
    emb = self.emb_loc(src)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 164, in forward
    return F.embedding(
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/nn/functional.py", line 2267, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not dict

@adonnini
Copy link
Author

@anijain2305 Sorry to bother you again. I have been waiting for the resolution of this issue for a few months now. Please let me know what I should do next.
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dynamo-must-fix These bugs affect TorchDynamo reliability. export-triaged This tag is used to tag issues that have been looked by PT2 Export team and determined the next step oncall: export triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Development

No branches or pull requests