-
Notifications
You must be signed in to change notification settings - Fork 270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Readme Example is broken #2193
Comments
Sorry I could not reproduce this error:
This will also work:
Can you please try with --
I can look into how to support directly instantiate the model with importing |
Yeah, I was able to work around this by making sure I was running the script from with the torchbench top level structure, I do think this is very counter-intutitive. Script I got working, all though many models threw failures with the --precision=bf16 args Feel free to close this issue if you think its not reproducible |
I will look into how to instantiate models by directly importing from Can we have a list of models that don't support bf16? We fixed |
Using this script: from transformer_nuggets.utils.shape_trace import ShapeLog
import torch
from pathlib import Path
from tqdm import tqdm
import logging
import json
logging.basicConfig(level=logging.INFO)
def main():
import torchbenchmark.models as models
models = []
success_count = 0
failure_count = 0
model_failures = {}
for file in Path("torchbenchmark/models/").iterdir():
if file.is_dir():
models.append(file.name)
for model_name in tqdm(models, desc="Logging models", aunit="model"):
try:
module = __import__(f"torchbenchmark.models.{model_name}", fromlist=[model_name])
model, example_inputs = module.Model(test="train", device="cuda", extra_args=["--precision=bf16",]).get_module()
model(*example_inputs)
success_count += 1
except Exception as e:
tqdm.write(f"Failed to log {module}: {e}")
failure_count += 1
model_failures[model_name] = str(e)
tqdm.write(f"Successfully logged {success_count} models")
tqdm.write(f"Failed to log {failure_count} models")
with open("model_failures_bf16.txt", "w") as f:
json.dump(model_failures, f)
if __name__ == "__main__":
main() returns the following model failures; |
Thanks for sharing the result, I created a new issue #2203 tracking the bf16 precision support of the models. The package import problem has been fixed, so I am closing this issue now. |
Summary
Steps to reproduce
Having Torchbench not be package though makes the path shennagins kind of hard to reason about so not sure if the why the model is not being picked up.
The text was updated successfully, but these errors were encountered: