Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add functionality to automatically upload logs to Vertex Tensorboard #570

Merged
merged 1 commit into from
Apr 10, 2024
Merged

Add functionality to automatically upload logs to Vertex Tensorboard #570

merged 1 commit into from
Apr 10, 2024

Conversation

SurbhiJainUSC
Copy link
Collaborator

@SurbhiJainUSC SurbhiJainUSC commented Mar 29, 2024 •
edited
Loading

  • Add functionality to upload data in config.tensorboard_dir to Tensorboard in Vertex AI
  • XPK users won't have to create a Tensorboard in Vertex AI. XPK will automatically handle that.
  • For non-XPK cases, users can choose to manually create Tensorboard instance in Vertex AI cloud console or set the configuration to create instance in MaxText

Note: Uploader to upload logs to Vertex Tensorboard is only supported for Tensorflow < 2.15.0. Vertex AI team is working on fixing the issue to support the latest TF versions.

rwitten
rwitten requested changes Mar 30, 2024
View reviewed changes
Copy link
Collaborator

@rwitten rwitten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of questions! But looks amazing!

MaxText/configs/base.yml Outdated Show resolved Hide resolved
MaxText/configs/base.yml Show resolved Hide resolved
MaxText/configs/base.yml Outdated Show resolved Hide resolved
MaxText/configs/base.yml Outdated Show resolved Hide resolved
MaxText/configs/base.yml Outdated Show resolved Hide resolved
MaxText/configs/base.yml Outdated Show resolved Hide resolved
MaxText/train.py Outdated Show resolved Hide resolved
MaxText/train.py Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@rwitten rwitten removed their assignment Mar 30, 2024
@SurbhiJainUSC
Copy link
Collaborator Author

Lots of questions! But looks amazing!

Thank you Rafi for your suggestions!
I have simplified the process of creating Vertex AI Tensorboard for testing MaxText on GCE. I have also updated the README to explain different scenarios to upload logs to Vertex Tensorboard. Let me know if this looks good.

rwitten
rwitten requested changes Apr 3, 2024
View reviewed changes
Copy link
Collaborator

@rwitten rwitten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit teardown is still a bit weird but looks good otherwise!

MaxText/configs/base.yml Show resolved Hide resolved
MaxText/train.py Outdated Show resolved Hide resolved
@rwitten rwitten removed their assignment Apr 3, 2024
@SurbhiJainUSC SurbhiJainUSC assigned rwitten and unassigned rwitten Apr 8, 2024
rwitten
rwitten approved these changes Apr 9, 2024
View reviewed changes
Copy link
Collaborator

@rwitten rwitten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (but there are some nits I want you to cleanup)

MaxText/configs/base.yml Show resolved Hide resolved
MaxText/train.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@rwitten rwitten removed their assignment Apr 9, 2024
@copybara-service copybara-service bot merged commit e366da6 into main Apr 10, 2024
7 of 8 checks passed
@copybara-service copybara-service bot deleted the vertex_tb branch April 10, 2024 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rwitten rwitten rwitten approved these changes

@gobbleturk gobbleturk Awaiting requested review from gobbleturk gobbleturk is a code owner

Assignees
No one assigned
Labels
pull ready
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@SurbhiJainUSC @rwitten