[Unified TorchTrainer] Add HF Transformers TorchTrainer Utilities (ra…

…y-project#38083) Signed-off-by: woshiyyya <[email protected]> Signed-off-by: Yunxuan Xiao <[email protected]> Co-authored-by: matthewdeng <[email protected]>
krfricke · Aug 12, 2023 · 3d69d15 · 3d69d15
1 parent 7185307
commit 3d69d15
Show file tree

Hide file tree

Showing 12 changed files with 1,105 additions and 19 deletions.
diff --git a/doc/source/_toc.yml b/doc/source/_toc.yml
@@ -65,10 +65,12 @@ parts:
  title: PyTorch
  - file: train/getting-started-pytorch-lightning
  title: PyTorch Lightning
+ - file: train/getting-started-transformers
+ title: Hugging Face Transformers
  - file: train/more-frameworks
  sections:
  - file: train/huggingface-transformers-accelerate
- title: HuggingFace Transformers & Accelerate
+ title: Hugging Face Transformers & Accelerate
  - file: train/distributed-tensorflow-keras
  title: TensorFlow & Keras
  - file: train/distributed-xgboost-lightgbm
@@ -86,6 +88,8 @@ parts:
  sections:
  - file: train/examples/pytorch/torch_fashion_mnist_example
  title: "PyTorch Fashion MNIST Example"
+ - file: train/examples/transformers/transformers_torch_trainer_basic
+ title: "Hugging Face Transformers Basic Example"
  - file: train/examples/lightning/lightning_mnist_example
  title: "PyTorch Lightning Basic Example"
  - file: train/examples/lightning/lightning_cola_advanced

diff --git a/doc/source/train/api/api.rst b/doc/source/train/api/api.rst
@@ -19,7 +19,7 @@ Ray Train Integrations
 PyTorch Ecosystem
 ~~~~~~~~~~~~~~~~~
 
-Scale out your PyTorch, Lightning, HuggingFace code with Ray TorchTrainer.
+Scale out your PyTorch, Lightning, Hugging Face code with Ray TorchTrainer.
 
 .. autosummary::
  :toctree: doc/
@@ -72,17 +72,28 @@ PyTorch Lightning
  ~train.lightning.LightningCheckpoint
  ~train.lightning.LightningPredictor
 
-HuggingFace Transformers
-************************
+Hugging Face Transformers
+*************************
+
+.. autosummary::
+ :toctree: doc/
+
+ ~train.huggingface.transformers.prepare_trainer
+ ~train.huggingface.transformers.RayTrainReportCallback
+
+.. note::
+
+ We will deprecate `TransformersTrainer`, `TransformersCheckpoint` in Ray 2.8. Please 
+ refer to the :ref:`migration guide <transformers-trainer-migration-guide>` for more info.
 
 .. autosummary::
  :toctree: doc/
 
  ~train.huggingface.TransformersTrainer
  ~train.huggingface.TransformersCheckpoint
 
-HuggingFace Accelerate
-**********************
+Hugging Face Accelerate
+***********************
 
 .. autosummary::
  :toctree: doc/

diff --git a/doc/source/train/examples/transformers/transformers_torch_trainer_basic.rst b/doc/source/train/examples/transformers/transformers_torch_trainer_basic.rst
@@ -0,0 +1,8 @@
+:orphan:
+
+.. _transformers_torch_trainer_basic_example :
+
+Ray Train Basic Example for HuggingFace Transformers
+====================================================
+
+.. literalinclude:: /../../python/ray/train/examples/transformers/transformers_torch_trainer_basic.py