Merge branch 'master' into air-docs-refactor

doc/source/_toc.yml

@@ -41,6 +41,54 @@ parts:
  - file: ray-core/examples/map_reduce
  - file: ray-core/api/index
 
+<<<<<<< HEAD
+=======
+ - file: ray-air/getting-started
+ title: "Ray AI Runtime (AIR)"
+ sections:
+ - file: ray-air/key-concepts
+ - file: ray-air/user-guides
+ sections:
+ - file: ray-air/preprocessors
+ - file: ray-air/trainers
+ - file: ray-air/check-ingest
+ - file: ray-air/tuner
+ - file: ray-air/predictors
+ - file: ray-air/computer-vision
+ - file: ray-air/examples/serving_guide
+ - file: ray-air/deployment
+ - file: ray-air/examples/index
+ sections:
+ - file: ray-air/examples/torch_image_example
+ - file: ray-air/examples/torch_detection
+ - file: ray-air/examples/convert_existing_pytorch_code_to_ray_air
+ - file: ray-air/examples/convert_existing_tf_code_to_ray_air
+ - file: ray-air/examples/tfx_tabular_train_to_serve
+ - file: ray-air/examples/huggingface_text_classification
+ - file: ray-air/examples/sklearn_example
+ - file: ray-air/examples/xgboost_example
+ - file: ray-air/examples/analyze_tuning_results
+ - file: ray-air/examples/lightgbm_example
+ - file: ray-air/examples/torch_incremental_learning
+ - file: ray-air/examples/rl_serving_example
+ - file: ray-air/examples/rl_online_example
+ - file: ray-air/examples/rl_offline_example
+ - file: ray-air/examples/upload_to_comet_ml
+ - file: ray-air/examples/upload_to_wandb
+ - file: ray-air/examples/feast_example
+ - file: ray-air/examples/automl_with_ray_air
+ - file: ray-air/examples/batch_tuning
+ - file: ray-air/examples/batch_forecasting
+ - file: ray-air/examples/stablediffusion_batch_prediction
+ - file: ray-air/examples/gptj_deepspeed_fine_tuning
+ - file: ray-air/examples/gptj_batch_prediction
+ - file: ray-air/examples/gptj_serving
+ - file: ray-air/examples/dreambooth_finetuning
+ - file: ray-air/examples/dolly_lightning_fsdp_finetuning
+ - file: ray-air/api/api
+ - file: ray-air/benchmarks
+
+>>>>>>> c89220e4b1327aff493423611e031871197243c9
  - file: data/data
  title: Ray Data
  sections:

doc/source/data/user-guide.rst

@@ -20,8 +20,8 @@ show you how achieve several tasks.
  working-with-images
  working-with-text
  working-with-tensors
- preprocessors
+ working-with-pytorch
+ batch_inference
  data-internals
  performance-tips
- batch_inference
-
+ preprocessors

doc/source/ray-air/getting-started.rst

@@ -44,3 +44,162 @@ AIR is for both data scientists and ML engineers alike.
  https://docs.google.com/drawings/d/1Qw_h457v921jWQkx63tmKAsOsJ-qemhwhCZvhkxWrWo/edit
 
 For data scientists, AIR can be used to scale individual workloads, and also end-to-end ML applications. For ML Engineers, AIR provides scalable platform abstractions that can be used to easily onboard and integrate tooling from the broader ML ecosystem.
+<<<<<<< HEAD
+=======
+
+Quick Start
+-----------
+
+Below, we walk through how AIR's unified ML API enables scaling of end-to-end ML workflows, focusing on
+a few of the popular frameworks AIR integrates with (XGBoost, Pytorch, and Tensorflow).
+
+..
+ https://docs.google.com/drawings/d/1z0r_Yc7-0NAPVsP2jWUkLV2jHVHdcJHdt9uN1GDANSY/edit
+
+.. .. figure:: images/why-air.svg
+
+.. AIR provides a unified API for the ML ecosystem.
+.. This diagram shows how AIR enables an ecosystem of libraries to be run at scale in just a few lines of code.
+
+Get started by installing Ray AIR:
+
+.. code:: bash
+
+ pip install -U "ray[air]"
+
+ # The below Ray AIR tutorial was written with the following libraries.
+ # Consider running the following to ensure that the code below runs properly:
+ pip install -U pandas>=1.3.5
+ pip install -U torch>=1.12
+ pip install -U numpy>=1.19.5
+ pip install -U tensorflow>=2.6.2
+ pip install -U pyarrow>=6.0.1
+
+Preprocessing
+~~~~~~~~~~~~~
+
+First, let's start by loading a dataset from storage:
+
+.. literalinclude:: examples/xgboost_starter.py
+ :language: python
+ :start-after: __air_generic_preprocess_start__
+ :end-before: __air_generic_preprocess_end__
+
+Then, we define a ``Preprocessor`` pipeline for our task:
+
+.. tabs::
+
+ .. group-tab:: XGBoost
+
+ .. literalinclude:: examples/xgboost_starter.py
+ :language: python
+ :start-after: __air_xgb_preprocess_start__
+ :end-before: __air_xgb_preprocess_end__
+
+ .. group-tab:: Pytorch
+
+ .. literalinclude:: examples/pytorch_tabular_starter.py
+ :language: python
+ :start-after: __air_pytorch_preprocess_start__
+ :end-before: __air_pytorch_preprocess_end__
+
+ .. group-tab:: Tensorflow
+
+ .. literalinclude:: examples/tf_tabular_starter.py
+ :language: python
+ :start-after: __air_tf_preprocess_start__
+ :end-before: __air_tf_preprocess_end__
+
+.. _air-getting-started-training:
+
+Training
+~~~~~~~~
+
+Train a model with a ``Trainer`` with common ML frameworks:
+
+.. tabs::
+
+ .. group-tab:: XGBoost
+
+ .. literalinclude:: examples/xgboost_starter.py
+ :language: python
+ :start-after: __air_xgb_train_start__
+ :end-before: __air_xgb_train_end__
+
+ .. group-tab:: Pytorch
+
+ .. literalinclude:: examples/pytorch_tabular_starter.py
+ :language: python
+ :start-after: __air_pytorch_train_start__
+ :end-before: __air_pytorch_train_end__
+
+ .. group-tab:: Tensorflow
+
+ .. literalinclude:: examples/tf_tabular_starter.py
+ :language: python
+ :start-after: __air_tf_train_start__
+ :end-before: __air_tf_train_end__
+
+.. _air-getting-started-tuning:
+
+Hyperparameter Tuning
+~~~~~~~~~~~~~~~~~~~~~
+
+You can specify a hyperparameter space to search over for each trainer:
+
+.. tabs::
+
+ .. group-tab:: XGBoost
+
+ .. literalinclude:: examples/xgboost_starter.py
+ :language: python
+ :start-after: __air_xgb_tuner_start__
+ :end-before: __air_xgb_tuner_end__
+
+ .. group-tab:: Pytorch
+
+ .. literalinclude:: examples/pytorch_tabular_starter.py
+ :language: python
+ :start-after: __air_pytorch_tuner_start__
+ :end-before: __air_pytorch_tuner_end__
+
+ .. group-tab:: Tensorflow
+
+ .. literalinclude:: examples/tf_tabular_starter.py
+ :language: python
+ :start-after: __air_tf_tuner_start__
+ :end-before: __air_tf_tuner_end__
+
+Then use the ``Tuner`` to run the search:
+
+.. literalinclude:: examples/pytorch_tabular_starter.py
+ :language: python
+ :start-after: __air_tune_generic_start__
+ :end-before: __air_tune_generic_end__
+
+Batch Inference
+~~~~~~~~~~~~~~~
+
+After running the steps in :ref:`Training <air-getting-started-training>` or
+:ref:`Tuning <air-getting-started-tuning>`, use the trained model for scalable batch
+prediction with :meth:`Dataset.map_batches() <ray.data.Dataset.map_batches>`.
+
+To learn more, see :ref:`End-to-end: Offline Batch Inference <batch_inference_home>`.
+
+
+Project Status
+--------------
+
+AIR is currently in **beta**. If you have questions for the team or are interested in getting involved in the development process, fill out `this short form <https://forms.gle/wCCdbaQDtgErYycT6>`__.
+
+For an overview of the AIR libraries, ecosystem integrations, and their readiness, check out the latest :ref:`AIR ecosystem map <air-ecosystem-map>`.
+
+Next Steps
+----------
+
+- :ref:`air-key-concepts`
+- :ref:`air-examples-ref`
+- :ref:`API reference <air-api-ref>`
+- :ref:`Technical whitepaper <whitepaper>`
+- To check how your application is doing, you can use the :ref:`Ray dashboard<observability-getting-started>`.
+>>>>>>> c89220e4b1327aff493423611e031871197243c9