This page maintains a list of codebases and examples on model scaling using PyTorch/XLA, typically on TPUs and in scenarios where the model cannot fit into the memory of a single device.

• Vision Transformers (ViT) with 10B+ parameters using Fully Sharded Data Parallel (FSDP)