Large time-series models, pre-training datasets, adaptation techniques, and benchmarks.

For deep time series models and task-specific benchmarks, we strongly recommend Time-Series-Library and this comprehensive Survey.

🚩 News (2024.10) We include four large time-series models, release pre-training logic, and provide scripts.

LTM (Large Time-Series Model) is a series of scalable deep models built on foundation backbones (e.g. Transformers) and large-scale pre-training, which will be applied to a variety of time series data and diverse downstream tasks. For more information, here we provide [Slides]!

• Moirai - Unified Training of Universal Time Series Forecasting Transformers. [ICML 2024], [Code]
• Moment - MOMENT: A Family of Open Time-series Foundation Model. [ICML 2024], [Code]
• Timer - Timer: Generative Pre-trained Transformers Are Large Time Series Models. [ICML 2024], [Code]
• Timer-XL - Timer-XL: Long-Context Transformer for Unified Time Series Forecasting. [arxiv 2024], [Code]

We will update the following models to the checklist after a comprehensive evaluation. Welcome to give your suggestion about any interesting works 🤗

• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models. [NeurIPS 2024], [Code]
• Chronos: Learning the Language of Time Series. [arxiv 2024], [Code]
• Time-MoE: Billion-Scale Time Series Foundation Models With Mixture Of Experts. [arxiv 2024], [Code]
• A Decoder-Only Foundation Model for Time-Series Forecasting. [arxiv], [Code]

1. Install Python 3.10. For convenience, execute the following command.

pip install -r requirements.txt

1. Place downloaded data in the folder ./dataset. Here is a dataset summary.

• For pre-training:

  • UTSD contains 1 billiion time points for large-scale pre-training (in numpy format): [Download].
  • ERA5-Familiy (40-year span, thousands of variables) for domain-specific model: [Download].

• For superwised training or modeling adaptation

  • Well-acknowlegded datasets from TSLib : [Download].

2. We provide pre-training and adaptation scripts under the folder ./scripts/. You can conduct experiments using the following examples:

# Supervised training
# (a) one-for-one forecasting
bash ./scripts/supervised/forecast/moirai_ecl.sh
# (b) one-for-all (rolling) forecasting
bash ./scripts/supervised/rolling_forecast/timer_xl_ecl.sh

# Large-scale pre-training
# (a) pre-training on UTSD
bash ./scripts/pretrain/timer_xl_utsd.sh
# (b) pre-training on ERA5
bash ./scripts/pretrain/timer_xl_era5.sh

# Model adaptation
# (a) full-shot fine-tune
bash ./scripts/adaptation/full_shot/timer_xl_etth1.sh
# (b) few-shot fine-tune
bash ./scripts/adaptation/few_shot/timer_xl_etth1.sh
# (c) zero-shot generalization
bash ./scripts/adaptation/zero_shot/timer_xl_etth1.sh

3. Develop your large time-series model.

• Add the model file to the folder ./models. You can follow the ./models/timer_xl.py.
• Include the newly added model in the Exp_Basic.model_dict of ./exp/exp_basic.py.
• Create the corresponding scripts under the folder ./scripts.

For the first four forecasting tasks, in addition to supervised training (current leaderboard), a LTM can also be evaluated on full-shot and few-shot tasks, depending on downstream data availability and whether or not a pre-trained model is used. For other two generalization tasks, please see the paper for details.

We present a theoretical proof of the computational complexity of Time-Series Transformers. See the paper for details.

If you find this repo helpful, please cite our paper.

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

We appreciate the following GitHub repos a lot for their valuable code and efforts:

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)
• Large-Time-Series-Model (https://github.com/thuml/Large-Time-Series-Model)
• AutoTimes (https://github.com/thuml/AutoTimes)

If you have any questions or want to use the code, feel free to contact:

• Yong Liu ([email protected])
• Guo Qin ([email protected])