Efficient Autoregressive Audio Modeling via Next-Scale Prediction
- (2024.08.24) Demo Released, tokenizer for other datasets will be available in two weeks.
- (2024.08.22) Add SAT and AAR code, demo will be released soon.
- (2024.08.20) Repo created. Code and checkpoints will be released this week.
- Install all packages via
pip3 install -r requirements.txt.
We download our Audioset from the website https://research.google.com/audioset/ and collect it as
AudioSet
├── audioset_unbalanced_train_mp3
├── unbalanced_train_segments.csv
└── audioset_eval_raw_mp3
We are currently training large-scale SAT for music, audio, and speech. We expect the checkpoint will be ready and released in Sept.
python3 train_SAT_mpi.py --config config/train/SAT.yaml --train_dir /path/to/audioset_unbalanced_train_mp3 --train_csv /path/to/csv --batch_size $bs --gpus $gpus --output_dir /path/to/save/ckpt --use_prefetcher True --resume latest
python3 inference_SAT.py --config config/inference/SAT.yaml --resume /path/to/ckpt.pth --test_dir /path/to/audioset_eval_raw_mp3 --batch_size $bs
We provide Audioset pre-trained SAT checkpoint as follows:
| model | # Scale | # Tokens | latent_dim | FAD | HF weights 🤗 |
|---|---|---|---|---|---|
| SAT | 16 | 455 | 64 | 1.09 | SAT.pth |
| SAT | 16 | 455 | 128 | 1.40 | SAT.pth |
python3 train_AAR_mpi.py --config config/train/AAR.yaml --train_dir /path/to/audioset_unbalanced_train_mp3 --train_csv /path/to/csv --batch_size $bs --gpus $gpus --output_dir /path/to/save/ckpt --use_prefetcher True --resume latest --vqvae_pretrained_path /path/to/vae/ckpt --latent_dim $latent --dimension $dim
python3 inference_AAR.py --config config/inference/AAR.yaml --aar_pretrained_path /path/to/aar.pth --vqvae_pretrained_path /path/to/vqvae.pth --test_dir /path/to/audioset_eval_raw_mp3 --batch_size $bs --output_dir /path/to/save
We provide Audioset pre-trained AAR checkpoint as follows:
| model | # Scale | # Tokens | latent_dim | FAD | HF weights 🤗 |
|---|---|---|---|---|---|
| SAT | 16 | 455 | 128 | 1.40 | SAT.pth |
| AAR | 16 | 455 | 128 | 6.01 | AAR.pth |
@misc{qiu2024efficient,
title={Efficient Autoregressive Audio Modeling via Next-Scale Prediction},
author={Kai Qiu and Xiang Li and Hao Chen and Jie Sun and Jinglu Wang and Zhe Lin and Marios Savvides and Bhiksha Raj},
year={2024},
eprint={2408.09027},
archivePrefix={arXiv},
primaryClass={cs.SD}
}