Project for the Language Processing exam.

The repository is divided into 2 parts:

• Grammar Design (grammar_design folder)
• NLP project (NLP folder)

number as the function arguments.

This project aims to develop a system for transcribing audio into text using the powerful transformers library. The primary models utilized in this project are wav2vec2 xlsr and Whisper, both of which are state-of-the-art models for speech recognition tasks.

The wav2vec2 xlsr model is particularly well-suited for multilingual speech recognition tasks, as it has been trained on a diverse range of languages. On the other hand, the Whisper model is specifically designed for low-resource languages, making it an excellent choice for improving transcription accuracy in challenging scenarios.

The code is runnable on Colab on this link:

• CTC:

  • Model Wav2Vec2 base:

    • Wav2vec2-base preaddestrato solo in poche lingue (non italiano)

    • Wav2vec2-base-it-voxpopuli-v2 (Modello base finetuning su dataset italiano)

  • Model Wav2Vec2 XLS-R (53 language):

    • Wav2vec2-large-xlsr-53 originale non preaddestrato

    • modello finetuning in italiano Large

    • Modello XLS-R 300M di parametri addestrato su 128 lingue (anche italiano):

      • modello base 300M preaddestrato
      • modello finetuning in italiano

• SEQ2SEQ:

  • Model Whisper:

    • ufficial Github Repo

    • Modello base addestrato in più lingue (anche italiano)

    • Modello Small (244M di parametri)

    • Modello Tiny (39M di parametri)

    • EdoAbati/whisper-large-v2-it

    • openai/whisper-large-v3

    • fsicoli/whisper-large-v3-it-cv16

To train and evaluate these models, we will be leveraging the common voice 11, common voice 16, and fleurs datasets. These datasets consist of a vast collection of multilingual audio recordings and their corresponding transcriptions, enabling us to build a robust and accurate transcription system.

Dataets used:

• PolyAI/minds14
• Mozilla/common_voice sono state prese 2 versioni:
  • Versione 11
  • Versione 16
• Google/fleurs

For training you need to define several options and you can do this either using environment variables or command line options. There are 2 main files:

• NLP/script/main_ctc.py -> script to train/test CTC based networks (Wav2Vec2)
• NLP/script/main_seq2seq.py -> script to train/test SEQ2SEQ based networks (Whisper)

The command always needs the --output_dir option which indicates the path where to save the information and the model.

For the training/evaluate of seq2seq model it is mandatory to set this 2 command line options if you want to get the metrics:

• --predict_with_generate
• --generation_max_length="250"

Example usage for training whisper-tiny model on google/fleurs dataset and evaluate on it at each EVAL_STEPS:

  python NLP/script/main_seq2seq.py --model_name="openai/whisper-tiny" --output_dir="training_whisper-tiny" --predict_with_generate --generation_max_length="250" --do_eval --do_train

Then it's mandatory to set up the propper Environment Variables for downloading and setting the dataset and training procedures.

We have implemented a bot on Telegram to make inference and transcribe audio files into text. The script can be started in the Telegram folder and through the code in bot.py, requirementx.txt inside the telegram folder contains only useful libraries to make the bot work. The bot uses the PyTelegramBot library to instantiate the bot using the appropriate token, then subsequently uses ASR models (seq2seq) for transcription.

The bot checks with each message whether the file is an audio file, if this is the case then it checks the length of the audio file:

If the audio lasts more than 30 seconds then the Faster-Whisper model is used to quickly generate the transcription of even very long audio and subsequently a summary is applied using the T5 model (a finetuning of the model is used by default: it5-base-summarization ) and subsequently the the summary is sent as a message to the user while the full transcript is written on a Telegraph page and sent via link to the user.

If the audio lasts less than 30 seconds, speculative inference and 2 models are exploited (they can be chosen via environment variables but one must be larger than the other) through which it is possible to save execution time by executing a first transcribe onto a smaller model and then give the larger model the more difficult parts.

must be set for the bot to work correctly:

• TELEGRAM_TOKEN = *your_token*
• TELEGRAM_BIG_MODEL = "*name_or_path_main_model*" (e.g. openai/whisper-large)
• TELEGRAM_SMALL_MODEL = "*name_or_path_assistent_model*" (e.g. openai/whisper-tiny)
• TELEGRAM_FASTER_MODEL = "medium" (possible options: tiny, base, small, large, large-v2, large-v3)
• TELEGRAM_SUMMARY_MODEL = "*name_or_path_assistent_model*" (e.g. efederici/it5-base-summarization)

• @sofiaalbini
• @duccioo

• Github Repo Ufficiale ASR

• Automatic Speech Recognition Hugging Face

• recording audio in google colab

• Esempio su come usare file audio per Inference

• Finetuning Wav2Vec2 XLSR

• Finetuning Whisper Model (seq2seq)

• COLAB: Finetuning Whisper Model (seq2seq)

• 1 script for XLSR and Whisper

• FineTune on Costum Datasets:

  • Make your Own Audio Dataset
  • HuggingSound

• State of Art and Optimizations!:

  • Distil Whisper
  • Streaming Datasets from Internet
  • Speculative Decoding (inference 2x faster)
  • Faster Whisper and Large audio Files