Feature integration in acoustic models using Low Rank Spectro-Temporal Decomposition in Convolutional Nets 🎤
Visit this site for interactive plots https://vaibhav016.github.io/FILRCN/ 😄
- (05/1/2021) Trained vanilla Contextnet over Librispeech Dataset https://arxiv.org/abs/2005.03191
- (05/6/2021) Implemented the Low rank decomposition on ContextNet on both spectrogram and raw audio input.
- (05/16/2021) Generated Loss Landscapes for the trained models, see demo_loss
- (05/20/2020) Trained a Low Rank Decomposition based Deep Net with wave input Low rank decomposition model
- (06/7/2020) Generated Integrated Gradients for trained models Keras Integrated Gradients documentation
- What's New?
- Table of Contents
- Publications
- Installation
- Setup training and testing
- Features Extraction
- Augmentations
- Training & Testing Tutorial
- Loss landscape visualisation and gradient attribution
- English Dataset
- References & Credits
- Contact
-
ContextNet (Reference: https://arxiv.org/abs/2005.03191) See examples/contextnet
-
**Raw Wwaveform Based CNN Through LOW-RANK Spectro-Temporal Decoupling ** (Reference: https://publications.idiap.ch/downloads/reports/2019/Abrol_Idiap-RR-11-2019.pdf) See tensorflow_asr/models/encoders
git clone https://github.com/vaibhav016/FILRCN.git
cd FILRCN
python setup.py build
python setup.py install
docker-compose up -d
-
For mixed precision training, use flag
--mxp
when running python scripts from examples -
For enabling XLA, run
TF_XLA_FLAGS=--tf_xla_auto_jit=2 python3 $path_to_py_script
) -
For hiding warnings, run
export TF_CPP_MIN_LOG_LEVEL=2
before running any examples
See augmentations
- Define config YAML file, see the
config.yml
files in the example folder for reference (you can copy and modify values such as parameters, paths, etc.. to match your local machine configuration) - Download your corpus (a.k.a datasets) and run
download_links.sh
scripts folder to download files For more detail, see datasets. Note: Make sure your data contain only characters in your language, for example, english hasa
toz
and'
. Do not usecache
if your dataset size is not fit in the RAM. - [Optional] Generate TFRecords to use
tf.data.TFRecordDataset
for better performance by using the script create_tfrecords.py - Create vocabulary file (characters or subwords/wordpieces) by defining
language.characters
, using the scripts generate_vocab_subwords.py or generate_vocab_sentencepiece.py. There're predefined ones in vocabularies - [Optional] Generate metadata file for your dataset by using script generate_metadata.py. This metadata file contains maximum lengths calculated with your
config.yml
and total number of elements in each dataset, for static shape training and precalculated steps per epoch. - run
create_transcripts_from_data.sh
from scrpts folder to generate .tsv files(the format in which the input is given is .tsv) - For training, see
train.py
files in the example folder to see the options - For testing, see
test.py
files in the example folder to see the options.
For visualisations, we have two kinds of scripts.
cd examples/contextnet/contextnet_visualisation
- for loss landscapes, cd into context_visualisation/loss_landscape_visualisation.
- run generate_lists.py(This generates the loss and accuracy lists)
- now run plot_loss.py (From those lists, images are drawn both 2d and 3d)
- now run video_create.py(It sews all the images into a single video)
- For gradient visualisation,
- run integrated_grad_vis.py, which will generate the integrated gradients for all the trained models
- then run plot_gradients.py
- Finally run video_create.py
For loss landscape, go to drive
For gradient attribution, go to drive
Name | Source | Hours |
---|---|---|
LibriSpeech | LibriSpeech | 970h |
Vaibhav Singh ([email protected])
Dr Vinayak Abrol ([email protected])