Team Name: TensorSlow

Members: Aditya Singh (@adityajaas) and Shashi Kant Gupta (@shashikg)

Report: It is available here. It contains 4 pages of main text + 1 references page + 2 pages of supplementary materials.

Speaker diarization has received significant interest within the speech community due to its promise to improve automatic speech transcription considerably. Commonly used approach to this problem include using embedding vectors such as d-vectors, i-vectors, or x-vectors with Spectral Clustering. We propose using Unsupervised Deep Embedding Clustering to cluster data in a more semantically meaningful latent representation with pre-trained Auto Encoders for improved imbalanced data separation. Stacked layers of Auto Encoders have been trained in a residual fashion in place of De-noising Auto Encoders for enhanced learning. We use VoxConverse and AMI Corpus split datasets to test our model. Our model shows considerable improvement over the Spectral Clustering approach. Clustering is perfomed on x-vectors extracted using Desplanques et al.'s ECAPA-TDNN framework. We use Silero-VAD for voice audio detection.

Model is tested on VoxConverse dataset (total 216 audio files). We randomly split the dataset into two parts: ‘test’ and ‘train’ with test data having 50 audio files. We also tested the model on AMI test dataset (total 16 audio files).

Original Video Link: here
Diarization Output Link: here

• Baseline<DATASET_NAME>.ipynb: To evaluate the DER score for the baseline models described in the report.
• Compare_Spectral_vs_DEC_<DATASET_PARAM>.ipynb: To evaluate the DER score for the DEC models described in the report and compare it against the Spectral clustering method.
• utilities/TrainAutoEncoder.ipynb: Output notebook file for training the AutoEncoder of the DEC model.
• utilities/ExtractVAD.ipynb: Used to extract and save all the VAD mapping for the audio files.
• utilities/ExtractXvectors.ipynb: Used to precompute X-vectors for the audio files and save it into a zip file to use it in the DiarizationDataset.

Documentation and details about functions isnide the core module.

• Defined in: utils.py
  • class DiarizationDataset
    • func __getitem__
    • func read_rttm
  • func make_rttm
  • func get_metrics
  • func plot_annot
• Defined in: baselineMethods.py
  • func diarizationOracleNumSpkrs
  • func diarizationEigenGapNumSpkrs
• Defined in: optimumSpeaker.py
  • class eigengap
    • func _get_refinement_operator
    • func compute_affinity_matrix
    • func compute_sorted_eigenvectors
    • func compute_number_of_clusters
    • func find
  • class AffinityRefinementOperation
    • func check_input
    • func refine
  • class CropDiagonal
    • func refine
  • class GaussianBlur
    • func refine
  • class RowWiseThreshold
    • func refine
  • class Symmetrize
    • func refine
  • class Diffuse
    • func refine
  • class RowWiseNormalize
    • func refine
• Defined in: DEC.py
  • class ResidualAutoEncoder
  • func load_encoder
  • class ClusteringModule
    • func init_centroid
  • class DEC
    • func fit
    • func predict
    • func clusterAccuracy
  • func diarizationDEC
• Defined in: colab_demo_utils.py
  • func downloadYouTube
  • func loadVideoFile
  • func read_rttm
  • func combine_audio
  • func createAnnotatedVideo