A collection of Jupyter Notebooks related to audio processing.
The notebooks act like interactive utility scripts for converting between different representations, usually stored in data/project/
where project
is the dataset you're working with. Generally, if you change data_root
near the top of the notebook and run the reset of the notebook, it will do something useful.
librosa currently needs some extra help on OS X, make sure to follow the instructions here first.
$ brew install ffmpeg # for loading and saving audio
$ git clone https://github.com/kylemcdonald/AudioNotebooks.git
$ cd AudioNotebooks.git
$ pip install -r requirements.txt
$ jupyter notebook
Here are some words used in the names of the notebooks, and what they mean:
- Samples refers to one-shot sounds, usually less than 1-2 seconds long. These can be loaded from a directory, like
data/project/samples/
or from a precomputed numpy matrix likedata/project/samples.npy
. When they are stored in a.npy
file, all the samples are necessarily concatenated or expanded to be the same length. - Multisamples refers to audio that needs to be segmented into samples.
- Fingerprints refer to small images, usually 32x32 pixels, representing a small chunk of time like 250ms or 500ms. These are either calculated with CQT, STFT, or another frequency domain analysis technique. They are useful for running t-SNE or training neural nets.
- Spritesheets are single files with multiple sounds, either visually as fingerprints or sonically as a sequence of sounds, organized carefully so they can be chopped up again later.
Some formats in use:
.npy
are numpy matrices. Numpy can load and save these very quickly, even for large datasets..tsv
are tab separated files referring to one sample per line, usally with normalized numbers in each column. These are good for loading into openFrameworks apps, or into the browser..txt
are like.tsv
but only have one item per line, usually a single string. Also good for loading into openFrameworks apps, or into the browser..pkl
are Pickle files, which is the native Python serialization format, and is used for saving and loading datastructures that have lists of objects with lots of different kinds of values (not just numbers or strings)..h5
is the way the Keras saves the weights for a neural net..json
is good for taking what would usually go into a Pickle file, and saving it in a format that can be loaded onto the web. It's also one of the formats used by Keras, part of a saved model.
- Collect Samples
- Samples to Audio Spritesheet
- Collect Samples
- Samples to Fingerprints
- Fingerprints to t-SNE (with
mode = "fingerprints"
)
The standard workflow is to create a t-SNE embedding from fingerprints, but it's also possible to create an embedding after learning a classifier:
- Collect Samples
- Samples to Fingerprints
- Collect Metadata
- Metadata to Labels
- Fingerprints and Labels to Classifier
- Fingerprints to t-SNE (with
mode = "combined"
)
Right this only really works with extracting phonemes from transcribed speech, using Gentle.
- Gentle to Samples (with
save_wav = True
) - Samples to Fingerprints
- Fingerprints to t-SNE
It's also possible to use Sphinx for speech that does not have transcriptions, but it can be very significantly slower:
- Sphinx to Samples
- Collect Samples
- Samples to Fingerprints
- Fingerprints to t-SNE
By virtue of creating a rectangular grid, you may lose some points. This technique will only work on 10-20k points maximum
- Collect Samples
- Samples to Fingerprints
- Fingerprints to t-SNE
- Run the
example-data
app from ofxAssignment or use CloudToGrid to convert a 2d t-SNE embedding to a grid embedding. - Fingerprints to Spritesheet
If you only want a spritesheet without any sorting, skip step 4 and only run step 5 partially.
- Collect Samples
- Samples to Fingerprints
- Collect Metadata
- Metadata to Labels
- Fingerprints and Labels to Classifier