To verify the computational efficiency of Panako there is a benchmark script available. The script measures runtimes to store audio and to query the index. The main aim is to see how performance is affected when the index gets larger.
Note that correctness is not checked with this script. Please have a look at the evaluation script for correctness.
The script needs access to a folder with music. For example the FMA-Medium dataset can be used. The FMA medium set contains 25k, tracks of 30s spread over 16 unbalanced genres (22 GiB). Of course it might be of interest to use the music or sound of interest for your use-case.
The evaluation script does the following: starting from an empty database first 2 audio files are stored. Next, the database size is doubled a couple of times until a chosen power of two (32k, 64k,…) files are stored. The time it takes to double the database size (indexing audio) is measured and, after a doubling in size, the database is queried with a number of fragments. The time it takes to query the database is also reported.
The following results are measured on an Apple M1 Pro with 8 cores. The speeds are reported in number seconds of audio that can be processed per wall clock second (s/s). For example, if a store speed of 750 is reported then 750 seconds of audio are indexed in a single second.
The absolute values might differ significantly from one machine to another and are not that relevant. The fact that store speed and query speed stay in the same range when the index grows is of greater importance: this shows that the system is scalable. This limited effect of index size is expected to be similar on all machines.
Fig. Benchmark results for the FFT based algorithm. This algorithm, internally refered to as Olaf, finds peaks in the FFT and is not robust to pitch, speed or tempo changes.
Fig. Benchmark results for Constant-Q based algorithm. This algorithm, the main Panako algorithm and internally refered to as Panako, finds peaks in the Constant-Q domain and is robust to pitch, speed or tempo changes to some degree.