AudioLoop Finds Rare Whale Calls 6x Faster than Random Selection

tl;dr: AudioLoop’s human-in-the-loop¹ active-learning loop, when used in “search” mode, surfaced rare (5% positive) Antarctic blue whale A calls from a ~50,000-clip audio dataset at a rate ~6.1x faster than random sampling across five different random-seed runs. On this rare-call search task, AudioLoop found about 384 positives where random review found about 63, using the same labeling budget.

Introduction

Motivation

I had two main use cases in mind when developing AudioLoop. The first one was the original impetus for the project: make it easier to label large unbalanced audio datasets. The second one presented itself as I developed the system and used it for my own testing: to find rare occurrences of rare signals in large unbalanced audio datasets. The second case is what I’ve tested here.

Label vs Search

In the configuration you can set your process to be in “label” mode (the first case) or “search” mode (the second case). At the moment the only difference between the two modes is the cycle-stopping strategy (how many training/labeling rounds to run), since overall model quality for label mode and rate of new-positive example discovery for search mode don’t necessarily peak at the same time.

This experiment tested the performance of search mode: how quickly can we surface positives given active human labeling feedback?

Finding

In the search mode test, AudioLoop substantially outperformed random clip selection. AudioLoop’s active learning loop found 6.1x more rare positive examples than random clip selection with the same labeling budget: 25 rounds of 50 reviewed clips, repeated across five seeds with low variance.

The Data

For these tests I’ve used the Australian Antarctic Data Centre’s AcousticTrends_BlueFinLibrary², a multi-location, multi-year labeled collection of Antarctic hydrophone recordings with labels for blue whale and fin whale vocalizations.

I’ve limited testing to the dataset’s “Bm_Ant-A” call type (the Antarctic blue whale’s A call), although this could easily be repeated with the several other types of labeled calls in the dataset. As usual with AudioLoop, we’re testing for presence or absence of a type of call in each example clip.

I used the raw sound files to produce 51,256 12-second clips, with 2,562 (5%) positive clips (centered on labeled calls), and 48,694 (95%) negative clips randomly sampled from other parts of the dataset. Recordings vary between hydrophone locations in the original dataset, so all audio was resampled to a 250Hz sample rate. Other calls are occasionally present in those negative clips. The error rate in the provided labels is unknown.

The Experiment

The Model

The training model used here is deliberately basic, in order to comparatively test random vs AudioLoop instead of testing the model itself. It’s the AudioLoop built-in “cnn5layer” model: three convolution layers and two linear layers. This model is used to select clips for labeling, and is retrained every loop on the working training set, which grows as the labeler verifies the selected clips each round. This model is powerful enough to learn the call to some extent, so the active learning loop isn’t just making up for a broken classifier.

The Labeler

For system testing like this with known datasets, AudioLoop has “evaluation mode”, which computes statistics on system performance and includes an optional auto-labeler, which labels clips based on ground truth without requiring the browser-based labeling tool. Using this, the loop can run several simulated labeling cycles without interruption.

The Run

The initial bootstrap training set included 20 positive and 20 negative examples. Starting from that training set the loop was run for 25 cycles, each producing 50 candidates, once with AudioLoop’s entropy-based selection strategy, and one with random selection. The model selected clips for the labeler to label based on the strategy in use. Already-labeled clips were removed from the unlabeled pool after each cycle.

This was repeated five times with five random starting seeds.

The Results

Each run generated 1,250 (25 cycles * 50 labels each cycle) new candidates. This table shows the positives found under each candidate selection method for each of the five random seed runs:

(± = 1 SD)

The number of positives found via random selection (63) matched the expected 5% of the 1,250 candidates (62.5), while the entropy strategy found over 6 times as many (383.8) with the same amount of labeling work applied.

Conclusion

On a 51,256-clip corpus of Antarctic blue-whale audio where only 5% of the clips contain the target call, AudioLoop’s active-learning loop using entropy-based candidate selection pulled out ~6.1x the positives of random selection, with all five random-seed runs in agreement.

Does this mean that using AudioLoop to find rare needle audio clips in a large haystack corpus is much easier than using random clip selection? At least with this corpus, it sure looks that way. There are, of course, other ways of limiting the search space. Scanning for strong signals in the A call frequency range would be the obvious one. For situations where you’re looking for “Blue whale ‘A’ call vs background noise,” that would likely work pretty well (it’s possible AudioLoop might work better, but that’s untested so far).

But for distinguishing between types of calls all in the same frequency range? This test actually covered that to an extent, by not removing other types of calls from the negative set. It would still be really interesting to see if using solely non-“A” calls as the negative clips would provide a significant improvement over random selection there. And that will almost certainly be my next test.

Code/Data

AudioLoop code is available. If you’d like the specific 51k corpus involved here, or the code I used to perform these tests, they’re not in a public repo yet, but feel free to email me if you’d like to take a look.