Classifying Environmental Sounds with CLAP and ONNX Runtime: A 1-NN Evaluation on ESC-50

Hello, everyone.

When people hear an animal call or keyboard typing, they can infer something about the source and setting. If we turn those sounds into numerical vectors, does their distance preserve the same kind of meaning?

Today, I will export the CLAP audio encoder to ONNX and compare its environmental sound embeddings across the 50 classes in ESC-50.

What I Tested

with ONNX Runtime and asks:

• Whether it consistently produces a normalized 512-dimensional embedding from a five-second environmental sound


• Whether a different recording from the same class becomes the nearest neighbor by cosine similarity


• Whether sounds still cluster near the correct one of five coarse categories when exact classes do not match


• How long the pipeline takes on an Apple Silicon CPU

The complete code and reproducible environment are available in the

The following commands fetch only this lab and run it:

git clone --depth 1 --filter=blob:none --sparse \

  https://github.com/kiarina/labs.git

cd labs

git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \

  2026/07/06/clap-onnx-esc50

mise -C 2026/07/06/clap-onnx-esc50 run

On the first run, the task downloads the checkpoint from Hugging Face, generates model.onnx inside the lab, and performs the evaluation. Later runs verify and reuse the ONNX model. The task also downloads ESC-50 metadata and the selected 100 WAV files from a pinned dataset revision.

Model and Dataset

The model is pinned to commit 84bcbbd1d619e407a8216371ddef36e458d95d93 of laion/clap-htsat-unfused. Using PyTorch 2.8.0, Transformers 4.57.3, and opset 18, the lab exports the audio encoder, projection layer, and L2 normalization.

SHA-256: b23099962830b1afa5398efbb6f5321ef8f63f8fcf93f5019837c47118a8a1c5

inputs: input_features [batch, 1, 1001, 64], is_longer [batch, 1]

output: embeddings [batch, 512]

This ONNX graph does not include the text encoder. The experiment therefore compares sounds with sounds instead of performing CLAP zero-shot classification from textual class names.

The input comes from ESC-50, a dataset of 2,000 five-second environmental recordings divided into 50 classes and five coarse categories:

• animals


• natural soundscapes and water


• human non-speech


• interior and domestic


• exterior and urban

The dataset is pinned to revision 33c8ce9eb2cf0b1c2f8bcf322eb349b6be34dbb6. The full ESC-50 dataset is distributed under CC BY-NC, so the noncommercial restriction applies to this experiment, which uses all 50 classes.

A Training-Free 1-NN Comparison

Instead of using all 2,000 recordings, the evaluation selects two clips per class, for 100 clips in total:

1. Take the first filename for every class in fold 1 as a 50-clip reference set


2. Take the first filename for every class in fold 5 as a 50-clip query set


3. Rank the reference clips for each query by cosine similarity


4. Check the nearest class, the top five classes, and the nearest coarse category

There is no classifier training and no five-fold cross-validation. This is a fixed 50-way, one-nearest-neighbor evaluation with only one reference recording representing each class.

The preprocessing pipeline is implemented in NumPy based on the checkpoint's preprocessor_config.json.

Each five-second ESC-50 clip is repeated and then padded to ten seconds. The implementation passes its log-mel spectrogram to ONNX and L2-normalizes the output again. For normalized embeddings, a dot product is equivalent to cosine similarity.

Results

One run on a Mac Studio without a warm-up produced:

fine accuracy@1:   45/50 = 90%

fine accuracy@5:   50/50 = 100%

coarse accuracy@1: 47/50 = 94%

100 clips elapsed: 5.458 s

seconds per clip:  0.055 s

For 45 of the 50 queries, the nearest reference was a different recording of the same fine-grained class. The correct class appeared among the top five references for every query. At the coarse-category level, 47 queries matched.

The five fine-grained top-1 errors were:

drinking_sipping → brushing_teeth and mouse_click → keyboard_typing missed the exact class but stayed within the same coarse category. They are also sounds associated with similar actions or settings. The remaining three errors crossed coarse-category boundaries.

The verification environment was:

• machine: Mac Studio (Mac16,9)


• chip: Apple M4 Max


• OS: macOS 26.5.1 (25F80), arm64


• Python: 3.12.11


• NumPy: 2.5.1


• ONNX Runtime: 1.27.0


• execution provider: CPUExecutionProvider

The elapsed time covers FFmpeg decoding, NumPy preprocessing, ONNX inference, and L2 normalization for all 100 files. It excludes downloads and model initialization. This was a single run without a warm-up, so it is a reference measurement rather than a rigorous benchmark.

Interpreting the Results

For these 100 selected clips, the CLAP audio embeddings preserved environmental sound categories remarkably well. Fine-grained top-1 accuracy reached 90%, and top-5 reached 100%, even though each class had only one reference. This suggests that the embedding distance captured information related to what produced the sound rather than merely matching identical acoustics.

The errors are interesting as well. Mouse clicks were close to keyboard typing, while drinking was close to tooth brushing. For audio search, returning semantically related sounds may be useful even when the exact label differs.

However, the 90% result is not directly comparable to a standard ESC-50 classification score. The evaluation uses only a fixed subset of folds 1 and 5 and depends heavily on one reference recording per class. It is neither cross-validation over all 2,000 clips nor a test of generalization to an unknown dataset. I also did not investigate possible overlap between CLAP's training data and ESC-50.

The ESC-50 repository documents potential information leakage from class-dependent preprocessing of the original source recordings. In addition, I have not verified that the NumPy preprocessing implementation is numerically identical to Hugging Face's ClapFeatureExtractor. Both are limitations when interpreting the metrics.

The pipeline took about 55 ms per clip. Because preprocessing represents each clip as ten seconds of audio, the result is fast enough to consider for real-time environmental sound retrieval. Still, this sequential 100-clip run does not compare batching or execution providers.

My Takeaway

After testing speaker embeddings, environmental sound embeddings make it clear that the phrase "turn audio into a vector" can hide very different kinds of retained information. Seeing separate recordings of frogs or typing return to nearby positions through nothing more than a 512-dimensional dot product makes CLAP feel promising as a foundation for audio search.

At the same time, a high similarity does not explain itself. The airplane → wind error scored 0.717; the model may have focused on wind noise in the airplane recording, or on something else entirely. This evaluation cannot tell us why.

Next, I would like to test an audio tagging model and see whether it can label multiple acoustic events within a single recording. This will offer a different perspective from embedding-based similarity search and help reveal how much we can identify about what is happening in an environment.