Using Qwen2-Audio to transcribe music into sheet music
Automatic music transcription is the process of converting audio files like MP3 and WAV into sheet music, guitar tablature, and any format a musician may want to learn a song on their instrument.
We’ll go over the best current tools for doing this, which happen to be deep learning-based, and a novel approach for it.
Current state of the art
The current state-of-the-art for this task comes from in 2021 which used a T5-inspired transformer model (similar to , achieving great results. The problem is approached as a sequence-to-sequence task using an encoder-decoder Transformer architecture. The encoder processes mel spectrogram frames as input and produces embeddings, while the decoder uses these embeddings via cross-attention to autoregressively generate a sequence of MIDI-like tokens. Their vocabulary consisted of four types of tokens:
- Note tokens (128 values for MIDI pitches)
- Velocity tokens (128 values including zero for note-off)
- Time tokens (6,000 values in 10ms bins for absolute timing)
- EOS token (to mark sequence end)
See the image below for a visualisation of the architecture and an example sequence of their custom MIDI tokens:
and is designed to handle text, audio and images “natively”. Although OpenAI has not released the technical details on this, it’s assumed that some weights in the network will process all modalities. It’s possible that the model uses a decoder-only architecture like language only GPT models without the need for encoder components to convert different modalities to a dense representation first. This design allows the model to seamlessly process and interpret inputs like text and images together, potentially offering performance benefits both computationally and in terms of model understanding.
Many multi-modal models take a simpler approach reminiscent of the encoder-decoder architecture: they combine two pre-trained models — an encoder for the specific input modality (like is a prime example of this architecture for vision-language tasks, while apply the same principle using audio encoders instead of vision encoders.
Here’s some pseudocode on how the models are stitched together:
# Extract features from final layer of audio encoder
# Shape: [batch_size, audio_seq_len, encoder_dim=1024]
audio_features = audio_model(audio_input)
# Project audio features to match LLM's embedding dimension
# Shape: [batch_size, audio_seq_len, llm_embed_dim=4096]
audio_embeddings = projection_layer(audio_features)
# Get text embeddings from LLM's embedding layer
# Shape: [batch_size, text_seq_len, llm_embed_dim=4096]
text_embeddings = llm.embed_text(text_input)
# Concatenate along sequence length dimension
# Shape: [batch_size, audio_seq_len + text_seq_len, llm_embed_dim=4096]
combined_input = concatenate([audio_embeddings, text_embeddings], dim=1)
# Feed them into the LLM as normal for generation
output = llm(combined_input)
Spotify Llark and Qwen2-Audio
Overview of architecture
Llark uses for the audio towers. Jukebox is a music generation model but it can also take in audio clips as input and outputs a continuation of the audio clip. Whisper is used for transcribing voice to text.
Given their purpose, the choice of audio module is clear: Llark specialises in music analysis, while Qwen2Audio primarily focuses on responding to voice instructions with some basic audio and music analysis capabilities.
Determining the optimal source for extracting embeddings from large pre-trained models involves research and experimentation. Additionally, deciding whether to fine-tune the entire module or freeze parts of it is a crucial design choice. For instance, LlaVa’s training strategy involves freezing the vision tower and focusing on fine-tuning the projection layer and language model. We’ll go over this aspect of each model below.
Llark: why Jukebox? Are these embeddings the best as of September 2024?
Determining the optimal location to extract embeddings from large models typically requires extensive probing. This involves testing various activations or extracted layers of the model on different classification tasks through a process of trial and error. For music generation models, this could include tasks like genre recognition, instrument detection, emotion detection, as well as analysis of harmonic structures and temporal patterns. Many commercial embedding models (lik, embeddings extracted from Jukebox appear to outperform MusicGen on average in classification tasks. The findings from this paper led to the authors of Llark using the following approach for extracting embeddings:
- Embeddings are derived from the output of the 36th layer of the Jukebox encoder following the approach described in
- :
We also plan to build larger Qwen2-Audio models to explore the scaling laws of audio language models.
For my own learning though, I did have a go at re-creating the model using torch and pre-trained models with the transformers library.
I also created datasets for Q&A data and embeddings. I generated short form Q&A data for the URMP dataset, e.g.: “What is the tempo of this track”, “What instruments are playing in this audio”.
.
as the output format that the language model is expected to transcribe the music in. Here’s an example of it:
X:1
M:4/4
L:1/16
K:none
Q:67
V:1 name="Electric Bass (finger)"
%%octave-default C4
GAA^2E3A2<A^2 | D^D^2E2A2A^4 A^2E2 | A2A^4A^2E2 A2A^4 | A^2E2A2A^4A^2E2A2 |
A^4 A^2E2 A2A^4A^2 E2 | A2A^4 |
V:2 name="Bright Acoustic Piano"
%%octave-default C5
[E3C3][E3C3][E3C3] [E3C3][A^,2E2A^2] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3] |
V:3 name="Electric Guitar (jazz)"
%%octave-default C5
E'3C'3A^4E'3C'3 | A^4E'3 C'3A^4E'3C'3 | A^4 E'3C'3A^4 E'3C'3 | A^4E'3C'3A^4E'3C'3 |
A^4E'3C'3 A^4E'3C'3 | A^4 |In this notation we have the time signature and tempo defined at the top denoted by ‘M’ and ‘Q’. The ‘L’ indicates the default note length of the notation, in this case a sixteenth note, which is the norm. We then define each instrument and the default octave they should adhere to when writing the notes for each of them. Here’s a summary of the key syntactical points for writing notes in ABC music notation:
- Notes are represented by letters A-G, with lowercase letters indicating higher octaves
- Sharps are denoted by ^ before the note, flats by _
- Natural signs are represented by =
- Note length is indicated by numbers after the note (C2 is twice as long as C)
- Dotted notes use a . after the note (C. is a dotted quarter note)
- Rests are represented by z, with numbers for duration (z2 is a half rest)
- Chords are enclosed in square brackets [CEG]
- Ties are shown with a hyphen -
- Bar lines are represented by |
- Broken rhythms use > or < between notes (C>D means dotted-C eighth note followed by D sixteenth note)
Why ABC?
The reasons for choosing this notation are:
- It’s a minimalist format for writing music
- It’s widely used and popular; language models already have good comprehension of ABC notation due to extensive pre-training on it.
- It’s flexible and can easily be extended to include tempo changes, time signature changes, additional playing styles like mentioned above, etc…
I converted the MIDI files provided by the datasets to ABC notation using .
Evaluation
To evaluate both the original model and each stage of fine-tuning I performed thereafter, I randomly selected 30 samples of varying complexity from the URMP dataset and ran the model three times on each sample, manually examining all responses.
Through manual testing, I found the optimal decoding parameters to be a temperature of 0.7 and a top_p of 1.2. The maximum number of tokens to return was capped at 2048. Adjusting the max seemed to have little difference on performance.
The original model performed poorly on this evaluation set. While it occasionally predicted the tempo and instruments correctly, it mostly failed to do so. A text file with the evaluation results is available on here explaining PPO!
The key insight of PPO is that instead of trying to directly backpropagate through the non-differentiable steps, it:
- Treats the model’s outputs as actions in a reinforcement learning framework
- Uses the custom loss function as a reward signal
- Updates the model’s policy (its probability distributions over tokens) to maximise expected reward
- Does this while ensuring the updated policy doesn’t deviate too far from the current one
This approach allows us to effectively train the model with the custom loss function, ensuring performance improvements without disrupting the core training dynamics. The PPO algorithm’s conservative update strategy helps maintain stability during training, which is particularly important when working with large language models.
Usually, this scoring function would be implemented as a separate LLM in the form of a “reward model” commonly used when fine-tuning models via RLHF, which was a breakthrough first introduced when ChatGPT came out. Due to the nature of this task, we can manually write code to score the responses, which uses fewer resources and is quicker.
For time signature and tempo recognition this is easy to calculate. We extract all predicted items with regex, for example extracting the metre:
def extract_metre(self, abc_string):
return re.search(r'M:(\S+)', abc_string).group(1)The model should learn the syntax and structure we want it to output in the SFT stage. If it outputs something that will cause our regex to not find anything or error, we can just skip that sample, assuming it’s a small minority of the dataset.
We extract the predicted tempo and write a function that is more forgiving for small errors but penalises larger errors more heavily:
- For small differences (≤10 BPM), it uses linear scaling.
- For larger differences, it switches to exponential scaling.
- The final loss is capped between 0 and 1.
Let’s break down the key components of this custom loss:
Code for the custom loss is .
I tracked various training metrics using Weights & Biases (WandB). Key metrics included the policy loss, value loss, total loss, KL divergence, and the reward model’s score.
For all hyperparameter runs, the logs no improvement in the rewards and loss calculated over time. The KL divergence remained within the pre-defined threshold.
Conclusion
While this initial experiment didn’t achieve the desired performance in music transcription, we’ve provided some groundwork for future developments in the space. The challenges encountered have provided valuable insights into both the technical requirements and potential approaches for tackling this complex task. Future work could explore several promising directions:
- Experimenting with larger pre-trained models as they become available
- Expanding the training dataset with more diverse musical examples
- Further refinement of the reward functions to capture more nuanced musical relationships
- Exploring hybrid approaches that combine traditional music processing techniques with language model capabilities
was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
↗ Original-Artikel auf towardsdatascience.com lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf towardsdatascience.com.
SOCIAL SHARE CARD GENERATOR