![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"source": [
"# noise2music-inspired automatic music captioning\n",
"\n",
"In [noise2music](https://google-research.github.io/noise2music/), the training dataset is created by pseudo-labeling a vast collection of unlabeled music audio using two advanced deep learning models. A large language model generates a diverse set of general music-related descriptive sentences to serve as potential captions. These captions are then matched to individual music clips through zero-shot classification, leveraging a pre-trained joint embedding model designed for music and text.\n",
"\n",
"So being curious, let's try the following:\n",
"\n",
"1. Generate a lot of music descriptions with a Meta Llama 3.2 LLM.\n",
"1. Embed the generated music descriptions with a LAION CLAP text encoder.\n",
"1. Index the text embeddings for nearest neighbor retrieval with FAISS.\n",
"1. Use the corresponding audio encoder to embed an audio example.\n",
"1. Use the audio embedding as search query for retrieving text embeddings.\n",
"\n",
"**Could this simple method produce reasonable audio captions?**"
],
"metadata": {
"id": "EVsGy9kxKMn-"
}
},
{
"cell_type": "code",
"source": [
"pip install -q datasets faiss-cpu"
],
"metadata": {
"id": "vhPejzFwbCo1"
},
"execution_count": 1,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import torch\n",
"import faiss\n",
"import transformers\n",
"import datasets\n",
"import polars as pl\n",
"import librosa as lr\n",
"import numpy as np\n",
"import tqdm.auto as tqdm\n",
"import seaborn as sns\n",
"\n",
"# Configure plotting.\n",
"pl.Config.set_fmt_str_lengths(256)\n",
"sns.set_style(\"ticks\")\n",
"sns.set_theme(\"notebook\")\n",
"\n",
"# Download some example audio files.\n",
"dataset = datasets.load_dataset(\"marsyas/gtzan\", trust_remote_code=True)\n",
"\n",
"# Download a pretrained text generation model.\n",
"text_generator = transformers.pipeline(\n",
" task=\"text-generation\",\n",
" model=\"meta-llama/Llama-3.2-1B-Instruct\",\n",
" torch_dtype=torch.bfloat16,\n",
" device_map=\"auto\",\n",
")\n",
"\n",
"# Download a pretrained CLAP model.\n",
"clap_model = transformers.ClapModel.from_pretrained(\"laion/larger_clap_general\")\n",
"clap_processor = transformers.ClapProcessor.from_pretrained(\"laion/larger_clap_general\")"
],
"metadata": {
"id": "Ktlv3YEs9agI"
},
"execution_count": 2,
"outputs": []
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 147
},
"id": "yrF4CdDk8irv",
"outputId": "a634a9f2-66cf-49de-fdf7-0652998e5f00"
},
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (1, 1)\n",
"┌──────────────────────────────────────────────────────────────────────────────────────────────────┐\n",
"│ generated_text │\n",
"│ --- │\n",
"│ str │\n",
"╞══════════════════════════════════════════════════════════════════════════════════════════════════╡\n",
"│ The piece features a haunting piano melody, punctuated by sparse strings and a subtle, pulsing │\n",
"│ bass line that creates an eerie, atmospheric backdrop for a whispered vocal │\n",
"└──────────────────────────────────────────────────────────────────────────────────────────────────┘"
],
"text/html": [
"| generated_text |
|---|
| str |
| "The piece features a haunting piano melody, punctuated by sparse strings and a subtle, pulsing bass line that creates an eerie, atmospheric backdrop for a whispered vocal" |
| generated_text | scores |
|---|---|
| str | f32 |
| "This piece of music features a haunting, atmospheric arrangement of eerie whispers and dissonant harmonies, punctuated by sudden, percussive bursts of sound that" | 2.301279 |
| "The piece is a haunting, atmospheric soundscape of pulsing synthesizers, eerie whispers, and a steady, pulsing heartbeat, evoking a sense of fore" | 2.252909 |
| "This piece features a haunting, atmospheric soundscape of whispers and creaks, punctuated by a sparse, pulsing rhythm that gradually builds into a crescendo" | 2.249522 |
| "The piece features a mesmerizing blend of eerie whispers, pulsating electronic beats, and haunting vocal harmonies that create an unsettling atmosphere, gradually building towards a dis" | 2.237405 |
| "This 5-minute composition features a gradual build-up of atmospheric textures, with layers of haunting piano and whispery vocals gradually giving way to a driving, pulsing" | 2.235468 |