{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "vcxHd6VB31a1" }, "source": [ "# Speaker Diarization with AssemblyAI" ] }, { "cell_type": "markdown", "metadata": { "id": "wwjCesZA4gqu" }, "source": [ "> 📚 This cookbook has an accompanying article with a complete walkthrough \"[Level up Your RAG Application with Speaker Diarization](https://haystack.deepset.ai/blog/level-up-rag-with-speaker-diarization)\"\n", "\n", "LLMs excel with text data, answering complex questions without manual reading or searching. When dealing with audio or video, providing transcription is key. Transcription captures spoken content of the audio or video, but in multi-speaker recordings, it may miss non-verbal information and fail to convey speaker count or individual remarks. Therefore, to maximize the LLM's potential with such recordings, **Speaker Diarization** is essential!\n", "\n", "In this example, we'll build a RAG application with speaker labels for audio files. This application will use [Haystack](https://github.com/deepset-ai/haystack) and speaker diarization models by [AssemblyAI](https://www.assemblyai.com/).\n", "\n", "📚 Useful Sources:\n", "* [Integration: AssemblyAI](https://haystack.deepset.ai/integrations/assemblyai)" ] }, { "cell_type": "markdown", "metadata": { "id": "FIJ2fJbR696M" }, "source": [ "## Install the Dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "tx0uReQyl9cD", "outputId": "3551a885-f16c-4a26-94be-3edff5b9c9be" }, "outputs": [], "source": [ "%%bash\n", "pip install haystack\n", "pip install assemblyai-haystack\n", "pip install \"sentence-transformers>=3.0.0\"\n", "pip install \"huggingface_hub>=0.23.0\"\n", "pip install --upgrade gdown" ] }, { "cell_type": "markdown", "metadata": { "id": "XnTLMNd-UxqR" }, "source": [ "## Download The Audio Files\n", "\n", "We extracted the audio from youtube videos and saved them in a Google Drive Folder for you: https://drive.google.com/drive/folders/10zsFuHmj3oytYMyGrLdytpW-6JzT9T_W?usp=drive_link\n", "\n", "You can run the code below to download the audio files to this colab notebook under \"Files\" tab on the left bar." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1uTvCiysUOtU", "outputId": "6c339a35-a91a-43f0-ee71-45cb8f78c5f7" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieving folder contents\n", "Processing file 12654ySXSYc2rZnPgNxXZwWt2hH-kTNDZ Netflix_Q4_2023_Earnings_Interview.mp3\n", "Processing file 1Zb15D_nrBzWlM3K8FuPOmyiCiYvsuJLD Panel_Discussion.mp3\n", "Processing file 1FFKGEZAUSmJayZgGaAe1uFP9HUtOK5m- Working_From_Home_Debate.mp3\n", "Retrieving folder contents completed\n", "Building directory structure\n", "Building directory structure completed\n", "Downloading...\n", "From: https://drive.google.com/uc?id=12654ySXSYc2rZnPgNxXZwWt2hH-kTNDZ\n", "To: /content/Netflix_Q4_2023_Earnings_Interview.mp3\n", "100% 39.1M/39.1M [00:00<00:00, 67.6MB/s]\n", "Downloading...\n", "From: https://drive.google.com/uc?id=1Zb15D_nrBzWlM3K8FuPOmyiCiYvsuJLD\n", "To: /content/Panel_Discussion.mp3\n", "100% 21.8M/21.8M [00:00<00:00, 60.4MB/s]\n", "Downloading...\n", "From: https://drive.google.com/uc?id=1FFKGEZAUSmJayZgGaAe1uFP9HUtOK5m-\n", "To: /content/Working_From_Home_Debate.mp3\n", "100% 4.45M/4.45M [00:00<00:00, 34.8MB/s]\n", "Download completed\n" ] } ], "source": [ "!gdown https://drive.google.com/drive/folders/10zsFuHmj3oytYMyGrLdytpW-6JzT9T_W -O \"/content\" --folder" ] }, { "cell_type": "markdown", "metadata": { "id": "GQ78gO-1VhGL" }, "source": [ "## Add Your API Keys\n", "\n", "Enter the API keys from [AssemblyAI](https://www.assemblyai.com/) and [Hugging Face](https://huggingface.co/settings/tokens):" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-fQs9gPBVkmS", "outputId": "284ce3ce-084c-4d24-c7ab-1a8e2b436d3e" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Enter your ASSEMBLYAI_API_KEY: ··········\n", "HF_API_TOKEN: ··········\n" ] } ], "source": [ "import os\n", "from getpass import getpass\n", "\n", "ASSEMBLYAI_API_KEY = getpass(\"Enter your ASSEMBLYAI_API_KEY: \")\n", "os.environ[\"HF_API_TOKEN\"] = getpass(\"HF_API_TOKEN: \")" ] }, { "cell_type": "markdown", "metadata": { "id": "JswXLocxyQX6" }, "source": [ "## Index Speaker Labels to Your DocumentStore\n", "\n", "Build a pipeline to generate speaker labels and index them into a DocumentStore with their embeddings. In this pipeline, you need:\n", "\n", "* [InMemoryDocumentStore](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore): to store your documents without external dependencies or extra setup\n", "* [AssemblyAITranscriber](https://github.com/AssemblyAI/assemblyai-haystack/blob/main/assemblyai_haystack/transcriber.py): to create speaker_labels for the given audio file and convert them into Haystack Documents\n", "* [DocumentSplitter](https://docs.haystack.deepset.ai/docs/documentsplitter): to split your documents into smaller chunks\n", "* [SentenceTransformersDocumentEmbedder](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder): to create embeddings for each document using sentence-transformers models\n", "* [DocumentWriter](https://docs.haystack.deepset.ai/docs/documentwriter): to write these documents into your document store\n", "\n", "Note: The speaker information will be saved in the `meta` of the Document object" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 972 }, "id": "MViU7DFQmO9q", "outputId": "01d0b380-e39a-4407-e888-e5fb71e1ae8b" }, "outputs": [], "source": [ "from haystack.components.writers import DocumentWriter\n", "from haystack.components.preprocessors import DocumentSplitter\n", "from haystack.components.embedders import SentenceTransformersDocumentEmbedder\n", "from haystack import Pipeline\n", "from haystack.document_stores.in_memory import InMemoryDocumentStore\n", "from assemblyai_haystack.transcriber import AssemblyAITranscriber\n", "from haystack.document_stores.types import DuplicatePolicy\n", "from haystack.utils import ComponentDevice\n", "\n", "speaker_document_store = InMemoryDocumentStore()\n", "transcriber = AssemblyAITranscriber(api_key=ASSEMBLYAI_API_KEY)\n", "speaker_splitter = DocumentSplitter(\n", " split_by = \"sentence\",\n", " split_length = 10,\n", " split_overlap = 1\n", ")\n", "speaker_embedder = SentenceTransformersDocumentEmbedder(device=ComponentDevice.from_str(\"cuda:0\"))\n", "speaker_writer = DocumentWriter(speaker_document_store, policy=DuplicatePolicy.SKIP)\n", "\n", "indexing_pipeline = Pipeline()\n", "indexing_pipeline.add_component(instance=transcriber, name=\"transcriber\")\n", "indexing_pipeline.add_component(instance=speaker_splitter, name=\"speaker_splitter\")\n", "indexing_pipeline.add_component(instance=speaker_embedder, name=\"speaker_embedder\")\n", "indexing_pipeline.add_component(instance=speaker_writer, name=\"speaker_writer\")\n", "\n", "indexing_pipeline.connect(\"transcriber.speaker_labels\", \"speaker_splitter\")\n", "indexing_pipeline.connect(\"speaker_splitter\", \"speaker_embedder\")\n", "indexing_pipeline.connect(\"speaker_embedder\", \"speaker_writer\")" ] }, { "cell_type": "markdown", "metadata": { "id": "60UE9CfnWTqo" }, "source": [ "Give an `audio_file_path` and run your pipeline" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 525, "referenced_widgets": [ "fb104e78e948418198e07f391b2a70bc", "a11b00ff64714026a2ea548673cf859a", "1996ac5964f648ec8380b588a6f4d219", "3ff328d985434d9f8fc459f2714f692d", "d6c5a025f7c7468480fd5547d36e7ea3", "aee161e374b44d29ada850abfca1ce11", "239be7572d5b450bb1cd62ceca7cfe54", "450f168e3ba34915ad4cdbdfd0baf86b", "6f042af0ee274cb498159ffd10e66f25", "788a6c4c2c4449cdb50ab3f9a0e38bd8", "2ce879e438774bc1a308687a14c46bd1", "64934d2f6397469a929486a3894f7b36", "52e2fdb239274133a2775025e8e58bde", "0d3547ee0f16477c8f72e9d265a3fb71", "90d48eaf9277495db1e2d6a103c3997d", "876fe8208a504262803c0a9d671face6", "702684e5122646d391b428041cacbbc8", "4b5e29e48cad42658c53554dd681e82a", "670bf2e27db547e3a43b99c5b3702769", "2fc404bcdc2f442889ac9b0e0cff25f6", "a58f8515a3f744a2825c4399805c7aba", "9540e478739144bfb45087c5c5cd3ee1", "754e6e42c8c04c5a85395e84762b94ed", "29c8b0fd63d847e2afbf325bf9eabe6c", "7bfe7814d7514816b59b165cd3a0ece7", "94f58859b1ff4e309827f8b0e489e3bd", "d8c186f4767f4c71a0d146eaec79b495", "c156ad4d803a4a7aac7f38bad94eb1b6", "6c446ed5adf2481b8db02c6f5b7bb6f1", "6860ebe81e24499b855a812e171e8a7d", "7142ac99a8264d839a7af22d6024149c", "60be5b31c15e4d848f0578ba0419264d", "90692f27c7714c6ab4ddb80e2e6ed6dc", "c13163bdfc0c4d44a959423d8c603cf9", "349c9bba0db64aad922fee469c72c631", "e13d399a56534456baffb772ec18a405", "21808338519046cbba116420cb84c6bd", "98d0e4e4af274b5caf523702153e71ac", "3d87128432a347a5a9b0dfe162eeddf8", "922ce9fbd5b54411a02c58780add9169", "cbec6ebaba4c4ff886961db00caad8dc", "4cb2ceb9636d492ca0a2f55e6953423e", "f376deae19bf482a9f5371bebfda8ec9", "a0da059ca41d4e388a60b55a3fb11be5", "fab3f3adabe2437787183fb2f07a5aaa", "6ce1a71e6ffa491e972f816b7a684946", "8fec563d490e4636a02cee698d682ed3", "264b39ac17ca48e7a97d18fab2e3f5c6", "42161d1a6ce245daba037eff91cd38e5", "c6b03d5589854052a192ec30c58a66a1", "cdd986790d294086a036a6053087dc5b", "ed06c4bdaaf043ea837536a8084c5764", "be0ed7af5339492abf9f84a14af8fc2e", "a566d00da3a341949871b7b186968476", "c8c2a9f4b5a14b36a94fc037d5df335f", "fb4bc2e0f4454f8da5d1b0d695b541bd", "37d397ee3b2947b081e4e8378bf1b25d", "76d9513978924cf39dbc0113a9a559fb", "771957cc6748459788729d95ae963f50", "e4b133bc05244e128866c3e48eeeb056", "1599ea30d1fb4bd49023ac7578d1f996", "e7955792cc1f48c1971a4444bc5b76aa", "62276a7d6af14eda84d965397433f412", "20e4a72c62e147b08a4aaad9b5e8de44", "78e67a8ad41240199888e8ce8f4eb89b", "1030963c2e1d47b0bcb56c270341b728", "73f4271a98744a33867613cdebc67778", "1a9e9c242db74f528a406f9463c36169", "fb258bc1bdae4ff9a448bd7a798b50b3", "8384a62b63774dbd9a4395c6200486f3", "995ccc308fac4d48966e3735aaaae76d", "c7178adf02dc474684337c2a77cdf1ee", "071fcdf48c514738924d7284492df098", "04d7a9e4fa044eca8a92e81714af57bf", "f496ac3f0fa54d7fa4fe623e38a0122b", "ae3b99f3782242f68799751d75296bbf", "307e30311e724e7881a802a6f7ed892e", "486d21eb304d4458aa28748285e1b914", "ed83d1904eca4939988fea72877e9ec4", "6de04454bd77487d9be9a906d9a4d379", "5eee0fca5d2a491fbc3d92d0f78a4973", "9c5e5888ccdc4c6bb90087113b09ca5e", "26e6007a9f254997b76873e854334025", "35349f1a5f27481db29c5ceebda7ea59", "8d40ca81c23240a69df52f868586e7d5", "c98297e0804d446b807b2506bf20543b", "18c5628ef91e42c1ba9beae30a5f5aa2", "9d69c164f9bd4a129a3796ebba362d14", "5d943f8626ff448f9e3b676632109961", "99fcecb6faf747bd93e3d63f2272557b", "1cc9cf4028f544e1b18cd7c0d9bbd6f1", "05b3e291666345bfa15df73d0854b085", "104371c093b54343ae80e2361c637abb", "b42e71935efa41fba373093302e918f9", "7b13b06b939743aabc2a95c778de246b", "39e9030e4dcd42b4966cbfa497ab755a", "bf2bd1c77a804abab085d86483e7bc6c", "e11fd4e8eb024bb695b244798aa573cc", "37efbefdba1a4b4ba70a8626a726dbcc", "b4cb3ca1d698494c914d19dc754ae7e0", "d6c8470a321a41e9a284085f0a6aa27c", "a5290376152642198e199ceb8590f827", "617423fc4eca4cf188c9861555e83e1f", "d85efa8a423649ff9a232973b8485630", "ab1e123b70954143aec283fadb3c922b", "2d9cf9bd82584d608271bdc9632c65f5", "22dd38cc067d4fdb9cd217d56ff43322", "bc3e89e97de343e594b8f66fade2a617", "03e49e129bd6475f95932ea77802412f", "e65c35c1b4f04fcdb375e3fe8b664386", "6bafe820272941e3a12386b3ae383412", "dbc3e8d7dc0c438f84f2ef869e98fb3a", "bf7b50e1b4bd44aead215d3bfe8c69b2", "af9d257812f542c3bd6e932e5e8edb36", "33ba9188c7214149a8eb3688c6fc2f90", "e46d3b15ef464c33bdc7053e9aa1cf6d", "3b550465d7a645e49c467294af319a79", "4bb91be2d7d94959835752f498341234", "0a62174702cc4db4b67826ff1bf6d99b", "0a5fb89a8bdc4148b2a41b372cf98d55", "eeff0e2f393f427b938d0c71a4fae63a", "c538120c2a9246e785ba02d0f837a4d9", "6b61c6a7e7784f749d26a713a861b517", "94b8b39d8bf34a41b9d279a6754cb526", "8fbb0e7e9b9c4d66951e885023c116a2", "6a05afe5498647788993cf088d9844f7", "26bbcfcf53e047679c599ef385c9ab3e", "7e382fa68b4d4cc18acfa26fece1f82f", "37d30e3fa3c54d08b93013916731efa7", "e445cb9d94b544839e8427bf2065636a", "d782db52c54c4c20aa3eea719329d844", "c6eb9cbd7f0a484c94d8b91271053f51" ] }, "id": "ey5Lnwvun5uU", "outputId": "a3205325-cf90-4f00-c3b6-8d647f902e53" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/sentence_transformers/SentenceTransformer.py:92: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v3 of SentenceTransformers.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fb104e78e948418198e07f391b2a70bc", "version_major": 2, "version_minor": 0 }, "text/plain": [ "modules.json: 0%| | 0.00/349 [00:00 The LLM in the example ([`mistralai/Mixtral-8x7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)) is a gated model. Make sure you have access to the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "h476nud3pWM0", "outputId": "560c4dc2-96d0-4b07-c722-e678b0df9219" }, "outputs": [], "source": [ "from haystack import Pipeline\n", "from haystack.components.builders.prompt_builder import PromptBuilder\n", "from haystack.components.generators import HuggingFaceAPIGenerator\n", "from haystack.components.embedders import SentenceTransformersTextEmbedder\n", "from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever\n", "from haystack.utils import ComponentDevice\n", "\n", "prompt = \"\"\"\n", "You will be provided with a transcription of a recording with each sentence or group of sentences attributed to a Speaker by the word \"Speaker\" followed by a letter representing the person uttering that sentence. Answer the given question based on the given context.\n", "If you think that given transcription is not enough to answer the question, say so.\n", "\n", "Transcription:\n", "{% for doc in documents %}\n", " {% if doc.meta[\"speaker\"] %} Speaker {{doc.meta[\"speaker\"]}}: {% endif %}{{doc.content}}\n", "{% endfor %}\n", "Question: {{ question }}\n", "<|end_of_turn|>\n", "Answer:\n", "\"\"\"\n", "\n", "retriever = InMemoryEmbeddingRetriever(speaker_document_store)\n", "text_embedder = SentenceTransformersTextEmbedder(device=ComponentDevice.from_str(\"cuda:0\"))\n", "answer_generator = HuggingFaceAPIGenerator(\n", " api_type=\"serverless_inference_api\",\n", " api_params={\"model\": \"mistralai/Mixtral-8x7B-Instruct-v0.1\"},\n", " generation_kwargs={\"max_new_tokens\":500})\n", "prompt_builder = PromptBuilder(template=prompt)\n", "\n", "speaker_rag_pipe = Pipeline()\n", "speaker_rag_pipe.add_component(\"text_embedder\", text_embedder)\n", "speaker_rag_pipe.add_component(\"retriever\", retriever)\n", "speaker_rag_pipe.add_component(\"prompt_builder\", prompt_builder)\n", "speaker_rag_pipe.add_component(\"llm\", answer_generator)\n", "\n", "speaker_rag_pipe.connect(\"text_embedder.embedding\", \"retriever.query_embedding\")\n", "speaker_rag_pipe.connect(\"retriever.documents\", \"prompt_builder.documents\")\n", "speaker_rag_pipe.connect(\"prompt_builder.prompt\", \"llm.prompt\")" ] }, { "cell_type": "markdown", "metadata": { "id": "hVRO6z5xx6mg" }, "source": [ "## Test RAG with Speaker Labels" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 137, "referenced_widgets": [ "2f25c7ffba8243a79c37ee8f02735376", "d5f16ef3ad4d458cb7ab311970145270", "6a008e72ce234297abd3606fd6cf6761", "dbae8bd7aeaa49339882b7b063d7b8e6", "9f8da80a7aaa45d09fe2754b77a4401a", "2e6bea12e93249be9d4f7ce35089fb0f", "ce0001183b37463c8db0ab5cbcea1c1d", "0b4c13a52f114b138ab00e15ffdfaaed", "385576dab98c46a898c9f13d4d1e95b2", "f7335e5e31744b6c9e36569eacd3d739", "14778956f5284932bb907100e546afd0" ] }, "id": "9wNEr8QrJxSS", "outputId": "4ec26893-bda9-4bf6-fdb2-c6da03567c12" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2f25c7ffba8243a79c37ee8f02735376", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00