![](\"https://haystack.deepset.ai/images/haystack-ogimage.png\")
![](\"https://github.com/prometheus-eval/prometheus-eval/blob/main/assets/logo.png?raw=true\"){kind=logo}
\n",
"\n",
"Evaluating the responses of Language Models and LLM-based applications often involves using model-based metrics that do not require ground truth labels. Large proprietary models like GPT-4 and Claude 3 Opus are frequently employed as evaluators and demonstrate a good correlation with human evaluations.\n",
"\n",
"However, relying on **closed models** poses several challenges:\n",
"- fairness: the training data of these models is unknown.\n",
"- controllability: the behavior of these models can change unpredictably.\n",
"- data privacy: sending data to external providers may raise privacy concerns.\n",
"- affordability: using these powerful models can be expensive.\n",
"\n",
"Using **open models** for evaluation is an active research area, but their practical use is often limited. They typically do not correlate well with human judgments and lack flexibility.\n",
"\n",
"**🔥 Prometheus 2** is a new family of open-source models designed to address these gaps:\n",
"- two variants, respectively fine-tuned from Mistral-7B and Mixtral8x7B\n",
"- trained on open-source data\n",
"- demonstrate high correlation with human evaluations and proprietary models\n",
"- highly flexible: capable of performing direct assessments and pairwise rankings, and allowing the definition of custom evaluation criteria.\n",
"\n",
"---\n",
"\n",
"In this experimental notebook, we will use Prometheus 2 to evaluate the responses of a RAG pipeline.\n",
"\n",
"First, we will build the RAG pipeline and collect some results. Then, we will code a custom Prometheus Evaluator component for Haystack. Finally, we will initialize three different evaluators and run them in an evaluation pipeline.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xJHqt7x3SsZM"
},
"source": [
"## Create the RAG pipeline to evaluate"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0l3W4IuOGW9J"
},
"source": [
"We want to use Prometheus 2 to evaluate the answers generated by a RAG, so we first need to build our RAG Pipeline.\n",
"\n",
"This part is quite similar to the [\"Evaluating RAG Pipelines\" tutorial](https://haystack.deepset.ai/tutorials/35_evaluating_rag_pipelines#create-the-rag-pipeline-to-evaluate). Take a look at it for more details.\n",
"\n",
"**If you want, you can simply read this section. We will provide the generated data for later evaluation steps.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6o7DaedMCtK9",
"outputId": "3f63c330-b856-477b-a327-04621e2f6e90"
},
"outputs": [],
"source": [
"!pip install haystack-ai datasets sentence-transformers accelerate huggingface_hub bitsandbytes"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j5q8l6pCHiYt"
},
"source": [
"We will be using [a labeled PubMed dataset](https://huggingface.co/datasets/vblagoje/PubMedQA_instruction/viewer/default/train?row=0) with questions, contexts and answers. This allows us to use the contexts as Documents and provides the necessary labeled data for some of the evaluation metrics we will define.\n",
"\n",
"In this example, we will use the first 100 rows.\n",
"\n",
"First, let's fetch the dataset and extract `all_documents`, `all_questions` and `all_ground_truth_answers`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "F8ncIBqJCxxE"
},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"from haystack import Document\n",
"\n",
"dataset = load_dataset(\"vblagoje/PubMedQA_instruction\", split=\"train\")\n",
"dataset = dataset.select(range(100))\n",
"all_documents = [Document(content=doc[\"context\"]) for doc in dataset]\n",
"all_questions = [doc[\"instruction\"] for doc in dataset]\n",
"all_ground_truth_answers = [doc[\"response\"] for doc in dataset]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "J-nvy3J4IAXh"
},
"source": [
"### Indexing pipeline\n",
"\n",
"Next, let’s build a simple indexing pipeline and write the documents into a Document Store."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156,
"referenced_widgets": [
"6153459519fd438d941bd350a1157a81",
"e8a1fc3777c14cb299d3af22e59a1e80",
"e22602f0fd8e4698865efcb5c3eb5ef8",
"80e58852e2944f03bc2c64545ef46b7c",
"025a1ecf541947859154192d6b4967eb",
"a20e817e91e64cd79eb3dd09c49261d6",
"a93d30b57c6142efb14d4d624d3af359",
"bb90005096ce4074b5b6705d7ead6ba0",
"d2ad7528fa394d40a095fe1f63043035",
"257e5391d78a4c979a9d22193ed9fecd",
"770b6517d9404ca1a72a224066ae02ad"
]
},
"id": "Z0AH0ysrCysR",
"outputId": "86880fde-ddaf-4dfd-9d34-38f2bc33ffb5"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/sentence_transformers/SentenceTransformer.py:173: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v3 of SentenceTransformers.\n",
" warnings.warn(\n",
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6153459519fd438d941bd350a1157a81",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Batches: 0%| | 0/4 [00:00\n",
"🚅 Components\n",
" - query_embedder: SentenceTransformersTextEmbedder\n",
" - retriever: InMemoryEmbeddingRetriever\n",
" - prompt_builder: PromptBuilder\n",
" - generator: HuggingFaceLocalGenerator\n",
" - answer_builder: AnswerBuilder\n",
"🛤️ Connections\n",
" - query_embedder.embedding -> retriever.query_embedding (List[float])\n",
" - retriever.documents -> prompt_builder.documents (List[Document])\n",
" - retriever.documents -> answer_builder.documents (List[Document])\n",
" - prompt_builder.prompt -> generator.prompt (str)\n",
" - generator.replies -> answer_builder.replies (List[str])"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import os\n",
"from getpass import getpass\n",
"from haystack import Pipeline\n",
"from haystack.components.builders import AnswerBuilder, PromptBuilder\n",
"from haystack.components.embedders import SentenceTransformersTextEmbedder\n",
"from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever\n",
"from haystack.components.generators import HuggingFaceLocalGenerator\n",
"from haystack.utils import ComponentDevice\n",
"\n",
"\n",
"# to access Gemma\n",
"# 1. you need a Hugging Face account\n",
"# 2. you have to accept Google conditions: https://huggingface.co/google/gemma-1.1-2b-it\n",
"# 3. copy your HF token (https://huggingface.co/settings/tokens) and paste it below\n",
"os.environ[\"HF_API_TOKEN\"] = getpass(\"Your Hugging Face token\")\n",
"\n",
"generator = HuggingFaceLocalGenerator(\n",
" \"google/gemma-1.1-2b-it\",\n",
" huggingface_pipeline_kwargs={\"device_map\": \"auto\"},\n",
" device=ComponentDevice.from_str(\"cuda:0\"),\n",
")\n",
"\n",
"template = \"\"\"\n",
"| \n", " | score | \n", "
|---|---|
| correctness_evaluator | \n", "3.9 | \n", "
| response_relevance_evaluator | \n", "4.3 | \n", "
| logical_robustness_evaluator | \n", "3.5 | \n", "
| \n", " | question | \n", "answer | \n", "predicted_answer | \n", "correctness_evaluator | \n", "response_relevance_evaluator | \n", "logical_robustness_evaluator | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "Is cDK1 and CDK2 activity a strong predictor of renal cell carcinoma recurrence? | \n", "CDK1SA of tumors and the CDK2SA are both associated with recurrence and prognosis. | \n", "**Yes, cDK1 and CDK2 activity is a strong predictor of renal cell carcinoma recurrence.**\\n\\nThe study found that tumors with low CDK2SA-CDK1SA ratio showed significantly better 5-year recurrence-free survival than those with high CDK2SA-CDK1SA ratio. | \n", "4 | \n", "3 | \n", "2 | \n", "
| 1 | \n", "Does metabolic control analysis of the Trypanosoma cruzi peroxide detoxification pathway identify tryparedoxin as a suitable drug target? | \n", "These quantitative kinetic and metabolic analyses pointed out to TXN as a convenient drug target due to its low catalytic efficiency, high control on the flux of peroxide detoxification and role as provider of reducing equivalents to the two main peroxidases in the parasite. | \n", "**Yes.**\\n\\nMetabolic control analysis of the Trypanosoma cruzi peroxide detoxification pathway identified tryparedoxin as a suitable drug target, with 90-100% of the pathway flux controlled by TXN-TXNPx and TXN-GPxA redox pairs. | \n", "5 | \n", "5 | \n", "4 | \n", "
| 2 | \n", "Does promoter variant rs2301228 on the neural cell adhesion molecule 1 gene confer risk of schizophrenia in Han Chinese? | \n", "Our results provide direct evidence for NCAM1 as a susceptibility gene for schizophrenia, which offers support to a neurodevelopmental model and neuronal connectivity hypothesis in the onset of schizophrenia. | \n", "**Yes.**\\n\\nThe study found that the promoter variant rs2301228 on the NCAM1 gene is associated with schizophrenia in Han Chinese. | \n", "3 | \n", "5 | \n", "1 | \n", "
| 3 | \n", "Does pancreatic polypeptide regulate glucagon release through PPYR1 receptors expressed in mouse and human alpha-cells? | \n", "Glucose stimulates PP secretion and PP inhibits glucagon release in mouse pancreatic islets. PP receptors are present in alpha-cells of mouse and human pancreatic islets. | \n", "**Yes.**\\n\\nThe context information indicates that PPYR1 receptors are expressed in the alpha-cell population of mouse and human islets, and that PP release increases upon glucose stimulation, leading to inhibition of glucagon release. | \n", "5 | \n", "4 | \n", "5 | \n", "
| 4 | \n", "Does tetraploid complementation prove pluripotency of induced pluripotent stem cells derived from adipose tissue? | \n", "We also directed differentiation of iPS cells into chondrocytes, thus adipose-derived iPS cells can be used as models to study chondrogenic differentiation and cartilage regeneration. | \n", "**Yes.**\\n\\nThe study shows that tetraploid complementation successfully generated viable and fertile adult mice derived entirely from reprogrammed ASC, indicating that cell types other than fibroblasts can also be restored to the embryonic level of pluripotency. | \n", "4 | \n", "5 | \n", "5 | \n", "
| 5 | \n", "Is osteoprotegerin associated with subclinical left ventricular systolic dysfunction in diabetic hypertensive patients : a speckle tracking study? | \n", "Plasma OPG values could predict subclinical LV systolic dysfunction in diabetic hypertensive patients. | \n", "**Yes**, osteoprotegerin (OPG) is associated with subclinical left ventricular systolic dysfunction in diabetic hypertensive patients.\\n\\n**Results:**\\n\\n* Diabetic hypertensive patients had higher diastolic peak early/early diastolic tissue velocity and lower systolic tissue velocity, GLS, GLS rate systolic, and GLS rate early diastolic than nondiabetic hypertensive patients.\\n* OPG was found to be an independent predictor of impaired GLS.\\n* Receiver operating characteristic curve analysis revealed that OPG values of > 6.45 (pmol/L) identified the patients with GLS ≤ 18.5. | \n", "5 | \n", "5 | \n", "5 | \n", "
| 6 | \n", "Is cD30 expression a novel prognostic indicator in extranodal natural killer/T-cell lymphoma , nasal type? | \n", "Our results showed that expression of CD30 was not related to response to treatment but was an independent prognostic factor for both OS and PFS in ENKTL, nasal type, which suggests a role for CD30 in the pathogenesis of this disease and may support the incorporation of anti-CD30-targeted therapy into the treatment paradigm for ENKTL. | \n", "**Yes**, CD30 expression is a novel prognostic indicator in extranodal natural killer/T-cell lymphoma, nasal type.\\n\\nThe study found that CD30 positivity was associated with shorter 5-year OS and PFS in patients with extranodal natural killer/T-cell lymphoma, nasal type. | \n", "5 | \n", "5 | \n", "4 | \n", "
| 7 | \n", "Does mild cognitive dysfunction affect diabetes mellitus control in minority elderly adults? | \n", "Cognitive dysfunction in the mild range did not seem to affect diabetes mellitus control parameters in this multiethnic cohort of older adults with diabetes mellitus, although it cannot be excluded that cognitive impairment was overcome through assistance from formal or informal caregivers. It is possible that more-severe cognitive dysfunction could affect control. | \n", "The provided text does not contain any information regarding whether mild cognitive dysfunction affects diabetes mellitus control in minority elderly adults, so I am unable to answer this question from the provided context. | \n", "1 | \n", "1 | \n", "1 | \n", "
| 8 | \n", "Do youth walking and biking rates vary by environments around 5 Louisiana schools? | \n", "There is a role and a need for the SRTS program. The environment and other factors influence biking and walking to school. | \n", "The provided text indicates that there is a significant difference in the walking and biking rates between the 5 Louisiana schools. More students walked than biked to school, and the sites with the highest PEDS score had the highest percentage of students who walked/biked to school. | \n", "3 | \n", "5 | \n", "4 | \n", "
| 9 | \n", "Are human enteroviruses the cause of neurological impairments in children at the Korle-Bu Teaching Hospital? | \n", "Non-polio enteroviruses and chikugunya virus were found not to be the etiologic agent responsible for the convulsion with neurologic morbidity observed in the Ghanaian children. Investigation for other viral agents is recommended. | \n", "The provided text suggests that enteroviruses were detected in cerebrospinal fluid (CSF) samples from children at the Korle-Bu Teaching Hospital, but further studies are needed to establish a causal relationship between enteroviruses and neurological impairments. | \n", "4 | \n", "5 | \n", "4 | \n", "