Upload README.md with huggingface_hub

README.md

@@ -1,7 +1,5 @@
 ---
-language:
-- en
-license: other
+base_model: Writer/Palmyra-x-004
 tags:
 - instruct
 - finetune
@@ -14,291 +12,162 @@ tags:
 - lifescience
 - Pharmaceutical
 - Pharma
-base_model: Writer/Palmyra-x-004
+model-index:
+- name: Palmyra-Med-70B-32k
+  results: []
+license: other
 license_name: writer-open-model-license
 license_link: https://writer.com/legal/open-model-license/
+language:
+- en
 widget:
-- example_title: Palmyra-Med-70B
+- example_title: Palmyra-Med-70B-32k
   messages:
   - role: system
-    content: You are a highly knowledgeable and experienced expert in the healthcare
-      and biomedical field, possessing extensive medical knowledge and practical expertise.
+    content: >-
+      You are a highly knowledgeable and experienced expert in the healthcare and biomedical field, possessing extensive medical knowledge and practical expertise.
   - role: user
-    content: Does danzhi Xiaoyao San ameliorate depressive-like behavior by shifting
-      toward serotonin via the downregulation of hippocampal indoleamine 2,3-dioxygenase?
+    content: Does danzhi Xiaoyao San ameliorate depressive-like behavior by shifting toward serotonin via the downregulation of hippocampal indoleamine 2,3-dioxygenase?
   output:
-    text: Danzhi Xiaoyao San (DXS) exhibited significant antidepressant-like effects
-      in rats subjected to chronic unpredictable mild stress (CUMS), a commonly used
-      model for inducing depressive-like behavior. The administration of DXS led to
-      a notable reduction in the activity of indoleamine 2,3-dioxygenase (IDO) in
-      the hippocampus. This reduction in IDO activity is crucial because IDO is an
-      enzyme that catalyzes the degradation of tryptophan into kynurenine (Kyn), thereby
-      diverting tryptophan away from the serotonin (5-HT) synthesis pathway. By downregulating
-      IDO, DXS effectively shifts the balance of tryptophan metabolism towards increased
-      serotonin production, thereby enhancing the availability of serotonin. This
-      shift in the Kyn/5-HT pathway favoring serotonin is likely a key mechanism underlying
-      the antidepressant effects observed with DXS treatment. Enhanced serotonin levels
-      in the hippocampus are associated with improved mood and reduction of depressive
-      symptoms, suggesting that DXS could be a promising therapeutic agent for managing
-      depression.
-model-index:
-- name: Palmyra-Med-70B
-  results: []
+    text: >-
+      Danzhi Xiaoyao San (DXS) exhibited significant antidepressant-like effects in rats subjected to chronic unpredictable mild stress (CUMS), a commonly used model for inducing depressive-like behavior. The administration of DXS led to a notable reduction in the activity of indoleamine 2,3-dioxygenase (IDO) in the hippocampus. This reduction in IDO activity is crucial because IDO is an enzyme that catalyzes the degradation of tryptophan into kynurenine (Kyn), thereby diverting tryptophan away from the serotonin (5-HT) synthesis pathway. By downregulating IDO, DXS effectively shifts the balance of tryptophan metabolism towards increased serotonin production, thereby enhancing the availability of serotonin. This shift in the Kyn/5-HT pathway favoring serotonin is likely a key mechanism underlying the antidepressant effects observed with DXS treatment. Enhanced serotonin levels in the hippocampus are associated with improved mood and reduction of depressive symptoms, suggesting that DXS could be a promising therapeutic agent for managing depression.
 ---
 
 
-
 <div align="center">
-  
   <h1>Palmyra-Med, a powerful LLM designed for healthcare</h1>
 </div>
 
 ## Model Details
 
-Palmyra-Med is a model built by Writer specifically to meet the needs of the healthcare industry. The leading LLM on biomedical benchmarks, with an average score of 85.87%, outperforming GPT-4, claude Opus, Gemini and Med-PaLM-2 base model and a medically trained human test-taker.
+
+Palmyra-Med-70b-32k, created by Writer, builds upon the foundation of Palmyra-Med-70b and offers an extended context length and meets the needs of the healthcare industry. The leading LLM on biomedical benchmarks, with an average score of 85.87%, outperforming GPT-4, claude Opus, Gemini and Med-PaLM-2 base model and a medically trained human test-taker.
 
 
 #### Specialized for Biomedical Applications
-Palmyra-Med-70B is meticulously designed to meet the unique linguistic and knowledge demands of the medical and life sciences sectors. It has been fine-tuned on an extensive collection of high-quality biomedical data, ensuring it can comprehend and generate text with precise domain-specific accuracy and fluency.
+Palmyra-Med-70B-32k is meticulously designed to meet the unique linguistic and knowledge demands of the medical and life sciences sectors. It has been fine-tuned on an extensive collection of high-quality biomedical data, ensuring it can comprehend and generate text with precise domain-specific accuracy and fluency.
 
+Our system integrates the DPO dataset and a well-crafted fine-tuning recipe along with a custom diverse medical instruction dataset, making it highly adept at handling the specific needs of this field. Key components of our training pipeline include:
 
-- **Policy Optimization**: [Direct Preference Optimization: Your Language Model is Secretly a Reward Model (DPO)](https://arxiv.org/abs/2305.18290)
+- **Policy Optimization**: Utilizing Direct Preference Optimization to enhance the model's performance. [DPO](https://arxiv.org/abs/2305.18290).
 - **Fine-tuning dataset**: Custom Medical Instruct dataset (Writer in-house build)
 
-
 ### Model Description
 
 - **Developed by:** Writer
 - **Model type:** Llama
 - **Language(s) (NLP):** English
-- **License:** Writer 
+- **License:** Writer License
 - **Finetuned from model:** Palmyra-X-004
+- **Context window:** 8192
 
 
 ## Intended Use
 
-**Intended Use Cases** Palmyra-X-Med is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
+**Intended Use Cases** Palmyra-Med-70B-32k is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
 
 **Out-of-scope** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Writer Open source License. Use in languages other than English**.
 
-**Note: Developers may fine-tune Palmyra-X-Med models for languages beyond English provided they comply with the Writer Open source License and the Acceptable Use Policy.
+**Note:** Developers may fine-tune Palmyra-Med-70B-32k models for languages beyond English provided they comply with the Writer Open source License and the Acceptable Use Policy.
 
 
 ### Use with transformers
 
-You can run conversational inference using the Transformers pipeline abstraction, or by leveraging the Auto classes with the `generate()` function. Let's see examples of both.
+You can run conversational inference using the Transformers Auto classes with the `generate()` function. Let's see an example.
 
 
-```
-import transformers
+```py
 import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "Writer/Palmyra-Med-70B-32k"
 
-model_id = "Writer/Palmyra-Med-70B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-pipeline = transformers.pipeline(
-    "text-generation",
-    model=model_id,
-    model_kwargs={"torch_dtype": torch.bfloat16},
-    device="auto",
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+    attn_implementation="flash_attention_2",
 )
 
 messages = [
-    {"role": "system", "content": "You are a highly knowledgeable and experienced expert in the healthcare and biomedical field, possessing extensive medical knowledge and practical expertise."},
-    {"role": "user", "content": "Does danzhi Xiaoyao San ameliorate depressive-like behavior by shifting toward serotonin via the downregulation of hippocampal indoleamine 2,3-dioxygenase?"},
+    {
+        "role": "system",
+        "content": "You are a highly knowledgeable and experienced expert in the healthcare and biomedical field, possessing extensive medical knowledge and practical expertise.",
+    },
+    {
+        "role": "user",
+        "content": "Does danzhi Xiaoyao San ameliorate depressive-like behavior by shifting toward serotonin via the downregulation of hippocampal indoleamine 2,3-dioxygenase?",
+    },
 ]
 
-prompt = pipeline.tokenizer.apply_chat_template(
-        messages, 
-        tokenize=False, 
-        add_generation_prompt=True
+input_ids = tokenizer.apply_chat_template(
+    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
 )
 
-terminators = [
-    pipeline.tokenizer.eos_token_id,
-    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
-]
+gen_conf = {
+    "max_new_tokens": 256,
+    "eos_token_id": [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|eot_id|>")],
+    "temperature": 0.0,
+    "top_p": 0.9,
+}
 
-outputs = pipeline(
-    prompt,
-    max_new_tokens=256,
-    eos_token_id=terminators,
-    do_sample=True,
-    temperature=0.0,
-    top_p=0.9,
-)
-print(outputs[0]["generated_text"][len(prompt):])
+with torch.inference_mode():
+    output_id = model.generate(input_ids, **gen_conf)
+
+output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :])
+
+print(output_text)
+```
 
 
+## Evaluation Results
 
+Palmyra-Med-70B-32k outperforms larger models like GPT-4, Gemini and Med-PaLM-1 across 9 diverse biomedical datasets, achieving state-of-the-art results with an average score of 85.9% despite having fewer parameters. Its strong performance in tasks like Clinical KG, Medical Genetics, and PubMedQA underscores its effective grasp of biomedical knowledge.
 
+
+### Performance on Biomedical Benchmarks
+
+![Palmyra-Med-70B-32k Performance](img/med_models_chart_32k.png)
+
+![Palmyra-Med-70B-32k Performance Heat Map](img/med_models_heatmap_32k.png)
+
+
+we ran the needle-in-haystack for Palmyra-Med-70B-32k and the results are as follows:
+
+![Palmyra-Med-70B-32k Performance NIH](img/nih_results_of_palmyra_med_70b_32k.png) 
+
+Following its evaluation on NIH, the Palmyra-Med-70B-32k model achieved almost perfect scores, highlighting its robust capability in efficiently processing extensive medical documents.
+
+
+### Medical Use Cases
+
+Palmyra-Med-70B-32k excels in analyzing and summarizing complex clinical notes, EHR data, and discharge summaries, extracting key information to generate concise, structured summaries. It can answer a wide range of medical questions and perform advanced clinical entity recognition, identifying key medical concepts such as diseases, symptoms, medications, procedures, and anatomical structures from unstructured text. 
+
+By leveraging its deep understanding of medical terminology, the model enhances information retrieval, data analysis, and knowledge discovery from EHRs, research articles, and other biomedical sources. These capabilities support applications like clinical decision support, pharmacovigilance, and medical research. 
+
+
+### Bias, Risks, and Limitations
+
+Palmyra-Med-70B-32k, despite leveraging high-quality data, may contain inaccuracies, biases, or misalignments and has not been rigorously evaluated in clinical trials or real-world healthcare settings. It is advised not to use the model for direct patient care, clinical decision support, or professional medical purposes. Instead, its use should be confined to research by qualified individuals who understand its limitations. Palmyra-Med-70B-32k should not replace professional medical judgment, and adapting it for medical use would require extensive additional work, including thorough testing, guideline alignment, bias mitigation, human oversight, and regulatory compliance. Always consult a qualified healthcare provider for personal medical needs.
+
+
+### Citation and Related Information
+
+To cite this model:
+
+```
+@misc{Palmyra-Med-70B,
+  author = {Writer Engineering team},
+  title = {{Palmyra-Med-70B: A powerful LLM designed for healthcare}},
+  howpublished = {\url{https://dev.writer.com}},
+  year = 2023,
+  month = March 
+}
 ```
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-
-## Evaluation
-
-
-<figure class="table">
-    <table class="ck-table-resized" style="border-collapse:collapse;font-family:Arial;font-size:10pt;table-layout:fixed;" cellspacing="0" cellpadding="0" dir="ltr" border="1" data-sheets-root="1">
-        <colgroup>
-            <col style="width:9.09%;" width="185">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="129">
-            <col style="width:9.09%;" width="145">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.09%;" width="100">
-            <col style="width:9.1%;" width="100">
-        </colgroup>
-        <tbody>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;overflow:hidden;padding:2px 3px;vertical-align:bottom;">&nbsp;</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Clinical KG&quot;}">Clinical KG</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Medical Genetics&quot;}">Medical Genetics</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Anatomy&quot;}">Anatomy</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Pro Medicine&quot;}">Pro Medicine</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;College Biology&quot;}">College Biology</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;College Medicine&quot;}">College Medicine</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;MedQA 4 opts&quot;}">MedQA 4 opts</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;PubMedQA&quot;}">PubMedQA</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;MedMCQA&quot;}">MedMCQA</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:center;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Avg&quot;}">Avg</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Palmyra-Med-70B&quot;}"><strong>Palmyra-Med-70B</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:90.94}"><strong>90.94</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:94}"><strong>94</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:83.7}"><strong>83.7</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:92.65}">92.65</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:94.44}">94.44</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:84.39}"><strong>84.39</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:78.63}">78.63</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:79.6}"><strong>79.6</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:74.44}"><strong>74.44</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:85.87}"><strong>85.87</strong></td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Med-PaLM-2 (5-shot)&quot;}">Med-PaLM-2 (5-shot)</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:88.3}">88.3</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:90}">90</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:77.8}">77.8</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:95.2}"><strong>95.2</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:94.4}">94.4</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:80.9}">80.9</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:79.7}">79.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:79.2}">79.2</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:71.3}">71.3</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:84.08}">84.08</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;GPT-4&quot;}">GPT-4</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:86.04}">86.04</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:91}">91</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:80}">80</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:93.01}">93.01</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:95.14}"><strong>95.14</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:76.88}">76.88</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:78.87}"><strong>78.87</strong></td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:75.2}">75.2</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:69.52}">69.52</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:82.85}">82.85</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Med-PaLM-1 (Flan-PaLM, 5-shot)&quot;}">Med-PaLM-1 (Flan-PaLM, 5-shot)</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:80.4}">80.4</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:75}">75</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:63.7}">63.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:83.8}">83.8</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:88.9}">88.9</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:76.3}">76.3</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:67.6}">67.6</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:79}">79</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:57.6}">57.6</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:74.7}">74.7</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Gemini&quot;}">Gemini</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:76.7}">76.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:75.8}">75.8</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:66.7}">66.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:77.7}">77.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:88}">88</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:69.2}">69.2</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:58}">58</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:70.7}">70.7</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:54.3}">54.3</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:70.79}">70.79</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;GPT-3.5 Turbo 1106&quot;}">GPT-3.5 Turbo 1106</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:74.71}">74.71</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:74}">74</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:72.79}">72.79</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:72.79}">72.79</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:72.91}">72.91</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:64.73}">64.73</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:57.71}">57.71</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:72.66}">72.66</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:53.79}">53.79</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:66}">66</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Meditron-70B (Llama)&quot;}">Meditron-70B (Llama)</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:66.79}">66.79</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:69}">69</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:53.33}">53.33</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:71.69}">71.69</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:76.38}">76.38</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:63}">63</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:57.1}">57.1</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:76.6}">76.6</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:46.85}">46.85</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:64.52}">64.52</td>
-            </tr>
-            <tr style="height:21px;">
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Mistral-7B-v0.1&quot;}">Mistral-7B-v0.1</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:68.68}">68.68</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:71}">71</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:55.56}">55.56</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:68.38}">68.38</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:68.06}">68.06</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:59.54}">59.54</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:50.82}">50.82</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:75.4}">75.4</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:48.2}">48.2</td>
-                <td style="background-color:#ffffff;font-family:Source Sans Pro;font-weight:normal;overflow:hidden;padding:2px 3px;text-align:right;vertical-align:bottom;" data-sheets-value="{&quot;1&quot;:3,&quot;3&quot;:62.85}">62.85</td>
-            </tr>
-        </tbody>
-    </table>
-</figure>`
-
-
-
-### Results
-
-[More Information Needed]
+
+Contact
+Hello@writer.com
+
+