Update README.md

README.md

@@ -1,3 +1,248 @@
----
-license: llama3.1
----
+---
+library_name: transformers  # Specify the library
+datasets:
+- HiTZ/latxa-corpus-v1.1
+language:
+- eu
+- en
+metrics:
+- accuracy
+pipeline_tag: text-generation
+model-index:
+- name: Latxa-Llama-3.1-70B-Instruct
+  results:
+  - task:
+      type: multiple-choice
+    dataset:
+      name: xstory_cloze
+      type: XStory
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 77.83
+  - task:
+      type: multiple-choice
+    dataset:
+      name: belebele
+      type: Belebele
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 91.00
+  - task:
+      type: multiple_choice
+    dataset:
+      name: eus_proficiency
+      type: EusProficiency
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 68.00
+  - task:
+      type: multiple_choice
+    dataset:
+      name: eus_reading
+      type: EusReading
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 78.98
+  - task:
+      type: multiple_choice
+    dataset:
+      name: eus_trivia
+      type: EusTrivia
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 74.17
+  - task:
+      type: multiple_choice
+    dataset:
+      name: eus_exams
+      type: EusExams
+    metrics:
+    - name: Accuracy (5-shot)
+      type: Accuracy (5-shot)
+      value: 71.56
+
+license: llama3.1
+base_model:
+- meta-llama/Llama-3.1-70B-Instruct
+
+co2_eq_emissions:
+  emissions: 1900.8
+  source: "CodeCarbon"
+  training_type: "pre-training"
+  geographical_location: "EU-West"
+  hardware_used: "256xA100 GPUs"
+---
+
+# Model Card for HiTZ/Latxa-Llama-3.1-70B-Instruct
+
+<p align="center">
+  <img src="https://github.com/hitz-zentroa/latxa/blob/b9aa705f60ee2cc03c9ed62fda82a685abb31b07/assets/latxa_round.png?raw=true" style="height: 350px;">
+</p>
+
+We introduce Latxa 3.1 70B Instruct, an instructed version of [Latxa](https://aclanthology.org/2024.acl-long.799/). This new Latxa is based on Llama-3.1 (Instruct), which we trained on our Basque corpus (Etxaniz et al., 2024) comprising 4.3M documents and 4.2B tokens using language adaptation techniques (paper in preparation). 
+> [!WARNING]
+> DISCLAIMER
+> 
+> This model is still under development.
+> Further training details will be released with the corresponding research paper in the near future.
+
+
+Our preliminary experimentation shows that Latxa 3.1 70B Instruct outperforms Llama-3.1-Instruct by a large margin on Basque standard benchmarks, and particularly, on chat conversations. In addition, we organized a public arena-based evaluation, on which Latxat competed against other baselines and proprietary models such as GPT-4o and Claude Sonnet. The results showed that Latxa ranked 3rd, just behind Claude and GPT-4 and above all the other same-size competitors.
+The official paper is coming soon.
+
+
+## Model Details
+
+### Model Description
+
+Latxa is a family of Large Language Models (LLM) based on Meta’s LLaMA models. Current LLMs exhibit incredible performance 
+for high-resource languages such as English, but, in the case of Basque and other low-resource languages, their performance 
+is close to a random guesser. These limitations widen the gap between high- and low-resource languages when it comes to 
+digital development. We present Latxa to overcome these limitations and promote the development of LLM-based technology and 
+research for the Basque language. Latxa models follow the same architecture as their original counterparts and were further 
+trained in [Latxa Corpus v1.1](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1), a high-quality Basque corpora. 
+
+- **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
+- **Model type:** Language model
+- **Language(s) (NLP):** eu
+- **License:** llama3.1
+- **Parent model:** meta-llama/Llama-3.1-70B-Instruct
+- **Contact:** [email protected]
+
+
+### Getting Started
+Use the code below to get started with the model.
+
+```python
+from transformers import pipeline
+
+pipe = pipeline('text-generation', model='HiTZ/Latxa-Llama-3.1-70B-Instruct')
+
+messages = [
+	{'role': 'user', 'content': 'Kaixo!'},
+]
+
+pipe(messages)
+
+>>
+[
+  {
+    'generated_text': [
+      {'role': 'user', 'content': 'Kaixo!'},
+      {'role': 'assistant', 'content': 'Kaixo! Zer moduz? Zer behar edo galdetu nahi duzu?'}
+    ]
+  }
+]
+```
+
+## Uses
+
+Latxa models are intended to be used with Basque data; for any other language the performance is not guaranteed. 
+Same as the original, Latxa inherits the [Llama-3.1 License](https://www.llama.com/llama3_1/license/) which allows for commercial and research use. 
+
+### Direct Use
+
+Latxa Instruct models are trained to follow instructions or to work as chat assistants. 
+
+### Out-of-Scope Use
+
+The model is not intended for malicious activities, such as harming others or violating human rights. Any downstream application must comply with current laws and regulations. 
+Irresponsible usage in production environments without proper risk assessment and mitigation is also discouraged. 
+
+## Bias, Risks, and Limitations
+
+In an effort to alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed 
+data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs (see [Latxa Corpus v1.1](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1)). Still, the 
+model is based on Llama 3.1 models and can potentially carry the same bias, risk and limitations.
+Please see the Llama’s Ethical Considerations and Limitations for further information.
+
+
+## Training Details
+
+> [!WARNING]
+> DISCLAIMER
+> 
+> Further training details will be released with the corresponding research paper in the near future.
+
+
+## Evaluation
+
+We evaluated the models 5-shot settings on multiple-choice tasks. We used the basque partitions of each dataset.
+
+The arena results will be released in the future.
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+- **Belebele** (Bandarkar et al.): Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. We evaluated the model in a 5-shot fashion.
+  - Data card: https://huggingface.co/datasets/facebook/belebele
+- **X-StoryCloze** (Lin et al.): XStoryCloze consists of the professionally translated version of the English StoryCloze dataset to 10 non-English languages. Story Cloze is a commonsense reasoning dataset which consists of choosing the correct ending to a four-sentence story. We evaluated the model in a 5-shot fashion.
+  - Data card: https://huggingface.co/datasets/juletxara/xstory_cloze
+- **EusProficiency** (Etxaniz et al., 2024): EusProficiency comprises 5,169 exercises on different topics from past EGA exams, the official C1-level certificate of proficiency in Basque.
+  - Data card: https://huggingface.co/datasets/HiTZ/EusProficiency
+- **EusReading** (Etxaniz et al., 2024): EusReading consists of 352 reading comprehension exercises (irakurmena) sourced from the same set of past EGA exams.
+  - Data card: https://huggingface.co/datasets/HiTZ/EusReading
+- **EusTrivia** (Etxaniz et al., 2024): EusTrivia consists of 1,715 trivia questions from multiple online sources. 56.3% of the questions are elementary level (grades 3-6), while the rest are considered challenging.
+  - Data card: https://huggingface.co/datasets/HiTZ/EusTrivia
+- **EusExams** (Etxaniz et al., 2024): EusExams is a collection of tests designed to prepare individuals for Public Service examinations conducted by several Basque institutions, including the public health system Osakidetza, the Basque Government, the City Councils of Bilbao and Gasteiz, and the University of the Basque Country (UPV/EHU).
+  - Data card: https://huggingface.co/datasets/HiTZ/EusExams
+
+#### Metrics
+
+We use Accuracy, as they are framed as Multiple Choice questions.
+
+### Results
+
+| Task | Llama-3.1 8B Instruct | Latxa 3.1 8B Instruct | Llama-3.1 70B Instruct | Latxa 3.1 70B Instruct |
+| :---- | :---: | :---: | :---: | :---: |
+| Belebele | 73.89 | 80.00 | 89.11 | 91.00
+| X-Story Cloze | 61.22 | 71.34 | 69.69 | 77.83 |
+| EusProficiency | 34.13 | 52.83 | 43.59 | 68.00 |
+| EusReading | 49.72 | 62.78 | 72.16 | 78.98 |
+| EusTrivia | 45.01 | 61.05 | 62.51 | 74.17 |
+| EusExams | 46.21 | 56.00 | 63.28 | 71.56 |
+
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+<!-- 62.50h x 256 GPU = 16.000h -->
+
+- **Hardware Type:** HPC Cluster, 4 x A100 64Gb nodes x64
+- **Hours used (total GPU hours):** 16000h
+- **Cloud Provider:** CINECA HPC
+- **Compute Region:**  Italy
+- **Carbon Emitted:** 1900.8kg CO2 eq
+
+## Acknowledgements
+
+This work has been partially supported by the Basque Government (IKER-GAITU project).
+
+It has also been partially supported by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project with reference 2022/TL22/00215335.
+
+The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013.
+
+
+## Citation
+
+Coming soon.
+
+Meanwhile, you can reference:
+```bibtex
+@misc{etxaniz2024latxa,
+  	title={{L}atxa: An Open Language Model and Evaluation Suite for {B}asque},
+  	author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
+  	year={2024},
+  	eprint={2403.20266},
+  	archivePrefix={arXiv},
+  	primaryClass={cs.CL}
+}