Update README.md

README.md

@@ -162,16 +162,9 @@ oneshot(
 
 ## Evaluation
 
-The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command.
-A modified version of ARC-C and GSM8k-cot was used for evaluations, in line with Llama 3.1's prompting. It can be accessed on the [Neural Magic fork of the lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct).
-Additional evaluations that were collected for the original Llama 3.1 models will be added in the future.
-```
-lm_eval \
-  --model vllm \
-  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,tensor_parallel_size=8,gpu_memory_utilization=0.755,add_bos_token=True,max_model_len=4096 \
-  --tasks openllm \
-  --batch_size auto
-```
+The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA.
+Evaluation was conducted using the Neural Magic fork of [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct) (branch llama_3.1_instruct) and the [vLLM](https://docs.vllm.ai/en/stable/) engine.
+This version of the lm-evaluation-harness includes versions of ARC-Challenge and GSM-8K that match the prompting style of [Meta-Llama-3.1-Instruct-evals](https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals).
 
 ### Accuracy
 
@@ -257,4 +250,71 @@ lm_eval \
    <td><strong>99.74%</strong>
    </td>
   </tr>
-</table>
+</table>
+
+
+### Reproduction
+
+The results were obtained using the following commands:
+
+#### MMLU
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks mmlu \
+  --num_fewshot 5 \
+  --batch_size auto
+```
+
+#### ARC-Challenge
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks arc_challenge_llama_3.1_instruct \
+  --apply_chat_template \
+  --num_fewshot 0 \
+  --batch_size auto
+```
+
+#### GSM-8K
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks gsm8k_cot_llama_3.1_instruct \
+  --apply_chat_template \
+  --num_fewshot 8 \
+  --batch_size auto
+```
+
+#### Hellaswag
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks hellaswag \
+  --num_fewshot 10 \
+  --batch_size auto
+```
+
+#### Winogrande
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks winogrande \
+  --num_fewshot 5 \
+  --batch_size auto
+```
+
+#### Hellaswag
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
+  --tasks truthfulqa_mc \
+  --num_fewshot 0 \
+  --batch_size auto
+```