Instructions to use keepsloading/icml_repro_scratch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keepsloading/icml_repro_scratch with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keepsloading/icml_repro_scratch")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("keepsloading/icml_repro_scratch")
model = AutoModelForCausalLM.from_pretrained("keepsloading/icml_repro_scratch", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keepsloading/icml_repro_scratch with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keepsloading/icml_repro_scratch"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keepsloading/icml_repro_scratch

SGLang

How to use keepsloading/icml_repro_scratch with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keepsloading/icml_repro_scratch" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keepsloading/icml_repro_scratch" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keepsloading/icml_repro_scratch with Docker Model Runner:
```
docker model run hf.co/keepsloading/icml_repro_scratch
```

keepsloading commited on 9 days ago

Commit

7f8c049

verified ·

1 Parent(s): e35ef17

Upload folder using huggingface_hub

Browse files

Files changed (36) hide show

.gitattributes +3 -0
outputs/eval/logs/vanilla/babilong.log +460 -0
outputs/eval/logs/vanilla/helmet.log +502 -0
outputs/eval/logs/vanilla/mrcr.log +62 -0
outputs/eval/logs/vanilla/ruler_a.log +433 -183
outputs/eval/logs/vanilla/ruler_b.log +369 -0
outputs/eval/vanilla/babilong.DONE +0 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/results_2026-07-18T14-41-29.909583.json +505 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa1_2026-07-18T14-41-29.909583.jsonl +3 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa2_2026-07-18T14-41-29.909583.jsonl +3 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa3_2026-07-18T14-41-29.909583.jsonl +3 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa4_2026-07-18T14-41-29.909583.jsonl +3 -0
outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa5_2026-07-18T14-41-29.909583.jsonl +3 -0
outputs/eval/vanilla/helmet.DONE +0 -0
outputs/eval/vanilla/helmet_icl8k_n50.jsonl +3 -0
outputs/eval/vanilla/helmet_icl8k_n50.summary.csv +251 -0
outputs/eval/vanilla/mrcr.DONE +0 -0
outputs/eval/vanilla/mrcr_8k_2_4_8needle_n10.jsonl +3 -0
outputs/eval/vanilla/mrcr_8k_2_4_8needle_n10.summary.csv +31 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/results_2026-07-18T14-27-50.984361.json +821 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_multikey_2_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_multiquery_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_single_1_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_single_3_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_fwe_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_qa_hotpot_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_vt_2026-07-18T14-27-50.984361.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/results_2026-07-18T14-38-04.340503.json +714 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multikey_1_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multikey_3_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multivalue_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_single_2_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_ruler_cwe_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_ruler_qa_squad_2026-07-18T14-38-04.340503.jsonl +3 -0
outputs/eval/vanilla/ruler_a.DONE +0 -0
outputs/eval/vanilla/ruler_b.DONE +0 -0

.gitattributes CHANGED Viewed

@@ -2,3 +2,6 @@
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.jsonl filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.jsonl filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-25/optimizer.pt filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-50/optimizer.pt filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-75/optimizer.pt filter=lfs diff=lfs merge=lfs -text

outputs/eval/logs/vanilla/babilong.log ADDED Viewed

@@ -0,0 +1,460 @@
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A

+2026-07-18:14:38:06 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:38:10 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:14:38:11 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:38:11 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:38:15 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:38:15 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:38:17 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:38:20 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:38:20 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:14:38:20 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:14:38:20 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:38:20 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:38:21 INFO     [evaluator:584] Running generate_until requests
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+2026-07-18:14:41:29 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:41:29 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/*.jsonl
+hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
+|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|-----:|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  | 0.60|±  |0.0295|
+| - babilong_qa1 |      0|none  |     2|acc   |↑  | 0.68|±  |0.0666|
+| - babilong_qa2 |      0|none  |     2|acc   |↑  | 0.48|±  |0.0714|
+| - babilong_qa3 |      0|none  |     2|acc   |↑  | 0.34|±  |0.0677|
+| - babilong_qa4 |      0|none  |     2|acc   |↑  | 0.72|±  |0.0641|
+| - babilong_qa5 |      0|none  |     2|acc   |↑  | 0.78|±  |0.0592|
+|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|------|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |  0.6|±  |0.0295|

outputs/eval/logs/vanilla/helmet.log ADDED Viewed

	@@ -0,0 +1,502 @@

+[eval] stop_token_ids=[151643, 151645]
+[helmet_icl] 1/250 trec_coarse tokens=6597
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 2/250 trec_coarse tokens=6549
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 3/250 trec_coarse tokens=6508
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 4/250 trec_coarse tokens=6608
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 5/250 trec_coarse tokens=6497
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 6/250 trec_coarse tokens=6545
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 7/250 trec_coarse tokens=6583
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 8/250 trec_coarse tokens=6455
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 9/250 trec_coarse tokens=6643
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 10/250 trec_coarse tokens=6568
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 11/250 trec_coarse tokens=6518
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 12/250 trec_coarse tokens=6424
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 13/250 trec_coarse tokens=6750
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 14/250 trec_coarse tokens=6642
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 15/250 trec_coarse tokens=6494
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 16/250 trec_coarse tokens=6427
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 17/250 trec_coarse tokens=6547
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 18/250 trec_coarse tokens=6612
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 19/250 trec_coarse tokens=6591
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 20/250 trec_coarse tokens=6485
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 21/250 trec_coarse tokens=6542
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 22/250 trec_coarse tokens=6409
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 23/250 trec_coarse tokens=6482
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 24/250 trec_coarse tokens=6502
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 25/250 trec_coarse tokens=6344
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 26/250 trec_coarse tokens=6479
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 27/250 trec_coarse tokens=6569
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 28/250 trec_coarse tokens=6389
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 29/250 trec_coarse tokens=6479
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 30/250 trec_coarse tokens=6653
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 31/250 trec_coarse tokens=6532
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 32/250 trec_coarse tokens=6491
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 33/250 trec_coarse tokens=6489
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 34/250 trec_coarse tokens=6630
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 35/250 trec_coarse tokens=6488
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 36/250 trec_coarse tokens=6499
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 37/250 trec_coarse tokens=6483
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 38/250 trec_coarse tokens=6660
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 39/250 trec_coarse tokens=6675
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 40/250 trec_coarse tokens=6544
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 41/250 trec_coarse tokens=6520
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 42/250 trec_coarse tokens=6359
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 43/250 trec_coarse tokens=6657
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 44/250 trec_coarse tokens=6546
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 45/250 trec_coarse tokens=6481
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 46/250 trec_coarse tokens=6506
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 47/250 trec_coarse tokens=6621
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 48/250 trec_coarse tokens=6623
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 49/250 trec_coarse tokens=6411
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 50/250 trec_coarse tokens=6577
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 51/250 trec_fine tokens=6858
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 52/250 trec_fine tokens=6973
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 53/250 trec_fine tokens=6977
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 54/250 trec_fine tokens=6952
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 55/250 trec_fine tokens=6919
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 56/250 trec_fine tokens=6979
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 57/250 trec_fine tokens=6957
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 58/250 trec_fine tokens=7013
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 59/250 trec_fine tokens=6824
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 60/250 trec_fine tokens=6833
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 61/250 trec_fine tokens=6868
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 62/250 trec_fine tokens=6914
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 63/250 trec_fine tokens=6759
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 64/250 trec_fine tokens=6933
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 65/250 trec_fine tokens=6868
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 66/250 trec_fine tokens=6839
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 67/250 trec_fine tokens=6838
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 68/250 trec_fine tokens=7000
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 69/250 trec_fine tokens=6800
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 70/250 trec_fine tokens=6829
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 71/250 trec_fine tokens=6971
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 72/250 trec_fine tokens=6907
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 73/250 trec_fine tokens=6818
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 74/250 trec_fine tokens=6902
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 75/250 trec_fine tokens=6941
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 76/250 trec_fine tokens=6830
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 77/250 trec_fine tokens=6934
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 78/250 trec_fine tokens=6879
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 79/250 trec_fine tokens=6915
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 80/250 trec_fine tokens=6953
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 81/250 trec_fine tokens=6875
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 82/250 trec_fine tokens=6847
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 83/250 trec_fine tokens=6847
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 84/250 trec_fine tokens=6944
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 85/250 trec_fine tokens=6818
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 86/250 trec_fine tokens=6868
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 87/250 trec_fine tokens=7022
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 88/250 trec_fine tokens=6930
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 89/250 trec_fine tokens=6863
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 90/250 trec_fine tokens=6807
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 91/250 trec_fine tokens=7001
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 92/250 trec_fine tokens=6890
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 93/250 trec_fine tokens=6952
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 94/250 trec_fine tokens=6845
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 95/250 trec_fine tokens=6902
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 96/250 trec_fine tokens=6879
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 97/250 trec_fine tokens=6896
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 98/250 trec_fine tokens=6770
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 99/250 trec_fine tokens=6933
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 100/250 trec_fine tokens=6886
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 101/250 banking77 tokens=6809
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 102/250 banking77 tokens=7162
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 103/250 banking77 tokens=6876
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 104/250 banking77 tokens=7033
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 105/250 banking77 tokens=6647
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 106/250 banking77 tokens=6818
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 107/250 banking77 tokens=6841
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 108/250 banking77 tokens=6944
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 109/250 banking77 tokens=6776
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 110/250 banking77 tokens=7100
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 111/250 banking77 tokens=6967
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 112/250 banking77 tokens=6894
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 113/250 banking77 tokens=6900
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 114/250 banking77 tokens=6964
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 115/250 banking77 tokens=7085
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 116/250 banking77 tokens=6953
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 117/250 banking77 tokens=7406
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 118/250 banking77 tokens=6802
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 119/250 banking77 tokens=7119
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 120/250 banking77 tokens=7202
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 121/250 banking77 tokens=7103
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 122/250 banking77 tokens=7009
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 123/250 banking77 tokens=6940
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 124/250 banking77 tokens=6857
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 125/250 banking77 tokens=6922
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 126/250 banking77 tokens=6988
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 127/250 banking77 tokens=6945
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 128/250 banking77 tokens=6835
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 129/250 banking77 tokens=7035
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 130/250 banking77 tokens=6787
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 131/250 banking77 tokens=6940
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 132/250 banking77 tokens=6661
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 133/250 banking77 tokens=6934
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 134/250 banking77 tokens=7235
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 135/250 banking77 tokens=6829
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 136/250 banking77 tokens=7169
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 137/250 banking77 tokens=7074
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 138/250 banking77 tokens=6797
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 139/250 banking77 tokens=7013
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 140/250 banking77 tokens=6811
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 141/250 banking77 tokens=6983
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 142/250 banking77 tokens=7183
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 143/250 banking77 tokens=6801
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 144/250 banking77 tokens=7174
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 145/250 banking77 tokens=6773
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 146/250 banking77 tokens=7134
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 147/250 banking77 tokens=6907
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 148/250 banking77 tokens=7341
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 149/250 banking77 tokens=7042
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 150/250 banking77 tokens=7304
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 151/250 clinic150 tokens=7241
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 152/250 clinic150 tokens=7282
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 153/250 clinic150 tokens=7261
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 154/250 clinic150 tokens=7292
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 155/250 clinic150 tokens=7217
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 156/250 clinic150 tokens=7280
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 157/250 clinic150 tokens=7293
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 158/250 clinic150 tokens=7250
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 159/250 clinic150 tokens=7264
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 160/250 clinic150 tokens=7238
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 161/250 clinic150 tokens=7243
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 162/250 clinic150 tokens=7191
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 163/250 clinic150 tokens=7333
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 164/250 clinic150 tokens=7299
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 165/250 clinic150 tokens=7225
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 166/250 clinic150 tokens=7409
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 167/250 clinic150 tokens=7277
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 168/250 clinic150 tokens=7376
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 169/250 clinic150 tokens=7383
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 170/250 clinic150 tokens=7270
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 171/250 clinic150 tokens=7173
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 172/250 clinic150 tokens=7106
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 173/250 clinic150 tokens=7413
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 174/250 clinic150 tokens=7253
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 175/250 clinic150 tokens=7224
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 176/250 clinic150 tokens=7307
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 177/250 clinic150 tokens=7272
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 178/250 clinic150 tokens=7215
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 179/250 clinic150 tokens=7344
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 180/250 clinic150 tokens=7224
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 181/250 clinic150 tokens=7310
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 182/250 clinic150 tokens=7222
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 183/250 clinic150 tokens=7260
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 184/250 clinic150 tokens=7351
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 185/250 clinic150 tokens=7301
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 186/250 clinic150 tokens=7155
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 187/250 clinic150 tokens=7253
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 188/250 clinic150 tokens=7248
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 189/250 clinic150 tokens=7128
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 190/250 clinic150 tokens=7282
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 191/250 clinic150 tokens=7301
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 192/250 clinic150 tokens=7388
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 193/250 clinic150 tokens=7244
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 194/250 clinic150 tokens=7186
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 195/250 clinic150 tokens=7309
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 196/250 clinic150 tokens=7241
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 197/250 clinic150 tokens=7214
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 198/250 clinic150 tokens=7264
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 199/250 clinic150 tokens=7219
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 200/250 clinic150 tokens=7228
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 201/250 nlu tokens=7063
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 202/250 nlu tokens=6959
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 203/250 nlu tokens=6985
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 204/250 nlu tokens=6970
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 205/250 nlu tokens=7025
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 206/250 nlu tokens=7034
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 207/250 nlu tokens=7004
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 208/250 nlu tokens=7062
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 209/250 nlu tokens=7062
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 210/250 nlu tokens=7045
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 211/250 nlu tokens=7028
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 212/250 nlu tokens=7018
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 213/250 nlu tokens=7075
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 214/250 nlu tokens=7068
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 215/250 nlu tokens=7025
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 216/250 nlu tokens=7022
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 217/250 nlu tokens=7112
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 218/250 nlu tokens=6967
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 219/250 nlu tokens=6980
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 220/250 nlu tokens=7031
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 221/250 nlu tokens=7093
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 222/250 nlu tokens=6949
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 223/250 nlu tokens=7046
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 224/250 nlu tokens=7039
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 225/250 nlu tokens=7126
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 226/250 nlu tokens=7064
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 227/250 nlu tokens=7057
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 228/250 nlu tokens=6971
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 229/250 nlu tokens=7034
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 230/250 nlu tokens=7094
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 231/250 nlu tokens=6979
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 232/250 nlu tokens=7123
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 233/250 nlu tokens=7144
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 234/250 nlu tokens=7024
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 235/250 nlu tokens=6943
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 236/250 nlu tokens=6961
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 237/250 nlu tokens=7005
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 238/250 nlu tokens=7080
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 239/250 nlu tokens=7179
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 240/250 nlu tokens=7115
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 241/250 nlu tokens=6946
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 242/250 nlu tokens=6985
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 243/250 nlu tokens=7131
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 244/250 nlu tokens=7100
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 245/250 nlu tokens=7190
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 246/250 nlu tokens=7040
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 247/250 nlu tokens=7013
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 248/250 nlu tokens=7080
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 249/250 nlu tokens=7098
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[helmet_icl] 250/250 nlu tokens=6891
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+/workspace/outputs/eval/vanilla/helmet_icl8k_n50.summary.csv

outputs/eval/logs/vanilla/mrcr.log ADDED Viewed

	@@ -0,0 +1,62 @@

+[eval] stop_token_ids=[151643, 151645]
+[mrcr] 1/30 8k_2needle tokens=4852
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 2/30 8k_2needle tokens=7119
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 3/30 8k_2needle tokens=5756
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 4/30 8k_2needle tokens=5072
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 5/30 8k_2needle tokens=5753
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 6/30 8k_2needle tokens=5481
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 7/30 8k_2needle tokens=5143
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 8/30 8k_2needle tokens=6399
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 9/30 8k_2needle tokens=5987
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 10/30 8k_2needle tokens=4380
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 11/30 8k_4needle tokens=5123
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 12/30 8k_4needle tokens=7908
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 13/30 8k_4needle tokens=5814
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 14/30 8k_4needle tokens=4856
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 15/30 8k_4needle tokens=7413
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 16/30 8k_4needle tokens=7764
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 17/30 8k_4needle tokens=4312
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 18/30 8k_4needle tokens=4847
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 19/30 8k_4needle tokens=6287
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 20/30 8k_4needle tokens=7841
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 21/30 8k_8needle tokens=4581
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 22/30 8k_8needle tokens=6357
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 23/30 8k_8needle tokens=6981
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 24/30 8k_8needle tokens=6308
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 25/30 8k_8needle tokens=6901
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 26/30 8k_8needle tokens=5407
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 27/30 8k_8needle tokens=6351
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 28/30 8k_8needle tokens=5657
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 29/30 8k_8needle tokens=7330
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mrcr] 30/30 8k_8needle tokens=8028
+The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+/workspace/outputs/eval/vanilla/mrcr_8k_2_4_8needle_n10.summary.csv

outputs/eval/logs/vanilla/ruler_a.log CHANGED Viewed

@@ -1,71 +1,72 @@
-2026-07-18:10:33:06 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
-2026-07-18:10:33:10 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
-2026-07-18:10:33:11 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-07-18:10:33:11 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
-2026-07-18:10:33:14 INFO     [models.huggingface:161] Using device 'cuda:0'
-2026-07-18:10:33:14 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
-2026-07-18:10:33:24 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:33:24 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
-2026-07-18:10:33:33 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:33:44 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:33:51 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:34:01 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:34:01 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
 Max length 500 | Current length 307 | Noises: 5
 Max length 500 | Current length 410 | Noises: 10
 Max length 500 | Current length 531 | Noises: 15
 Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/1 [00:00<?, ?it/s][A
 Max length 8192 | Current length 791 | Noises: 10
 Max length 8192 | Current length 1025 | Noises: 20
 Max length 8192 | Current length 1273 | Noises: 30
@@ -101,142 +102,391 @@ Max length 8192 | Current length 8230 | Noises: 320
 Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 60/500 [00:01<00:07, 59.22it/s][A
 24%|██▍       | 120/500 [00:02<00:06, 57.62it/s][A
 36%|███▌      | 178/500 [00:03<00:05, 57.45it/s][A
 47%|████▋     | 237/500 [00:04<00:04, 57.69it/s][A
 59%|█████▉    | 295/500 [00:05<00:03, 56.89it/s][A
 70%|███████   | 352/500 [00:06<00:02, 56.14it/s][A
 82%|████████▏ | 409/500 [00:07<00:01, 56.31it/s][A
 94%|█████████▎| 468/500 [00:08<00:00, 56.85it/s][A
-2026-07-18:10:34:10 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 12%|█▏        | 59/500 [00:01<00:07, 58.56it/s][A
 24%|██▎       | 118/500 [00:02<00:06, 57.94it/s][A
 35%|███▌      | 176/500 [00:03<00:05, 57.91it/s][A
 47%|████▋     | 234/500 [00:04<00:04, 56.14it/s][A
 58%|█████▊    | 291/500 [00:05<00:03, 55.52it/s][A
 69%|██████▉   | 347/500 [00:06<00:02, 54.63it/s][A
 80%|████████  | 402/500 [00:07<00:01, 54.48it/s][A
 91%|█████████▏| 457/500 [00:08<00:00, 54.48it/s][A
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:10:34:24 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-Traceback (most recent call last):
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
-    sock = connection.create_connection(
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
-    raise err
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
-    sock.connect(sa)
-TimeoutError: [Errno 110] Connection timed out
-The above exception was the direct cause of the following exception:
-Traceback (most recent call last):
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
-    response = self._make_request(
-               ^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 493, in _make_request
-    conn.request(
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 494, in request
-    self.endheaders()
-  File "/opt/conda/lib/python3.11/http/client.py", line 1298, in endheaders
-    self._send_output(message_body, encode_chunked=encode_chunked)
-  File "/opt/conda/lib/python3.11/http/client.py", line 1058, in _send_output
-    self.send(msg)
-  File "/opt/conda/lib/python3.11/http/client.py", line 996, in send
-    self.connect()
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 325, in connect
-    self.sock = self._new_conn()
-                ^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
-    raise ConnectTimeoutError(
-urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
-The above exception was the direct cause of the following exception:
-Traceback (most recent call last):
-  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 644, in send
-    resp = conn.urlopen(
-           ^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
-    retries = retries.increment(
-              ^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
-    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
-    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
-During handling of the above exception, another exception occurred:
-Traceback (most recent call last):
-  File "<frozen runpy>", line 198, in _run_module_as_main
-  File "<frozen runpy>", line 88, in _run_code
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/__main__.py", line 14, in <module>
-    cli_evaluate()
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/__main__.py", line 10, in cli_evaluate
-    parser.execute(args)
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/_cli/harness.py", line 60, in execute
-    args.func(args)
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/_cli/run.py", line 379, in _execute
-    results = simple_evaluate(
-              ^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/utils.py", line 498, in _wrapper
-    return fn(*args, **kwargs)
-           ^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/evaluator.py", line 292, in simple_evaluate
-    task_dict = get_task_dict(
-                ^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 646, in get_task_dict
-    task_name_from_string_dict = task_manager.load_task_or_group(
-                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 428, in load_task_or_group
-    collections.ChainMap(
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 430, in <lambda>
-    lambda task: self._load_individual_task_or_group(task),
-                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 328, in _load_individual_task_or_group
-    return _load_task(task_config, task=name_or_config)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 288, in _load_task
-    task_object = ConfigurableTask(config=config)
-                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/api/task.py", line 748, in __init__
-    self.download(self.config.dataset_kwargs)
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/api/task.py", line 860, in download
-    self.dataset = self.config.custom_dataset(
-                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 240, in get_hotpotqa
-    return get_qa_dataset("hotpotqa", **kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 222, in get_qa_dataset
-    qas, docs = read_hotpotqa()
-                ^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 79, in read_hotpotqa
-    data = download_json(url)
-           ^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 38, in download_json
-    response = requests.get(url)
-               ^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 73, in get
-    return request("get", url, params=params, **kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
-    return session.request(method=method, url=url, **kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
-    resp = self.send(prep, **send_kwargs)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
-    r = adapter.send(request, **kwargs)
-        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
-    raise ConnectTimeout(e, request=request)
-requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:14:17:11 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:17:15 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:14:17:16 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:17:16 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:17:20 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:17:20 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:17:29 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:17:29 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:14:17:39 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:17:50 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:17:58 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:18:08 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:18:08 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
 Max length 500 | Current length 307 | Noises: 5
 Max length 500 | Current length 410 | Noises: 10
 Max length 500 | Current length 531 | Noises: 15
 Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/1 [00:00<?, ?it/s][A
 Max length 8192 | Current length 791 | Noises: 10
 Max length 8192 | Current length 1025 | Noises: 20
 Max length 8192 | Current length 1273 | Noises: 30
 Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 60/500 [00:01<00:07, 59.22it/s][A
 24%|██▍       | 120/500 [00:02<00:06, 57.62it/s][A
 36%|███▌      | 178/500 [00:03<00:05, 57.45it/s][A
 47%|████▋     | 237/500 [00:04<00:04, 57.69it/s][A
 59%|█████▉    | 295/500 [00:05<00:03, 56.89it/s][A
 70%|███████   | 352/500 [00:06<00:02, 56.14it/s][A
 82%|████████▏ | 409/500 [00:07<00:01, 56.31it/s][A
 94%|█████████▎| 468/500 [00:08<00:00, 56.85it/s][A
 12%|█▏        | 59/500 [00:01<00:07, 58.56it/s][A
 24%|██▎       | 118/500 [00:02<00:06, 57.94it/s][A
 35%|███▌      | 176/500 [00:03<00:05, 57.91it/s][A
 47%|████▋     | 234/500 [00:04<00:04, 56.14it/s][A
 58%|█████▊    | 291/500 [00:05<00:03, 55.52it/s][A
 69%|██████▉   | 347/500 [00:06<00:02, 54.63it/s][A
 80%|████████  | 402/500 [00:07<00:01, 54.48it/s][A
 91%|█████████▏| 457/500 [00:08<00:00, 54.48it/s][A
+2026-07-18:14:18:18 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:18:32 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:18:55 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:14:18:55 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:18:56 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/*.jsonl
+hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multiquery|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.9875|±  |   N/A|
+|niah_single_1  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_3  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7667|±  |   N/A|
+|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4500|±  |   N/A|
+|ruler_vt       |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.8600|±  |   N/A|

outputs/eval/logs/vanilla/ruler_b.log ADDED Viewed

@@ -0,0 +1,369 @@
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:14:27:53 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:27:57 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:14:27:58 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:27:58 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:28:01 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:28:02 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:28:11 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:11 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:14:28:21 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:30 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:36 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:46 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:46 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:14:28:53 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:29:05 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:29:05 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:14:29:05 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:14:29:05 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:14:29:05 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:14:29:05 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:29:05 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:29:05 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:29:05 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:29:05 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:38:04 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:38:04 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/*.jsonl
+hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_1|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multikey_3|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7500|±  |   N/A|
+|niah_multivalue|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_2  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|ruler_cwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4250|±  |   N/A|
+|ruler_qa_squad |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4208|±  |   N/A|

outputs/eval/vanilla/babilong.DONE ADDED Viewed

File without changes

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/results_2026-07-18T14-41-29.909583.json ADDED Viewed

	@@ -0,0 +1,505 @@

+{
+  "results": {
+    "babilong_longctx": {
+      "acc,none": 0.6,
+      "acc_stderr,none": 0.02948261334746155,
+      "alias": "babilong_longctx"
+    },
+    "babilong_qa1": {
+      "alias": " - babilong_qa1",
+      "acc,none": 0.68,
+      "acc_stderr,none": 0.06663945022680343
+    },
+    "babilong_qa2": {
+      "alias": " - babilong_qa2",
+      "acc,none": 0.48,
+      "acc_stderr,none": 0.07137140569598172
+    },
+    "babilong_qa3": {
+      "alias": " - babilong_qa3",
+      "acc,none": 0.34,
+      "acc_stderr,none": 0.06767268161329718
+    },
+    "babilong_qa4": {
+      "alias": " - babilong_qa4",
+      "acc,none": 0.72,
+      "acc_stderr,none": 0.06414269805898185
+    },
+    "babilong_qa5": {
+      "alias": " - babilong_qa5",
+      "acc,none": 0.78,
+      "acc_stderr,none": 0.059178043363451345
+    }
+  },
+  "groups": {
+    "babilong_longctx": {
+      "acc,none": 0.6,
+      "acc_stderr,none": 0.02948261334746155,
+      "alias": "babilong_longctx"
+    }
+  },
+  "group_subtasks": {
+    "babilong_longctx": [
+      "babilong_qa1",
+      "babilong_qa2",
+      "babilong_qa3",
+      "babilong_qa4",
+      "babilong_qa5"
+    ]
+  },
+  "configs": {
+    "babilong_qa1": {
+      "task": "babilong_qa1",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa1"
+      },
+      "test_split": "qa1",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about positions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person was in different locations, use the latest location to answer the question.\nAlways return your answer in the following format:\nThe most recent location of 'person' is 'location'. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Charlie went to the hallway. Judith come back to the kitchen. Charlie travelled to balcony.",
+            "question": "Where is Charlie?",
+            "target": "The most recent location of Charlie is balcony."
+          },
+          {
+            "input": "Alan moved to the garage. Charlie went to the beach. Alan went to the shop. Rouse travelled to balcony.",
+            "question": "Where is Alan?",
+            "target": "The most recent location of Alan is shop."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa2": {
+      "task": "babilong_qa2",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa2"
+      },
+      "test_split": "qa2",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about locations and actions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person got an item in the first location and travelled to the second location the item is also in the second location. If a person dropped an item in the first location and moved to the second location the item remains in the first location.\nAlways return your answer in the following format:\nThe 'item' is in 'location'. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Charlie went to the kitchen. Charlie got a bottle. Charlie moved to the balcony.",
+            "question": "Where is the bottle?",
+            "target": "The bottle is in the balcony."
+          },
+          {
+            "input": "Alan moved to the garage. Alan got a screw driver. Alan moved to the kitchen.",
+            "question": "Where is the screw driver?",
+            "target": "The screw driver is in the kitchen."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa3": {
+      "task": "babilong_qa3",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa3"
+      },
+      "test_split": "qa3",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I give you context with the facts about locations and actions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person got an item in the first location and travelled to the second location the item is also in the second location. If a person dropped an item in the first location and moved to the second location the item remains in the first location.\nAlways return your answer in the following format:\nBefore the $location_1$ the $item$ was in the $location_2$. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "John journeyed to the bedroom. Mary grabbed the apple. Mary went back to the bathroom. Daniel journeyed to the bedroom. Daniel moved to the garden. Mary travelled to the kitchen.",
+            "question": "Where was the apple before the kitchen?",
+            "target": "Before the kitchen the apple was in the bathroom."
+          },
+          {
+            "input": "John went back to the bedroom. John went back to the garden. John went back to the kitchen. Sandra took the football. Sandra travelled to the garden. Sandra journeyed to the bedroom.",
+            "question": "Where was the football before the bedroom?",
+            "target": "Before the bedroom the football was in the garden."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa4": {
+      "task": "babilong_qa4",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa4"
+      },
+      "test_split": "qa4",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about different people, their location and actions, hidden in some random text and a question. You need to answer the question based only on the information from the facts.\nYour answer should contain only one word - location. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "The hallway is south of the kitchen. The bedroom is north of the kitchen.",
+            "question": "What is the kitchen south of?",
+            "target": "bedroom"
+          },
+          {
+            "input": "The garden is west of the bedroom. The bedroom is west of the kitchen.",
+            "question": "What is west of the bedroom?",
+            "target": "garden"
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa5": {
+      "task": "babilong_qa5",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa5"
+      },
+      "test_split": "qa5",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about locations and their relations hidden in some random text and a question. You need to answer the question based only on the information from the facts.\nYour answer should contain only one word. Do not write anything else after that. Do not explain your answer.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Mary picked up the apple there. Mary gave the apple to Fred. Mary moved to the bedroom. Bill took the milk there.",
+            "question": "Who did Mary give the apple to?",
+            "target": "Fred"
+          },
+          {
+            "input": "Jeff took the football there. Jeff passed the football to Fred. Jeff got the milk there. Bill travelled to the bedroom.",
+            "question": "Who gave the football?",
+            "target": "Jeff"
+          },
+          {
+            "input": "Fred picked up the apple there. Fred handed the apple to Bill. Bill journeyed to the bedroom. Jeff went back to the garden.",
+            "question": "What did Fred give to Bill?",
+            "target": "apple"
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    }
+  },
+  "versions": {
+    "babilong_longctx": 0.0,
+    "babilong_qa1": 0.0,
+    "babilong_qa2": 0.0,
+    "babilong_qa3": 0.0,
+    "babilong_qa4": 0.0,
+    "babilong_qa5": 0.0
+  },
+  "n-shot": {
+    "babilong_qa1": 2,
+    "babilong_qa2": 2,
+    "babilong_qa3": 2,
+    "babilong_qa4": 2,
+    "babilong_qa5": 2
+  },
+  "higher_is_better": {
+    "babilong_longctx": {
+      "acc": true
+    },
+    "babilong_qa1": {
+      "acc": true
+    },
+    "babilong_qa2": {
+      "acc": true
+    },
+    "babilong_qa3": {
+      "acc": true
+    },
+    "babilong_qa4": {
+      "acc": true
+    },
+    "babilong_qa5": {
+      "acc": true
+    }
+  },
+  "n-samples": {
+    "babilong_qa1": {
+      "original": 1000,
+      "effective": 50
+    },
+    "babilong_qa2": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa3": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa4": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa5": {
+      "original": 999,
+      "effective": 50
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 1720574976,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 50.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784385490.7183907,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                5999.99\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "babilong_qa1": "bbdcd109bea1bb245e118e814bd0563de6384bc875a21df5218ab4ed75b27380",
+    "babilong_qa2": "fa4f8d12bdd481e02a1051414304811f648caf7bce8f89dcf03ecbf5b86f1e92",
+    "babilong_qa3": "737b64396c8b2e5f96da9796bcd5e448f08e5a13e32ae7b610d31a60269b6e48",
+    "babilong_qa4": "298a25727c69c9babddcd78f438cae1aa21a388e14e7db90b2206eff753c90c0",
+    "babilong_qa5": "e60afcdcdd1c4b3e24b002ffbf8f87085f5ec4a02fcd72318f042dcdafe230f4"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace",
+  "model_name_sanitized": "__workspace",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "202.4258166590007"
+}

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa1_2026-07-18T14-41-29.909583.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7dc0efd1b7fa32f44cdbcbc69a595406a69fad96d7ee9cc70b239280be5e8864
+size 3164110

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa2_2026-07-18T14-41-29.909583.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4fe2652bb38bcd261174fbc66a7385060c563dc46c3de1dc631122a72a3c43e1
+size 3147520

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa3_2026-07-18T14-41-29.909583.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:44c6580367ef0fac22336fd456f584c948466729af4c41206f1e0b2e4285f118
+size 3126960

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa4_2026-07-18T14-41-29.909583.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b46c44cd2585ae77c39c98e86fc7029938668286a9bddea9f7c9d4f478e1f181
+size 3137616

outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/samples_babilong_qa5_2026-07-18T14-41-29.909583.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:af8bdf34f1a7bcc55f1403126e730ea9fc18a4bc56110b95e7d72ad2c16cd5d2
+size 3135608

outputs/eval/vanilla/helmet.DONE ADDED Viewed

File without changes

outputs/eval/vanilla/helmet_icl8k_n50.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ae00423c60486e03e549e1b0f0522138862581236d7d861e2ce3859c14367880
+size 6467522

outputs/eval/vanilla/helmet_icl8k_n50.summary.csv ADDED Viewed

	@@ -0,0 +1,251 @@

+method,benchmark,config,n,score,prompt_tokens
+vanilla,helmet_icl,trec_coarse,1,1.0,6597
+vanilla,helmet_icl,trec_coarse,1,0.0,6549
+vanilla,helmet_icl,trec_coarse,1,0.0,6508
+vanilla,helmet_icl,trec_coarse,1,1.0,6608
+vanilla,helmet_icl,trec_coarse,1,1.0,6497
+vanilla,helmet_icl,trec_coarse,1,0.0,6545
+vanilla,helmet_icl,trec_coarse,1,1.0,6583
+vanilla,helmet_icl,trec_coarse,1,1.0,6455
+vanilla,helmet_icl,trec_coarse,1,1.0,6643
+vanilla,helmet_icl,trec_coarse,1,1.0,6568
+vanilla,helmet_icl,trec_coarse,1,0.0,6518
+vanilla,helmet_icl,trec_coarse,1,1.0,6424
+vanilla,helmet_icl,trec_coarse,1,1.0,6750
+vanilla,helmet_icl,trec_coarse,1,1.0,6642
+vanilla,helmet_icl,trec_coarse,1,1.0,6494
+vanilla,helmet_icl,trec_coarse,1,0.0,6427
+vanilla,helmet_icl,trec_coarse,1,1.0,6547
+vanilla,helmet_icl,trec_coarse,1,0.0,6612
+vanilla,helmet_icl,trec_coarse,1,1.0,6591
+vanilla,helmet_icl,trec_coarse,1,1.0,6485
+vanilla,helmet_icl,trec_coarse,1,1.0,6542
+vanilla,helmet_icl,trec_coarse,1,1.0,6409
+vanilla,helmet_icl,trec_coarse,1,0.0,6482
+vanilla,helmet_icl,trec_coarse,1,1.0,6502
+vanilla,helmet_icl,trec_coarse,1,0.0,6344
+vanilla,helmet_icl,trec_coarse,1,1.0,6479
+vanilla,helmet_icl,trec_coarse,1,1.0,6569
+vanilla,helmet_icl,trec_coarse,1,1.0,6389
+vanilla,helmet_icl,trec_coarse,1,1.0,6479
+vanilla,helmet_icl,trec_coarse,1,0.0,6653
+vanilla,helmet_icl,trec_coarse,1,1.0,6532
+vanilla,helmet_icl,trec_coarse,1,1.0,6491
+vanilla,helmet_icl,trec_coarse,1,1.0,6489
+vanilla,helmet_icl,trec_coarse,1,0.0,6630
+vanilla,helmet_icl,trec_coarse,1,1.0,6488
+vanilla,helmet_icl,trec_coarse,1,1.0,6499
+vanilla,helmet_icl,trec_coarse,1,0.0,6483
+vanilla,helmet_icl,trec_coarse,1,1.0,6660
+vanilla,helmet_icl,trec_coarse,1,0.0,6675
+vanilla,helmet_icl,trec_coarse,1,0.0,6544
+vanilla,helmet_icl,trec_coarse,1,1.0,6520
+vanilla,helmet_icl,trec_coarse,1,1.0,6359
+vanilla,helmet_icl,trec_coarse,1,0.0,6657
+vanilla,helmet_icl,trec_coarse,1,0.0,6546
+vanilla,helmet_icl,trec_coarse,1,1.0,6481
+vanilla,helmet_icl,trec_coarse,1,1.0,6506
+vanilla,helmet_icl,trec_coarse,1,0.0,6621
+vanilla,helmet_icl,trec_coarse,1,0.0,6623
+vanilla,helmet_icl,trec_coarse,1,0.0,6411
+vanilla,helmet_icl,trec_coarse,1,1.0,6577
+vanilla,helmet_icl,trec_fine,1,0.0,6858
+vanilla,helmet_icl,trec_fine,1,1.0,6973
+vanilla,helmet_icl,trec_fine,1,0.0,6977
+vanilla,helmet_icl,trec_fine,1,1.0,6952
+vanilla,helmet_icl,trec_fine,1,0.0,6919
+vanilla,helmet_icl,trec_fine,1,1.0,6979
+vanilla,helmet_icl,trec_fine,1,1.0,6957
+vanilla,helmet_icl,trec_fine,1,0.0,7013
+vanilla,helmet_icl,trec_fine,1,1.0,6824
+vanilla,helmet_icl,trec_fine,1,1.0,6833
+vanilla,helmet_icl,trec_fine,1,0.0,6868
+vanilla,helmet_icl,trec_fine,1,1.0,6914
+vanilla,helmet_icl,trec_fine,1,1.0,6759
+vanilla,helmet_icl,trec_fine,1,1.0,6933
+vanilla,helmet_icl,trec_fine,1,0.0,6868
+vanilla,helmet_icl,trec_fine,1,0.0,6839
+vanilla,helmet_icl,trec_fine,1,0.0,6838
+vanilla,helmet_icl,trec_fine,1,0.0,7000
+vanilla,helmet_icl,trec_fine,1,0.0,6800
+vanilla,helmet_icl,trec_fine,1,0.0,6829
+vanilla,helmet_icl,trec_fine,1,0.0,6971
+vanilla,helmet_icl,trec_fine,1,1.0,6907
+vanilla,helmet_icl,trec_fine,1,0.0,6818
+vanilla,helmet_icl,trec_fine,1,1.0,6902
+vanilla,helmet_icl,trec_fine,1,1.0,6941
+vanilla,helmet_icl,trec_fine,1,0.0,6830
+vanilla,helmet_icl,trec_fine,1,0.0,6934
+vanilla,helmet_icl,trec_fine,1,1.0,6879
+vanilla,helmet_icl,trec_fine,1,1.0,6915
+vanilla,helmet_icl,trec_fine,1,0.0,6953
+vanilla,helmet_icl,trec_fine,1,0.0,6875
+vanilla,helmet_icl,trec_fine,1,1.0,6847
+vanilla,helmet_icl,trec_fine,1,0.0,6847
+vanilla,helmet_icl,trec_fine,1,0.0,6944
+vanilla,helmet_icl,trec_fine,1,0.0,6818
+vanilla,helmet_icl,trec_fine,1,0.0,6868
+vanilla,helmet_icl,trec_fine,1,0.0,7022
+vanilla,helmet_icl,trec_fine,1,0.0,6930
+vanilla,helmet_icl,trec_fine,1,0.0,6863
+vanilla,helmet_icl,trec_fine,1,0.0,6807
+vanilla,helmet_icl,trec_fine,1,0.0,7001
+vanilla,helmet_icl,trec_fine,1,0.0,6890
+vanilla,helmet_icl,trec_fine,1,0.0,6952
+vanilla,helmet_icl,trec_fine,1,1.0,6845
+vanilla,helmet_icl,trec_fine,1,0.0,6902
+vanilla,helmet_icl,trec_fine,1,0.0,6879
+vanilla,helmet_icl,trec_fine,1,1.0,6896
+vanilla,helmet_icl,trec_fine,1,0.0,6770
+vanilla,helmet_icl,trec_fine,1,0.0,6933
+vanilla,helmet_icl,trec_fine,1,1.0,6886
+vanilla,helmet_icl,banking77,1,1.0,6809
+vanilla,helmet_icl,banking77,1,0.0,7162
+vanilla,helmet_icl,banking77,1,1.0,6876
+vanilla,helmet_icl,banking77,1,1.0,7033
+vanilla,helmet_icl,banking77,1,1.0,6647
+vanilla,helmet_icl,banking77,1,1.0,6818
+vanilla,helmet_icl,banking77,1,1.0,6841
+vanilla,helmet_icl,banking77,1,1.0,6944
+vanilla,helmet_icl,banking77,1,1.0,6776
+vanilla,helmet_icl,banking77,1,1.0,7100
+vanilla,helmet_icl,banking77,1,0.0,6967
+vanilla,helmet_icl,banking77,1,1.0,6894
+vanilla,helmet_icl,banking77,1,1.0,6900
+vanilla,helmet_icl,banking77,1,0.0,6964
+vanilla,helmet_icl,banking77,1,1.0,7085
+vanilla,helmet_icl,banking77,1,1.0,6953
+vanilla,helmet_icl,banking77,1,0.0,7406
+vanilla,helmet_icl,banking77,1,0.0,6802
+vanilla,helmet_icl,banking77,1,1.0,7119
+vanilla,helmet_icl,banking77,1,0.0,7202
+vanilla,helmet_icl,banking77,1,0.0,7103
+vanilla,helmet_icl,banking77,1,0.0,7009
+vanilla,helmet_icl,banking77,1,1.0,6940
+vanilla,helmet_icl,banking77,1,0.0,6857
+vanilla,helmet_icl,banking77,1,0.0,6922
+vanilla,helmet_icl,banking77,1,1.0,6988
+vanilla,helmet_icl,banking77,1,1.0,6945
+vanilla,helmet_icl,banking77,1,1.0,6835
+vanilla,helmet_icl,banking77,1,0.0,7035
+vanilla,helmet_icl,banking77,1,1.0,6787
+vanilla,helmet_icl,banking77,1,0.0,6940
+vanilla,helmet_icl,banking77,1,0.0,6661
+vanilla,helmet_icl,banking77,1,1.0,6934
+vanilla,helmet_icl,banking77,1,0.0,7235
+vanilla,helmet_icl,banking77,1,1.0,6829
+vanilla,helmet_icl,banking77,1,1.0,7169
+vanilla,helmet_icl,banking77,1,0.0,7074
+vanilla,helmet_icl,banking77,1,0.0,6797
+vanilla,helmet_icl,banking77,1,0.0,7013
+vanilla,helmet_icl,banking77,1,1.0,6811
+vanilla,helmet_icl,banking77,1,0.0,6983
+vanilla,helmet_icl,banking77,1,0.0,7183
+vanilla,helmet_icl,banking77,1,1.0,6801
+vanilla,helmet_icl,banking77,1,0.0,7174
+vanilla,helmet_icl,banking77,1,1.0,6773
+vanilla,helmet_icl,banking77,1,1.0,7134
+vanilla,helmet_icl,banking77,1,1.0,6907
+vanilla,helmet_icl,banking77,1,1.0,7341
+vanilla,helmet_icl,banking77,1,1.0,7042
+vanilla,helmet_icl,banking77,1,1.0,7304
+vanilla,helmet_icl,clinic150,1,1.0,7241
+vanilla,helmet_icl,clinic150,1,0.0,7282
+vanilla,helmet_icl,clinic150,1,1.0,7261
+vanilla,helmet_icl,clinic150,1,0.0,7292
+vanilla,helmet_icl,clinic150,1,1.0,7217
+vanilla,helmet_icl,clinic150,1,0.0,7280
+vanilla,helmet_icl,clinic150,1,1.0,7293
+vanilla,helmet_icl,clinic150,1,1.0,7250
+vanilla,helmet_icl,clinic150,1,1.0,7264
+vanilla,helmet_icl,clinic150,1,1.0,7238
+vanilla,helmet_icl,clinic150,1,1.0,7243
+vanilla,helmet_icl,clinic150,1,1.0,7191
+vanilla,helmet_icl,clinic150,1,1.0,7333
+vanilla,helmet_icl,clinic150,1,1.0,7299
+vanilla,helmet_icl,clinic150,1,1.0,7225
+vanilla,helmet_icl,clinic150,1,1.0,7409
+vanilla,helmet_icl,clinic150,1,1.0,7277
+vanilla,helmet_icl,clinic150,1,1.0,7376
+vanilla,helmet_icl,clinic150,1,1.0,7383
+vanilla,helmet_icl,clinic150,1,0.0,7270
+vanilla,helmet_icl,clinic150,1,1.0,7173
+vanilla,helmet_icl,clinic150,1,1.0,7106
+vanilla,helmet_icl,clinic150,1,1.0,7413
+vanilla,helmet_icl,clinic150,1,1.0,7253
+vanilla,helmet_icl,clinic150,1,0.0,7224
+vanilla,helmet_icl,clinic150,1,1.0,7307
+vanilla,helmet_icl,clinic150,1,1.0,7272
+vanilla,helmet_icl,clinic150,1,0.0,7215
+vanilla,helmet_icl,clinic150,1,1.0,7344
+vanilla,helmet_icl,clinic150,1,1.0,7224
+vanilla,helmet_icl,clinic150,1,1.0,7310
+vanilla,helmet_icl,clinic150,1,1.0,7222
+vanilla,helmet_icl,clinic150,1,0.0,7260
+vanilla,helmet_icl,clinic150,1,1.0,7351
+vanilla,helmet_icl,clinic150,1,1.0,7301
+vanilla,helmet_icl,clinic150,1,1.0,7155
+vanilla,helmet_icl,clinic150,1,1.0,7253
+vanilla,helmet_icl,clinic150,1,1.0,7248
+vanilla,helmet_icl,clinic150,1,1.0,7128
+vanilla,helmet_icl,clinic150,1,0.0,7282
+vanilla,helmet_icl,clinic150,1,1.0,7301
+vanilla,helmet_icl,clinic150,1,1.0,7388
+vanilla,helmet_icl,clinic150,1,1.0,7244
+vanilla,helmet_icl,clinic150,1,1.0,7186
+vanilla,helmet_icl,clinic150,1,0.0,7309
+vanilla,helmet_icl,clinic150,1,1.0,7241
+vanilla,helmet_icl,clinic150,1,1.0,7214
+vanilla,helmet_icl,clinic150,1,0.0,7264
+vanilla,helmet_icl,clinic150,1,0.0,7219
+vanilla,helmet_icl,clinic150,1,1.0,7228
+vanilla,helmet_icl,nlu,1,1.0,7063
+vanilla,helmet_icl,nlu,1,1.0,6959
+vanilla,helmet_icl,nlu,1,0.0,6985
+vanilla,helmet_icl,nlu,1,1.0,6970
+vanilla,helmet_icl,nlu,1,1.0,7025
+vanilla,helmet_icl,nlu,1,0.0,7034
+vanilla,helmet_icl,nlu,1,1.0,7004
+vanilla,helmet_icl,nlu,1,0.0,7062
+vanilla,helmet_icl,nlu,1,0.0,7062
+vanilla,helmet_icl,nlu,1,0.0,7045
+vanilla,helmet_icl,nlu,1,1.0,7028
+vanilla,helmet_icl,nlu,1,1.0,7018
+vanilla,helmet_icl,nlu,1,1.0,7075
+vanilla,helmet_icl,nlu,1,1.0,7068
+vanilla,helmet_icl,nlu,1,1.0,7025
+vanilla,helmet_icl,nlu,1,1.0,7022
+vanilla,helmet_icl,nlu,1,1.0,7112
+vanilla,helmet_icl,nlu,1,1.0,6967
+vanilla,helmet_icl,nlu,1,0.0,6980
+vanilla,helmet_icl,nlu,1,1.0,7031
+vanilla,helmet_icl,nlu,1,0.0,7093
+vanilla,helmet_icl,nlu,1,0.0,6949
+vanilla,helmet_icl,nlu,1,1.0,7046
+vanilla,helmet_icl,nlu,1,1.0,7039
+vanilla,helmet_icl,nlu,1,0.0,7126
+vanilla,helmet_icl,nlu,1,1.0,7064
+vanilla,helmet_icl,nlu,1,1.0,7057
+vanilla,helmet_icl,nlu,1,0.0,6971
+vanilla,helmet_icl,nlu,1,0.0,7034
+vanilla,helmet_icl,nlu,1,1.0,7094
+vanilla,helmet_icl,nlu,1,1.0,6979
+vanilla,helmet_icl,nlu,1,1.0,7123
+vanilla,helmet_icl,nlu,1,1.0,7144
+vanilla,helmet_icl,nlu,1,1.0,7024
+vanilla,helmet_icl,nlu,1,1.0,6943
+vanilla,helmet_icl,nlu,1,0.0,6961
+vanilla,helmet_icl,nlu,1,1.0,7005
+vanilla,helmet_icl,nlu,1,1.0,7080
+vanilla,helmet_icl,nlu,1,1.0,7179
+vanilla,helmet_icl,nlu,1,1.0,7115
+vanilla,helmet_icl,nlu,1,1.0,6946
+vanilla,helmet_icl,nlu,1,1.0,6985
+vanilla,helmet_icl,nlu,1,1.0,7131
+vanilla,helmet_icl,nlu,1,1.0,7100
+vanilla,helmet_icl,nlu,1,1.0,7190
+vanilla,helmet_icl,nlu,1,1.0,7040
+vanilla,helmet_icl,nlu,1,1.0,7013
+vanilla,helmet_icl,nlu,1,0.0,7080
+vanilla,helmet_icl,nlu,1,0.0,7098
+vanilla,helmet_icl,nlu,1,1.0,6891

outputs/eval/vanilla/mrcr.DONE ADDED Viewed

File without changes

outputs/eval/vanilla/mrcr_8k_2_4_8needle_n10.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3955786bd7950d0cc58fc0ccbd3d339f9ef4e8926eaf4e6c451e3f7fe4cb342d
+size 1003361

outputs/eval/vanilla/mrcr_8k_2_4_8needle_n10.summary.csv ADDED Viewed

	@@ -0,0 +1,31 @@

+method,benchmark,config,n,score,prompt_tokens
+vanilla,mrcr,8k_2needle,1,0.07609013754755634,4852
+vanilla,mrcr,8k_2needle,1,0.983710407239819,7119
+vanilla,mrcr,8k_2needle,1,0.038006952491309386,5756
+vanilla,mrcr,8k_2needle,1,0.05052372150338879,5072
+vanilla,mrcr,8k_2needle,1,0.07633928571428572,5753
+vanilla,mrcr,8k_2needle,1,0.5607594936708861,5481
+vanilla,mrcr,8k_2needle,1,1.0,5143
+vanilla,mrcr,8k_2needle,1,1.0,6399
+vanilla,mrcr,8k_2needle,1,1.0,5987
+vanilla,mrcr,8k_2needle,1,0.0767583006069261,4380
+vanilla,mrcr,8k_4needle,1,0.06784059314179797,5123
+vanilla,mrcr,8k_4needle,1,0.03232242617717478,7908
+vanilla,mrcr,8k_4needle,1,0.02783400809716599,5814
+vanilla,mrcr,8k_4needle,1,0.9595443833464258,4856
+vanilla,mrcr,8k_4needle,1,0.0,7413
+vanilla,mrcr,8k_4needle,1,0.007665505226480836,7764
+vanilla,mrcr,8k_4needle,1,0.08235294117647059,4312
+vanilla,mrcr,8k_4needle,1,0.03953445409200074,4847
+vanilla,mrcr,8k_4needle,1,0.08090698273640815,6287
+vanilla,mrcr,8k_4needle,1,0.3413333333333333,7841
+vanilla,mrcr,8k_8needle,1,0.009378663540445486,4581
+vanilla,mrcr,8k_8needle,1,0.18287107589149648,6357
+vanilla,mrcr,8k_8needle,1,0.0962800875273523,6981
+vanilla,mrcr,8k_8needle,1,0.04247286455875413,6308
+vanilla,mrcr,8k_8needle,1,0.0,6901
+vanilla,mrcr,8k_8needle,1,0.02224694104560623,5407
+vanilla,mrcr,8k_8needle,1,0.1317365269461078,6351
+vanilla,mrcr,8k_8needle,1,0.9895,5657
+vanilla,mrcr,8k_8needle,1,0.0,7330
+vanilla,mrcr,8k_8needle,1,0.076951811729548,8028

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/results_2026-07-18T14-27-50.984361.json ADDED Viewed

	@@ -0,0 +1,821 @@

+{
+  "results": {
+    "niah_multikey_2": {
+      "alias": "niah_multikey_2",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multiquery": {
+      "alias": "niah_multiquery",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.9875,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_1": {
+      "alias": "niah_single_1",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_3": {
+      "alias": "niah_single_3",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_fwe": {
+      "alias": "ruler_fwe",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.7666666666666666,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_qa_hotpot": {
+      "alias": "ruler_qa_hotpot",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.45,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_vt": {
+      "alias": "ruler_vt",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.8600000000000001,
+      "8192_stderr,none": "N/A"
+    }
+  },
+  "group_subtasks": {
+    "niah_single_1": [],
+    "niah_single_3": [],
+    "niah_multikey_2": [],
+    "niah_multiquery": [],
+    "ruler_vt": [],
+    "ruler_fwe": [],
+    "ruler_qa_hotpot": []
+  },
+  "configs": {
+    "niah_multikey_2": {
+      "task": "niah_multikey_2",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_2(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"needle\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"needle\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multiquery": {
+      "task": "niah_multiquery",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multiquery(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_q=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_1": {
+      "task": "niah_single_1",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_1(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"repeat\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"repeat\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_3": {
+      "task": "niah_single_3",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_3(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"uuids\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_fwe": {
+      "task": "ruler_fwe",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def fwe_download(**kwargs):\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", {}))\n    df = (\n        get_dataset(pretrained, max_seq_length=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 50,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_qa_hotpot": {
+      "task": "ruler_qa_hotpot",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_hotpotqa(**kwargs):\n    return get_qa_dataset(\"hotpotqa\", **kwargs)\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results_part(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_part(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 32,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_vt": {
+      "task": "ruler_vt",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_vt_dataset(**kwargs) -> dict[str, datasets.Dataset]:\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", \"\"))\n    df = (\n        get_dataset(tokenizer=get_tokenizer(pretrained), seq=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 30,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    }
+  },
+  "versions": {
+    "niah_multikey_2": 1.0,
+    "niah_multiquery": 1.0,
+    "niah_single_1": 1.0,
+    "niah_single_3": 1.0,
+    "ruler_fwe": 1.0,
+    "ruler_qa_hotpot": 1.0,
+    "ruler_vt": 1.0
+  },
+  "n-shot": {
+    "niah_multikey_2": 0,
+    "niah_multiquery": 0,
+    "niah_single_1": 0,
+    "niah_single_3": 0,
+    "ruler_fwe": 0,
+    "ruler_qa_hotpot": 0,
+    "ruler_vt": 0
+  },
+  "higher_is_better": {
+    "niah_multikey_2": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multiquery": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_1": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_3": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_fwe": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_qa_hotpot": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_vt": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    }
+  },
+  "n-samples": {
+    "ruler_qa_hotpot": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_fwe": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_vt": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multiquery": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_2": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_3": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_1": {
+      "original": 500,
+      "effective": 20
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 1720574976,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 20.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784384235.5860372,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                5999.99\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "ruler_qa_hotpot": "fd48cb605efc8c5adac7e3771533db4d4c354b71bf91ce00841c814604ba6404",
+    "ruler_fwe": "c4633bcd0acf3ba4b96f905b5ba269ba94cf45a4c8ca0c13aae1468227cc3a10",
+    "ruler_vt": "af011da859c8b86f63f57f6256cf59077d3d3d8536084ab67ccf39e18e9a615a",
+    "niah_multiquery": "f9a36d2854dfd2bb52ffc7a6bf3f53490fb5303ca9da0046256bc28b55c2adb2",
+    "niah_multikey_2": "86e49b8f875843454b2923143b9124ba82e8a2e23b145f792726cd82f4e49c24",
+    "niah_single_3": "905e40146a287e5f0c8e2a81f64f082108c45a23862949e6a99bfcbbf561cad2",
+    "niah_single_1": "426e8138e992480aa416d01e6b9cef5865a68c00f532400e448a0c20ee026cff"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace",
+  "model_name_sanitized": "__workspace",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "638.6952040239994"
+}

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_multikey_2_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:361b487bcd3e2e312e8e4441fa89adee971d7220e70e4cd5369b258dfe11ebb3
+size 987656

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_multiquery_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cb22b0f677274eb81acd2a4c5d25588730c15c64831db15e9a8e5118f40be994
+size 1440839

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_single_1_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8e4fcb6babd01a97926f269bce4b8a4f4094d0c45e5ee43a0e2c765cdee5d41c
+size 1231702

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_niah_single_3_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0af0fa3b3696d5cd6d6146d52ffaa97166940d2a8d880a1ee91ebb119cd5766b
+size 1435106

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_fwe_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4ba6e97fbf67abd0dc6764e64c4887c73bde6f1d6561b76f88453c4aa0c2e008
+size 827107

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_qa_hotpot_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4034286d389591e80daeda1a9caf84252288fdcd48fdb871faf236566e2e47ea
+size 1172370

outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/samples_ruler_vt_2026-07-18T14-27-50.984361.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b8669268b4a1595f6edcd39938b92155d4aae0453616ca5cf2b531f4244bd3
+size 1228996

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/results_2026-07-18T14-38-04.340503.json ADDED Viewed

	@@ -0,0 +1,714 @@

+{
+  "results": {
+    "niah_multikey_1": {
+      "alias": "niah_multikey_1",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multikey_3": {
+      "alias": "niah_multikey_3",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.75,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multivalue": {
+      "alias": "niah_multivalue",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_2": {
+      "alias": "niah_single_2",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_cwe": {
+      "alias": "ruler_cwe",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.425,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_qa_squad": {
+      "alias": "ruler_qa_squad",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.4208333333333334,
+      "8192_stderr,none": "N/A"
+    }
+  },
+  "group_subtasks": {
+    "niah_single_2": [],
+    "niah_multikey_1": [],
+    "niah_multikey_3": [],
+    "niah_multivalue": [],
+    "ruler_cwe": [],
+    "ruler_qa_squad": []
+  },
+  "configs": {
+    "niah_multikey_1": {
+      "task": "niah_multikey_1",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_1(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_k=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multikey_3": {
+      "task": "niah_multikey_3",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_3(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"needle\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"needle\",\n            type_needle_k=\"uuids\",\n            type_needle_v=\"uuids\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multivalue": {
+      "task": "niah_multivalue",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multivalue(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_v=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_2": {
+      "task": "niah_single_2",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_2(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_cwe": {
+      "task": "ruler_cwe",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_cw_dataset(**kwargs):\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", {}))\n    df = (\n        get_dataset(pretrained, seq=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": "\n\n",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": "\n\n"
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 120,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_qa_squad": {
+      "task": "ruler_qa_squad",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_squad(**kwargs):\n    return get_qa_dataset(\"squad\", **kwargs)\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results_part(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_part(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 32,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    }
+  },
+  "versions": {
+    "niah_multikey_1": 1.0,
+    "niah_multikey_3": 1.0,
+    "niah_multivalue": 1.0,
+    "niah_single_2": 1.0,
+    "ruler_cwe": 1.0,
+    "ruler_qa_squad": 1.0
+  },
+  "n-shot": {
+    "niah_multikey_1": 0,
+    "niah_multikey_3": 0,
+    "niah_multivalue": 0,
+    "niah_single_2": 0,
+    "ruler_cwe": 0,
+    "ruler_qa_squad": 0
+  },
+  "higher_is_better": {
+    "niah_multikey_1": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multikey_3": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multivalue": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_2": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_cwe": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_qa_squad": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    }
+  },
+  "n-samples": {
+    "ruler_qa_squad": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_cwe": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multivalue": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_3": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_1": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_2": {
+      "original": 500,
+      "effective": 20
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 1720574976,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 20.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784384877.4890916,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                5999.99\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "ruler_qa_squad": "8c658a728ccd3d67ddd3f45e8e76527cf15a1ae4119033d8246f293ecc68cf11",
+    "ruler_cwe": "f68206129fc7d709951d550ec9269fed05e73bea2fc86ce9f2c758974db5db71",
+    "niah_multivalue": "29646655c230b4f155a8de1fbc7ddf22c49e43045bd8fdab5770e7cb45026c7c",
+    "niah_multikey_3": "de2cec52885e38d56e0bda0e344e989909969c71ab5f8048b9368a011dd3616b",
+    "niah_multikey_1": "338f81e9efa98dbe6979d4f178f7f06cdbd6d367072318667a932b43c5032ab8",
+    "niah_single_2": "88ba10e924c2fc6c47543eb646b9e6f7487c7ca026c9391d6cd7054fde59eaad"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace",
+  "model_name_sanitized": "__workspace",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "610.0602397490002"
+}

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multikey_1_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:493f48f51a7983128854573427065eb25400a5133e6677df10d77e8c732f2314
+size 1445254

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multikey_3_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:16640bcab759eca3acd8f774ccd690dd64ba6aca8372f3156953491f05e66628
+size 502636

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_multivalue_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a65a4a964718a8b1cb274ada7752eccc6324315f1eb5e6550147e6e5c09d69b2
+size 1432692

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_niah_single_2_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:44f68816f94020359ab07c983dd48406b1e4add11067dd0d5e0eb5d03ecc4356
+size 1438132

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_ruler_cwe_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:076ac6d26f881a75b2849588fa9d29d4f35fff8cb9e0eddc478212dd5f921a01
+size 634811

outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/samples_ruler_qa_squad_2026-07-18T14-38-04.340503.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:511e125acd68b2c72221cb76bfe7b130500e426a655225cb703a95f1760a7880
+size 1387264

outputs/eval/vanilla/ruler_a.DONE ADDED Viewed

File without changes

outputs/eval/vanilla/ruler_b.DONE ADDED Viewed

File without changes