README.md

metadata

base_model: Delta-Vector/Austral-24B-Winton
base_model_relation: quantized
quantized_by: ArtusDev
license: apache-2.0
language:
  - en
library_name: transformers
tags:
  - roleplay
  - finetune
  - axolotl
  - adventure
  - creative-writing
  - Mistral
  - 24B
  - exl2

EXL2 Quants of Delta-Vector/Austral-24B-Winton

EXL2 quants of Delta-Vector/Austral-24B-Winton using exllamav2 for quantization.

Quants

Quant(Revision)	Bits per Weight	Head Bits
4.0_H6	4.0	6
4.5_H6	4.5	6

Downloading quants with huggingface-cli

Click to view download instructions

Install hugginface-cli:

pip install -U "huggingface_hub[cli]"

Download quant by targeting the specific quant revision (branch):

huggingface-cli download ArtusDev/Delta-Vector_Austral-24B-Winton-EXL2 --revision "5.0bpw_H6" --local-dir ./