Hugging Face's logo

Triangle104
/
Magistral-Small-2506-Q3_K_L-GGUF

Text Generation
GGUF
mistral
unsloth
llama-cpp
gguf-my-repo
text2text-generation
conversational
Model card Files
xet
Community
YAML Metadata Warning: The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other

Triangle104/Magistral-Small-2506-Q3_K_L-GGUF

This model was converted to GGUF format from unsloth/Magistral-Small-2506 using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Building upon Mistral Small 3.1 (2503), with added reasoning capabilities, undergoing SFT from Magistral Medium traces and RL on top, it's a small, efficient reasoning model with 24B parameters.

Magistral Small can be deployed locally, fitting within a single RTX 4090 or a 32GB RAM MacBook once quantized.

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI: 

llama-cli --hf-repo Triangle104/Magistral-Small-2506-Q3_K_L-GGUF --hf-file magistral-small-2506-q3_k_l.gguf -p "The meaning to life and the universe is"

Server: 

llama-server --hf-repo Triangle104/Magistral-Small-2506-Q3_K_L-GGUF --hf-file magistral-small-2506-q3_k_l.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Magistral-Small-2506-Q3_K_L-GGUF --hf-file magistral-small-2506-q3_k_l.gguf -p "The meaning to life and the universe is"

or 

./llama-server --hf-repo Triangle104/Magistral-Small-2506-Q3_K_L-GGUF --hf-file magistral-small-2506-q3_k_l.gguf -c 2048
Downloads last month
1
GGUF
Model size
24B params
Architecture
llama
Hardware compatibility
Log In to view the estimation

3-bit

Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Triangle104/Magistral-Small-2506-Q3_K_L-GGUF

Base model

mistralai/Mistral-Small-3.1-24B-Base-2503
Finetuned
mistralai/Mistral-Small-3.1-24B-Instruct-2503
Finetuned
mistralai/Magistral-Small-2506
Finetuned
unsloth/Magistral-Small-2506
Quantized
(10)
this model

Collection including Triangle104/Magistral-Small-2506-Q3_K_L-GGUF