Instructions to use btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ")
model = AutoModelForMultimodalLM.from_pretrained("btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ

SGLang

How to use btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ with Docker Model Runner:
```
docker model run hf.co/btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ
```

Troublesome Tool Calling

by codes2808 - opened Jan 13

Discussion

codes2808

Jan 13

Howdy! I've got this deployed on 2x 7900xtx and I'm getting about 50tok/s. Which is faster than running on Ollama on a single card, so 👍

I'm running this vllm command on nightly, with the latest transformers 5 rc3 installed
# VLLM_DISABLED_KERNELS=ConchLinearKernel OMP_NUM_THREADS=48 vllm serve --tensor-parallel-size 2 btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ --quantization compressed-tensors --max-model-len 97632 --kv-cache-memory-bytes=4g --kv-cache-dtype=fp8_e4m3 --dtype=half --tool-call-parser mistral --enable-auto-tool-choice

I'm having a lot of trouble with tool calls though, I'm using https://github.com/adamwlarson/LLMToolCallingTester and running on the --quick test I'm getting 30% tool calls pass, while Ollama is able to pass at 100%.

This model on vllm :

python tool_tester_v2.py --api-base http://localhost:8000/v1 --api-key local --model "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ" --quick

============================================================
FINAL SCORE
============================================================
Scenario Success Rate: 100.0%
Tool Precision (LLM only): 16.7%
Tool Recall (LLM only): 100.0%
Tool F1 (LLM only): 28.6%
Parameter Accuracy (structural): 100.0%
Parameter Accuracy (LLM only): 100.0%
Parameter Accuracy (semantic): 16.7%
Parameter Accuracy (semantic, LLM only): 16.7%
Execution Success Rate: 100.0%
Execution Success Rate (LLM only): 100.0%
Weighted Complexity Score: 100.0%

OVERALL SCORE: 30.7% (F)

Ollama's `devstral-small-2:latest` :

python tool_tester_v2.py --api-base http://ollama/v1 --api-key local --model "devstral-small-2:latest" --quick

============================================================
FINAL SCORE
============================================================
Scenario Success Rate: 100.0%
Tool Precision (LLM only): 100.0%
Tool Recall (LLM only): 100.0%
Tool F1 (LLM only): 100.0%
Parameter Accuracy (structural): 100.0%
Parameter Accuracy (LLM only): 100.0%
Parameter Accuracy (semantic): 100.0%
Parameter Accuracy (semantic, LLM only): 100.0%
Execution Success Rate: 100.0%
Execution Success Rate (LLM only): 100.0%
Weighted Complexity Score: 100.0%

OVERALL SCORE: 100.0% (A+)

Are you getting similar performance on your 4x MI100?

What can I do to try to get the tool calling performance up? Changing the template makes a marginal amount of difference, changing the tool parser to match the template also makes a marginal amount of difference (a couple percent, nothing wild).

I see your chat template is the same as the stock model, but the stock model doesn't load on my cards for lack of memory so I can't test it.

codes2808

Jan 13

•

edited Jan 13

I am getting a ton of these errors in the logs as well, something is not working. I don't feel like mistral_tool_parser should be throwing tracebacks like this...

(APIServer pid=20) INFO:     xxx.xx.xx.138:65374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=20) INFO 01-13 18:06:08 [mistral_tool_parser.py:85] Non-Mistral tokenizer detected when using a Mistral model...
(APIServer pid=20) INFO 01-13 18:06:09 [mistral_tool_parser.py:85] Non-Mistral tokenizer detected when using a Mistral model...
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] Error in extracting tool call from response: {e}
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] Traceback (most recent call last):
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]   File "/usr/local/lib/python3.12/dist-packages/vllm/tool_parsers/mistral_tool_parser.py", line 186, in extract_tool_calls
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]     tool_calls = json.loads(stringified_tool_calls)
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]   File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]     return _default_decoder.decode(s)
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]   File "/usr/lib/python3.12/json/decoder.py", line 338, in decode
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]   File "/usr/lib/python3.12/json/decoder.py", line 356, in raw_decode
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]     raise JSONDecodeError("Expecting value", s, err.value) from None
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] During handling of the above exception, another exception occurred:
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] Traceback (most recent call last):
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]   File "/usr/local/lib/python3.12/dist-packages/vllm/tool_parsers/mistral_tool_parser.py", line 193, in extract_tool_calls
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]     raw_tool_call = self.tool_call_regex.findall(
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=20) ERROR 01-13 18:06:09 [mistral_tool_parser.py:198] IndexError: list index out of range
(APIServer pid=20) INFO:     xxx.xx.xx.138:65374 - "POST /v1/chat/completions HTTP/1.1" 200 OK

codes2808

Jan 13

•

edited Jan 13

I produced this script (based on the test script from mistral) that can reproduce the issue:

pip install requests
pip install argparse
pip install huggingface_hub

import requests
import json
from huggingface_hub import hf_hub_download
import pprint
import argparse

parser = argparse.ArgumentParser(description='Chat with Devstral Small 2 model')
parser.add_argument('--url', type=str, required=True, help='URL of the server endpoint')
parser.add_argument('--model', type=str, required=True, help='which model to load for request')
args = parser.parse_args()

url = args.url
model = args.model

# url = "http://flux:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

# model = "mistralai/Devstral-Small-2-24B-Instruct-2512"

def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    return system_prompt

SYSTEM_PROMPT = load_system_prompt("mistralai/Devstral-Small-2-24B-Instruct-2512", "CHAT_SYSTEM_PROMPT.txt")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "clone my gitlab.com repository at /codes2808/scripts.git to the ~/Projects directory",
            },
        ],
    },
]

data = {"model": model, "messages": messages, "temperature": 0.15}

# Devstral Small 2 supports tool calling. If you want to use tools, follow this:
tools = [ # Define tools for vLLM
    {
        "type": "function",
        "function": {
            "name": "git_clone",
            "description": "Clone a git repository",
            "parameters": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The url of the git repository",
                    },
                },
                "required": ["url"],
            },
        },
    }
]
data = {"model": model, "messages": messages, "temperature": 0.15, "tools": tools} # Pass tools to payload.

response = requests.post(url, headers=headers, data=json.dumps(data))
#:print(response.json()["choices"][0]["message"]["content"])

pprint.pprint(response.json())

with ollama I get the following output, which seems correct:

$ python devstral-small-2-test.py --url http://ollama/v1/chat/completions --model "devstral-small-2:latest"
{'choices': [{'finish_reason': 'tool_calls',
              'index': 0,
              'message': {'content': '',
                          'role': 'assistant',
                          'tool_calls': [{'function': {'arguments': '{"url":"https://gitlab.com/codes2808/scripts.git"}',
                                                       'name': 'git_clone'},
                                          'id': 'call_dg522iq1',
                                          'index': 0,
                                          'type': 'function'}]}}],
 'created': 1768340033,
 'id': 'chatcmpl-663',
 'model': 'devstral-small-2:latest',
 'object': 'chat.completion',
 'system_fingerprint': 'fp_ollama',
 'usage': {'completion_tokens': 24, 'prompt_tokens': 603, 'total_tokens': 627}}

with vllm I am getting this output and the error on the console from above:

$  python devstral-small-2-test.py --url http://vllm:8000/v1/chat/completions --model "btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ"
{'choices': [{'finish_reason': 'stop',
              'index': 0,
              'logprobs': None,
              'message': {'annotations': None,
                          'audio': None,
                          'content': '[TOOL_CALLS]git_clone[ARGS]{"url": '
                                     '"gitlab.com/codes2808/scripts.git"}',
                          'function_call': None,
                          'reasoning': None,
                          'reasoning_content': None,
                          'refusal': None,
                          'role': 'assistant',
                          'tool_calls': []},
              'stop_reason': None,
              'token_ids': None}],
 'created': 1768340158,
 'id': 'chatcmpl-9308d611b0cdd5fd',
 'kv_transfer_params': None,
 'model': 'btbtyler09/Devstral-Small-2-24B-Instruct-INT4-INT8-Mixed-GPTQ',
 'object': 'chat.completion',
 'prompt_logprobs': None,
 'prompt_token_ids': None,
 'service_tier': None,
 'system_fingerprint': None,
 'usage': {'completion_tokens': 22,
           'prompt_tokens': 619,
           'prompt_tokens_details': None,
           'total_tokens': 641}}

btbtyler09

Owner Jan 13

I will give this a try, and compare to the original model. I haven't had much free time to test this model since I quantized it. I use the original version at work without issues, and was intending to swap it out for this one. I never got around to it though, so I haven't thoroughly tested tool calling. I'm still dialing in my procedures for quantization and mistral models, so I might have made some errors that need to be fixed.

codes2808

Jan 13

Thanks,

I've been sluthing today, and it appears that the warning about the tokenizer might actually be a real clue, but I haven't gotten this model to load with the base model's tokenizer, but it appears that without that loaded up the mistral_tool_parser() is considering this a model before v11, and so it's looking for tool calls that are in the wrong format.

this function needs to return Falsefor the right format to get used.
https://github.com/vllm-project/vllm/blob/2a60ac91d0f5c24cdb2863b178d2f5405fae50b8/vllm/tool_parsers/mistral_tool_parser.py#L68C1-L71C6

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Troublesome Tool Calling

This model on vllm :

Ollama's devstral-small-2:latest :

Ollama's `devstral-small-2:latest` :