Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,17 @@
+MINIMAX COMMUNITY LICENSE
+Copyright (c) 2026 MiniMax
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software for non-commercial purposes, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or provide copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+2. If the Software (or any derivative works thereof) is used for any Commercial Use for your products or services: 
+  1. you shall prominently display “Built with MiniMax M3” on a related website, user interface, blogpost, about page or product documentation. 
+  2. you shall obtain a separate, prior written authorization from MiniMax by contacting api@minimax.io with the subject line “M3 licensing - authorization request”, if such products and services generate more than 20 million US dollars (or equivalent in other currencies) in yearly revenue; otherwise, you only need to send a one-time notice to api@minimax.io with the subject “M3 licensing — notice”.
+3. “Commercial Use” means any use of the Software or any derivative work thereof that is primarily intended for commercial advantage or monetary compensation, which includes, without limitation: (i) offering products or services to third parties for a fee, which utilize, incorporate, or rely on the Software or its derivatives, (ii) the commercial use of APIs provided by or for the Software or its derivatives, including to support or enable commercial products, services, or operations, whether in a cloud-based, hosted, or other similar environment, and (iii) the deployment or provision of the Software or its derivatives that have been subjected to post-training, fine-tuning, instruction-tuning, or any other form of modification, for any commercial purpose.
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+Appendix: Prohibited Uses 
+You agree you will not use, or allow others to use, the Software or any derivatives of the Software to:
+1. Generate or disseminate content prohibited by applicable laws or regulations.
+2. Assist with, engage in or otherwise support any military purpose.
+3. Exploit, harm, or attempt to exploit or harm minors.
+4. Generate or disseminate false or misleading information with the intent to cause harm.
+5. Promote discrimination, hate speech, or harmful behavior against individuals or groups based on race or ethnic origin, religion, disability, age, nationality and national origin, veteran status, sexual orientation, gender or gender identity, caste, immigration status, or any other characteristic that is associated with systemic discrimination or marginalization.

README.md

@@ -0,0 +1,70 @@
+---
+license: other
+license_name: minimax-community
+license_link: LICENSE
+base_model: MiniMaxAI/MiniMax-M3
+base_model_relation: quantized
+pipeline_tag: text-generation
+library_name: mlx
+tags:
+- mlx
+- moe
+- minimax
+- minimax-m3
+- text-generation
+---
+
+# MiniMax-M3-MLX-4bit
+
+**Built with MiniMax M3.**
+
+This is an **MLX** (Apple Silicon) conversion of
+[MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3), quantized to
+**4-bit (balanced default)**.
+
+It is a **text-only** extraction of the M3 backbone (the vision tower, multimodal
+projector and multi-token-prediction heads are not included). The model is a
+~427B-parameter Mixture-of-Experts (128 experts, top-4, + 1 shared expert; first
+3 layers dense), with per-head QK-norm, partial RoPE, Gemma-style RMSNorm and the
+SwiGLU-OAI activation.
+
+## Attention / context note
+
+MiniMax Sparse Attention (MSA) is implemented here as **full causal attention**.
+This is numerically exact for sequences up to 2048 tokens (MSA selects every key
+block at that length) and is the dense, un-approximated attention that MSA
+approximates beyond it — so quality is preserved, at the cost of MSA's
+long-context speed/memory savings.
+
+## Use with mlx-lm
+
+```bash
+pip install mlx-lm
+```
+
+This build requires the `minimax_m3` model class
+([`mlx_lm/models/minimax_m3.py`](https://huggingface.co/pipenetwork/MiniMax-M3-MLX-4bit/blob/main/minimax_m3.py),
+included in this repo — copy it into your `mlx_lm/models/` directory).
+
+```python
+from mlx_lm import load, generate
+
+model, tokenizer = load("pipenetwork/MiniMax-M3-MLX-4bit")
+prompt = tokenizer.apply_chat_template(
+    [{"role": "user", "content": "Explain Mixture-of-Experts in one paragraph."}],
+    add_generation_prompt=True,
+)
+print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))
+```
+
+## License
+
+Released under the **MiniMax Community License** (see `LICENSE`). Use is
+**non-commercial** by default; commercial use requires displaying
+"Built with MiniMax M3" and may require prior authorization from MiniMax — see the
+license text for details.
+
+## Provenance
+
+Converted from the BF16 checkpoint with `mlx-lm` quantization. Quantization
+config: `{"group_size": 64, "bits": 4, "mode": "affine", "model.layers.3.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.4.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.5.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.6.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.7.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.8.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.9.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.10.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.11.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.12.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.13.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.14.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.15.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.16.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.17.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.18.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.19.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.20.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.21.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.22.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.23.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.24.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.25.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.26.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.27.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.28.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.29.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.30.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.31.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.32.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.33.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.34.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.35.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.36.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.37.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.38.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.39.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.40.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.41.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.42.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.43.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.44.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.45.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.46.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.47.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.48.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.49.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.50.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.51.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.52.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.53.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.54.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.55.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.56.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.57.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.58.block_sparse_moe.gate": {"group_size": 64, "bits": 8}, "model.layers.59.block_sparse_moe.gate": {"group_size": 64, "bits": 8}}`.

chat_template.jinja

@@ -0,0 +1,247 @@
+{# ---------- special token variables ---------- #}
+{%- set ns_token               = ']<]minimax[>['                  -%}
+{%- set bod_token              = ']~!b['                          -%}
+{%- set bos_token              = ']~b]'                           -%}
+{%- set eos_token              = '[e~['                           -%}
+{%- set toolcall_begin_token   = ns_token ~ '<tool_call>'         -%}
+{%- set toolcall_end_token     = ns_token ~ '</tool_call>'        -%}
+{%- set think_begin_token      = '<mm:think>'                     -%}
+{%- set think_end_token        = '</mm:think>'                    -%}
+{%- set image_token            = ']<]image[>['                    -%}
+{%- set video_token            = ']<]video[>['                    -%}
+{#- Thinking mode: "enabled" / "disabled" / "adaptive" / not defined -#}
+{#- Recursive XML renderer for tool_call arguments ======================== -#}
+{#- None values are intentionally skipped in mapping iteration so that
+    `<key>null</key>` (which would round-trip to the literal string "null")
+    never appears in the rendered tool_call. The convention is: omit the
+    field entirely. The top-level `_args` loop applies the same rule.
+    The `val is none` branch below is a safety net only — upstream cleaning
+    (drop_none_in_tool_arguments) should ensure no None ever reaches here. -#}
+{%- macro to_xml(val, ns) -%}
+{%- if val is mapping -%}
+{%- for k, v in val.items() if v is not none -%}
+{{ ns }}<{{ k }}>{{ to_xml(v, ns) }}{{ ns }}</{{ k }}>
+{%- endfor -%}
+{%- elif val is iterable and val is not string -%}
+{%- for item in val -%}
+{{ ns }}<item>{{ to_xml(item, ns) }}{{ ns }}</item>
+{%- endfor -%}
+{%- elif val is none -%}
+{#- Should be unreachable when upstream cleaning is applied. -#}
+{%- elif val is boolean -%}
+{{ val | tojson }}
+{%- else -%}
+{{ val }}
+{%- endif -%}
+{%- endmacro -%}
+{#- Tool Rendering Functions ============================================== -#}
+{%- macro render_tool_namespace(namespace_name, tool_list) -%}
+{%- for tool in tool_list -%}
+<tool>{{ tool.function | tojson(ensure_ascii=False) }}</tool>
+{% endfor -%}
+{%- endmacro -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{ content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is mapping and item.type == 'image' -%}
+                {{- image_token }}
+            {%- elif item is mapping and item.type == 'video' -%}
+                {{- video_token}}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- elif content is none -%}
+        {{- '' }}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{#- System Message Construction ============================================ -#}
+{%- macro build_system_message(system_message) -%}
+    {%- if system_message and system_message.content -%}
+        {{- visible_text(system_message.content) }}
+    {%- else -%}
+        {{- 'Your model version is MiniMax-M3, developed by MiniMax. Knowledge cutoff: January 2026. Founded in early 2022, MiniMax is a global AI foundation model company committed to advancing the frontiers of AI towards AGI.' }}
+    {%- endif -%}
+
+    {#- Thinking mode instructions -#}
+    {{- '\n\n<thinking_instructions>\n' }}
+    {{- 'You have a thinking capability that allows you to reason step by step before responding. When thinking is enabled, wrap your reasoning in ' ~ think_begin_token ~ think_end_token ~ ' tags before your response. When thinking is disabled, begin your response directly after the ' ~ think_end_token ~ ' prefix. When thinking is adaptive, decide on your own whether to think for the current turn.\n' }}
+    {%- if thinking_mode is defined -%}
+        {%- if thinking_mode == "enabled" -%}
+            {{- 'Current thinking mode: enabled. You MUST think step by step before every response, including after receiving function/tool results.\n' }}
+        {%- elif thinking_mode == "disabled" -%}
+            {{- 'Current thinking mode: disabled. Do not output any thinking process.\n' }}
+        {%- elif thinking_mode == "adaptive" -%}
+            {{- 'Current thinking mode: adaptive. You are encouraged to think for complex decision-making, multi-step reasoning, or when analyzing function/tool results.\n' }}
+        {%- endif -%}
+    {%- else -%}
+        {{- 'Current thinking mode: adaptive. You are encouraged to think for complex decision-making, multi-step reasoning, or when analyzing function/tool results.\n' }}
+    {%- endif -%}
+    {{- '</thinking_instructions>' }}
+{%- endmacro -%}
+{%- macro build_developer_message(developer_message) -%}
+    {%- if developer_message and developer_message.content -%}
+        {{- visible_text(developer_message.content) }}
+    {%- else -%}
+        {%- if model_identity is not defined -%}
+            {%- set model_identity = "You are a helpful assistant." -%}
+        {%- endif -%}
+        {{- model_identity }}
+    {%- endif -%}
+{%- endmacro -%}
+{#- Main Template Logic ================================================= -#}
+{#- Role mapping: root -> system sp (high priority), system/developer -> developer sp (low priority) -#}
+{%- set system_message = none -%}
+{%- set developer_message = none -%}
+{%- set conversation_messages = messages -%}
+{%- if messages and messages[0].role == "root" -%}
+    {%- set system_message = messages[0] -%}
+    {%- set conversation_messages = messages[1:] -%}
+    {%- if conversation_messages and conversation_messages[0].role in ["system", "developer"] -%}
+        {%- set developer_message = conversation_messages[0] -%}
+        {%- set conversation_messages = conversation_messages[1:] -%}
+    {%- endif -%}
+{%- elif messages and messages[0].role in ["system", "developer"] -%}
+    {%- set developer_message = messages[0] -%}
+    {%- set conversation_messages = messages[1:] -%}
+{%- endif -%}
+{#- Render system sp (higher priority, root role only) -#}
+{{- bod_token ~ bos_token ~ 'system' ~ '\n' }}
+{{- build_system_message(system_message) }}
+{{- eos_token ~ '\n' }}
+
+{#- Render developer sp (lower priority: system/developer role + tools) -#}
+{{- bos_token ~ 'developer' ~ '\n' }}
+{{- build_developer_message(developer_message) }}
+{%- if tools -%}
+    {{- '\n\n' ~ '# Tools' ~ '\n' ~ 'You may call one or more tools to assist with the user query.\nHere are the tools available in JSONSchema format:' ~ '\n' }}
+    {{- '\n' ~ '<tools>' ~ '\n' }}
+    {{- render_tool_namespace("functions", tools) }}
+    {{- '</tools>' ~ '\n\n' }}
+    {{- 'To call tools, wrap all invocations in a single ' ~ toolcall_begin_token ~ toolcall_end_token ~ ' block. Parameter values containing nested objects or arrays are recursively expanded into XML elements. Example:\n' }}
+    {{- '\n' ~ toolcall_begin_token ~ '\n' }}
+    {{- ns_token + '<invoke name="tool-name-1">' }}
+    {{- ns_token + '<param-1>value-1' + ns_token + '</param-1>' }}
+    {{- ns_token + '<param-2>' }}
+    {{- ns_token + '<item>' }}
+    {{- ns_token + '<key-a>val-a' + ns_token + '</key-a>' }}
+    {{- ns_token + '<key-b>val-b' + ns_token + '</key-b>' }}
+    {{- ns_token + '</item>' }}
+    {{- ns_token + '</param-2>' }}
+    {{- ns_token + '</invoke>\n' }}
+    {{- ns_token + '<invoke name="tool-name-2">' }}
+    {{- ns_token + '<param-1>value-1' + ns_token + '</param-1>' }}
+    {{- ns_token + '</invoke>\n' }}
+    {{- toolcall_end_token }}
+{%- endif -%}
+{{- eos_token ~ '\n' }}
+
+{#- Render messages -#}
+{%- set last_tool_call = namespace(name=none) -%}
+{%- for message in conversation_messages -%}
+    {%- if message.role == 'assistant' -%}
+        {{- bos_token ~ 'ai' ~ '\n' }}
+
+        {%- set reasoning_content = '' %}
+        {%- set content = visible_text(message.content) %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if think_end_token in content %}
+                {%- set reasoning_content = content.split(think_end_token)[0].strip('\n').split(think_begin_token)[-1].strip('\n') %}
+                {%- set content = content.split(think_end_token)[-1].strip('\n') %}
+            {%- endif %}
+        {%- endif %}
+
+        {%- if reasoning_content -%}
+            {#- Render thinking for every assistant turn (all-turn visible) -#}
+            {{- think_begin_token ~ reasoning_content ~ think_end_token }}
+        {%- else -%}
+            {#- No thinking rendered → prefix with think_end_token -#}
+            {{- think_end_token }}
+        {%- endif -%}
+
+        {%- if content -%}
+            {{- content }}
+        {%- endif -%}
+        {%- if message.tool_calls -%}
+            {{- toolcall_begin_token ~ '\n' }}
+
+            {%- for tool_call in message.tool_calls -%}
+                {%- if tool_call.function -%}
+                    {%- set tool_call = tool_call.function -%}
+                {%- endif -%}
+{{- ns_token + '<invoke name="' + tool_call.name + '">' }}
+{%- set _args = tool_call.arguments -%}
+{%- for k, v in _args.items() if v is not none %}
+{{- ns_token + '<' + k + '>' -}}
+{{- to_xml(v, ns_token) -}}
+{{- ns_token + '</' + k + '>' }}
+{%- endfor -%}
+{{- ns_token + '</invoke>' ~ '\n' }}
+            {%- endfor -%}
+
+            {{- toolcall_end_token }}
+            {%- if message.tool_calls[-1].function -%}
+                {%- set last_tool_call.name = message.tool_calls[-1].function.name -%}
+            {%- else -%}
+                {%- set last_tool_call.name = message.tool_calls[-1].name -%}
+            {%- endif -%}
+        {%- else -%}
+            {%- set last_tool_call.name = none -%}
+        {%- endif -%}
+        {{- eos_token ~ '\n' }}
+
+    {%- elif message.role == 'tool' -%}
+        {%- if last_tool_call.name is none -%}
+            {{- raise_exception("Message has tool role, but there was no previous assistant message with a tool call!") }}
+        {%- endif -%}
+        {%- if loop.first or (conversation_messages[loop.index0 - 1].role != 'tool') -%}
+            {{- bos_token ~ 'tool' }}
+        {%- endif -%}
+        {{- '\n<response>' }}
+        {%- if message.content is string -%}
+            {{- message.content }}
+        {%- else -%}
+            {%- for tr in message.content -%}
+                {%- if tr is mapping and tr.type is defined and tr.type == 'image' -%}
+                    {{- image_token }}
+                {%- elif tr is mapping and tr.type is defined and tr.type == 'video' -%}
+                    {{- video_token }}
+                {%- else -%}
+                    {{- tr.output if tr.output is defined else (tr.text if tr.type == 'text' and tr.text is defined else tr) }}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+        {{- '</response>' }}
+        {%- if loop.last or (conversation_messages[loop.index0 + 1].role != 'tool') -%}
+            {{- eos_token ~ '\n' -}}
+        {%- endif -%}
+
+    {%- elif message.role == 'user' -%}
+        {{- bos_token ~ 'user' ~ '\n' }}
+        {{- visible_text(message.content) }}
+        {{- eos_token ~ '\n' }}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- Generation prompt -#}
+{%- if add_generation_prompt -%}
+{{- bos_token ~ 'ai' ~ '\n' }}
+{%- if thinking_mode is defined and thinking_mode == "disabled" -%}
+    {{- think_end_token }}
+{%- elif thinking_mode is defined and thinking_mode == "adaptive" -%}
+    {#- adaptive: no prefix, let model decide -#}
+{%- elif thinking_mode is defined and thinking_mode == "enabled" -%}
+    {#- enabled or not defined: default to think -#}
+    {{- think_begin_token }}
+{%- else -%}
+    {#- adaptive: no prefix, let model decide -#}
+{%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,551 @@
+{
+    "dense_intermediate_size": 12288,
+    "eos_token_id": 200020,
+    "head_dim": 128,
+    "hidden_size": 6144,
+    "intermediate_size": 3072,
+    "max_position_embeddings": 1048576,
+    "mlp_layer_types": [
+        "dense",
+        "dense",
+        "dense",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse",
+        "sparse"
+    ],
+    "model_type": "minimax_m3",
+    "num_attention_heads": 64,
+    "num_experts_per_tok": 4,
+    "num_hidden_layers": 60,
+    "num_key_value_heads": 4,
+    "num_local_experts": 128,
+    "quantization": {
+        "group_size": 64,
+        "bits": 4,
+        "mode": "affine",
+        "model.layers.3.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.4.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.5.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.6.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.7.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.8.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.9.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.10.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.11.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.12.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.13.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.14.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.15.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.16.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.17.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.18.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.19.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.20.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.21.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.22.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.23.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.24.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.25.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.26.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.27.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.28.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.29.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.30.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.31.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.32.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.33.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.34.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.35.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.36.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.37.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.38.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.39.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.40.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.41.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.42.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.43.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.44.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.45.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.46.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.47.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.48.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.49.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.50.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.51.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.52.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.53.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.54.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.55.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.56.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.57.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.58.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.59.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        }
+    },
+    "quantization_config": {
+        "group_size": 64,
+        "bits": 4,
+        "mode": "affine",
+        "model.layers.3.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.4.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.5.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.6.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.7.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.8.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.9.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.10.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.11.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.12.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.13.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.14.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.15.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.16.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.17.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.18.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.19.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.20.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.21.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.22.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.23.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.24.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.25.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.26.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.27.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.28.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.29.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.30.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.31.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.32.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.33.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.34.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.35.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.36.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.37.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.38.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.39.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.40.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.41.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.42.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.43.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.44.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.45.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.46.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.47.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.48.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.49.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.50.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.51.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.52.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.53.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.54.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.55.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.56.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.57.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.58.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.layers.59.block_sparse_moe.gate": {
+            "group_size": 64,
+            "bits": 8
+        }
+    },
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 5000000,
+    "rotary_dim": 64,
+    "routed_scaling_factor": 2.0,
+    "shared_intermediate_size": 3072,
+    "swiglu_alpha": 1.702,
+    "swiglu_limit": 7.0,
+    "tie_word_embeddings": false,
+    "vocab_size": 200064
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "bos_token_id": 200019,
+  "do_sample": true,
+  "eos_token_id": 200020,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "4.46.1"
+}

minimax_m3.py

@@ -0,0 +1,323 @@
+# Copyright © 2026
+# MiniMax-M3 text backbone (text-only LLM extraction of MiniMaxM3VL).
+#
+# M3 extends MiniMax-M2 with: Gemma-style RMSNorm (scale by 1+w, fp32),
+# per-head QK-norm, partial RoPE (rotary_dim < head_dim), SwiGLU-OAI activation
+# (clamped gate/up with an (up+1) term), a shared expert + routed_scaling_factor
+# in the MoE, and the first few layers being dense MLPs instead of MoE.
+#
+# MiniMax Sparse Attention (MSA) is implemented here as full causal attention.
+# For sequences up to index_topk_blocks * index_block_size (= 2048) tokens the
+# MSA indexer selects *every* key block, so full attention is numerically exact;
+# beyond that it is the dense (un-approximated) attention MSA approximates, so
+# quality is preserved at the cost of long-context speed/memory.
+
+from dataclasses import dataclass, field
+from typing import Any, List, Optional
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .base import BaseModelArgs, create_attention_mask, scaled_dot_product_attention
+from .switch_layers import SwitchGLU
+
+
+@dataclass
+class ModelArgs(BaseModelArgs):
+    model_type: str
+    hidden_size: int
+    intermediate_size: int
+    dense_intermediate_size: int
+    shared_intermediate_size: int
+    num_attention_heads: int
+    num_key_value_heads: int
+    num_hidden_layers: int
+    num_local_experts: int
+    num_experts_per_tok: int
+    rms_norm_eps: float
+    rope_theta: float
+    rotary_dim: int
+    vocab_size: int
+    head_dim: int = 128
+    max_position_embeddings: int = 1048576
+    routed_scaling_factor: float = 2.0
+    swiglu_alpha: float = 1.702
+    swiglu_limit: float = 7.0
+    scoring_func: str = "sigmoid"
+    use_qk_norm: bool = True
+    tie_word_embeddings: bool = False
+    # Per-layer MLP dispatch: "sparse" -> MoE block, "dense" -> dense MLP.
+    mlp_layer_types: Optional[List[str]] = None
+
+
+class GemmaRMSNorm(nn.Module):
+    """Gemma-style RMSNorm: normalize in fp32 and scale by ``weight + 1``."""
+
+    def __init__(self, dims: int, eps: float = 1e-6):
+        super().__init__()
+        self.weight = mx.zeros((dims,))
+        self.eps = eps
+
+    def _extra_repr(self):
+        return f"{self.weight.shape[0]}, eps={self.eps}"
+
+    def __call__(self, x):
+        ot = x.dtype
+        x = x.astype(mx.float32)
+        x = x * mx.rsqrt(x.square().mean(-1, keepdims=True) + self.eps)
+        return (x * (1.0 + self.weight.astype(mx.float32))).astype(ot)
+
+
+def swiglu_oai(x_gate, x_up, alpha: float, limit: float):
+    """GPT-OSS / MiniMax-M3 clamped SwiGLU: (clamp(up)+1) * gate*sigmoid(alpha*gate)."""
+    gate = mx.minimum(x_gate, limit)
+    up = mx.clip(x_up, -limit, limit)
+    return (up + 1.0) * (gate * mx.sigmoid(gate * alpha))
+
+
+class SwiGLUOAI(nn.Module):
+    """Activation callable for SwitchGLU: receives (x_up, x_gate)."""
+
+    def __init__(self, alpha: float, limit: float):
+        super().__init__()
+        self.alpha = alpha
+        self.limit = limit
+
+    def __call__(self, x_up, x_gate):
+        return swiglu_oai(x_gate, x_up, self.alpha, self.limit)
+
+
+class MiniMaxM3MLP(nn.Module):
+    """Dense SwiGLU-OAI MLP (used by the first dense layers and the shared expert)."""
+
+    def __init__(self, args: ModelArgs, intermediate_size: int):
+        super().__init__()
+        self.alpha = args.swiglu_alpha
+        self.limit = args.swiglu_limit
+        self.gate_proj = nn.Linear(args.hidden_size, intermediate_size, bias=False)
+        self.up_proj = nn.Linear(args.hidden_size, intermediate_size, bias=False)
+        self.down_proj = nn.Linear(intermediate_size, args.hidden_size, bias=False)
+
+    def __call__(self, x):
+        return self.down_proj(
+            swiglu_oai(self.gate_proj(x), self.up_proj(x), self.alpha, self.limit)
+        )
+
+
+class MiniMaxM3Attention(nn.Module):
+    def __init__(self, args: ModelArgs):
+        super().__init__()
+        self.num_attention_heads = args.num_attention_heads
+        self.num_key_value_heads = args.num_key_value_heads
+        self.head_dim = head_dim = args.head_dim
+        self.scale = head_dim**-0.5
+
+        self.q_proj = nn.Linear(
+            args.hidden_size, self.num_attention_heads * head_dim, bias=False
+        )
+        self.k_proj = nn.Linear(
+            args.hidden_size, self.num_key_value_heads * head_dim, bias=False
+        )
+        self.v_proj = nn.Linear(
+            args.hidden_size, self.num_key_value_heads * head_dim, bias=False
+        )
+        self.o_proj = nn.Linear(
+            self.num_attention_heads * head_dim, args.hidden_size, bias=False
+        )
+
+        # M3 uses per-head Gemma QK-norm over the head dimension.
+        self.q_norm = GemmaRMSNorm(head_dim, eps=args.rms_norm_eps)
+        self.k_norm = GemmaRMSNorm(head_dim, eps=args.rms_norm_eps)
+
+        self.rope = nn.RoPE(args.rotary_dim, traditional=False, base=args.rope_theta)
+
+    def __call__(self, x, mask=None, cache=None):
+        B, L, _ = x.shape
+
+        queries = self.q_proj(x).reshape(B, L, self.num_attention_heads, self.head_dim)
+        keys = self.k_proj(x).reshape(B, L, self.num_key_value_heads, self.head_dim)
+        values = self.v_proj(x).reshape(B, L, self.num_key_value_heads, self.head_dim)
+
+        # Per-head QK-norm over the head dim, before transpose / RoPE.
+        queries = self.q_norm(queries).transpose(0, 2, 1, 3)
+        keys = self.k_norm(keys).transpose(0, 2, 1, 3)
+        values = values.transpose(0, 2, 1, 3)
+
+        if cache is not None:
+            queries = self.rope(queries, offset=cache.offset)
+            keys = self.rope(keys, offset=cache.offset)
+            keys, values = cache.update_and_fetch(keys, values)
+        else:
+            queries = self.rope(queries)
+            keys = self.rope(keys)
+
+        output = scaled_dot_product_attention(
+            queries, keys, values, cache=cache, scale=self.scale, mask=mask
+        )
+        output = output.transpose(0, 2, 1, 3).reshape(B, L, -1)
+        return self.o_proj(output)
+
+
+class MiniMaxM3SparseMoeBlock(nn.Module):
+    def __init__(self, args: ModelArgs):
+        super().__init__()
+        self.num_experts_per_tok = args.num_experts_per_tok
+        self.routed_scaling_factor = args.routed_scaling_factor
+
+        self.gate = nn.Linear(args.hidden_size, args.num_local_experts, bias=False)
+        self.e_score_correction_bias = mx.zeros((args.num_local_experts,))
+        self.switch_mlp = SwitchGLU(
+            args.hidden_size,
+            args.intermediate_size,
+            args.num_local_experts,
+            activation=SwiGLUOAI(args.swiglu_alpha, args.swiglu_limit),
+        )
+        self.shared_experts = MiniMaxM3MLP(args, args.shared_intermediate_size)
+
+    def __call__(self, x):
+        gates = self.gate(x.astype(mx.float32))
+        scores = mx.sigmoid(gates)
+        orig_scores = scores
+        scores = scores + self.e_score_correction_bias
+
+        k = self.num_experts_per_tok
+        inds = mx.argpartition(-scores, kth=k - 1, axis=-1)[..., :k]
+        weights = mx.take_along_axis(orig_scores, inds, axis=-1)
+        weights = weights / (mx.sum(weights, axis=-1, keepdims=True) + 1e-20)
+        weights = (weights * self.routed_scaling_factor).astype(x.dtype)
+
+        y = self.switch_mlp(x, inds)
+        y = (y * weights[..., None]).sum(axis=-2)
+        return y + self.shared_experts(x)
+
+
+class MiniMaxM3DecoderLayer(nn.Module):
+    def __init__(self, args: ModelArgs, layer_idx: int):
+        super().__init__()
+        self.self_attn = MiniMaxM3Attention(args)
+        self.is_sparse = (args.mlp_layer_types or ["sparse"] * args.num_hidden_layers)[
+            layer_idx
+        ] == "sparse"
+        if self.is_sparse:
+            self.block_sparse_moe = MiniMaxM3SparseMoeBlock(args)
+        else:
+            self.mlp = MiniMaxM3MLP(args, args.dense_intermediate_size)
+        self.input_layernorm = GemmaRMSNorm(args.hidden_size, eps=args.rms_norm_eps)
+        self.post_attention_layernorm = GemmaRMSNorm(
+            args.hidden_size, eps=args.rms_norm_eps
+        )
+
+    def __call__(self, x, mask=None, cache=None):
+        r = x + self.self_attn(self.input_layernorm(x), mask, cache)
+        mlp = self.block_sparse_moe if self.is_sparse else self.mlp
+        return r + mlp(self.post_attention_layernorm(r))
+
+
+class MiniMaxM3Model(nn.Module):
+    def __init__(self, args: ModelArgs):
+        super().__init__()
+        self.embed_tokens = nn.Embedding(args.vocab_size, args.hidden_size)
+        self.layers = [
+            MiniMaxM3DecoderLayer(args, i) for i in range(args.num_hidden_layers)
+        ]
+        self.norm = GemmaRMSNorm(args.hidden_size, eps=args.rms_norm_eps)
+
+    def __call__(self, inputs, mask=None, cache=None):
+        h = self.embed_tokens(inputs)
+        if cache is None:
+            cache = [None] * len(self.layers)
+        if mask is None:
+            mask = create_attention_mask(h, cache[0])
+        for layer, c in zip(self.layers, cache):
+            h = layer(h, mask, c)
+        return self.norm(h)
+
+
+class Model(nn.Module):
+    def __init__(self, args: ModelArgs):
+        super().__init__()
+        self.args = args
+        self.model_type = args.model_type
+        self.model = MiniMaxM3Model(args)
+        if not args.tie_word_embeddings:
+            self.lm_head = nn.Linear(args.hidden_size, args.vocab_size, bias=False)
+
+    def __call__(self, inputs, mask=None, cache=None):
+        out = self.model(inputs, mask, cache)
+        if self.args.tie_word_embeddings:
+            return self.model.embed_tokens.as_linear(out)
+        return self.lm_head(out)
+
+    def sanitize(self, weights):
+        skip_prefixes = (
+            "vision_tower",
+            "multi_modal_projector",
+            "patch_merge_mlp",
+            "model.vision_tower",
+            "model.multi_modal_projector",
+        )
+
+        def keep(k):
+            if k.startswith(skip_prefixes):
+                return False
+            if ".self_attn.index_" in k:  # MSA lightning indexer — dropped
+                return False
+            if ".mtp." in k or k.startswith("mtp.") or "model.mtp" in k:
+                return False
+            return True
+
+        def rename(k):
+            if k.startswith("language_model.model."):
+                return "model." + k[len("language_model.model.") :]
+            if k.startswith("language_model.lm_head."):
+                return "lm_head." + k[len("language_model.lm_head.") :]
+            if k.startswith("language_model."):
+                return k[len("language_model.") :]
+            return k
+
+        renamed = {}
+        for k, v in weights.items():
+            if not keep(k):
+                continue
+            renamed[rename(k)] = v
+        weights = renamed
+
+        # Stack per-expert w1/w2/w3 into SwitchGLU's batched experts.
+        if (
+            "model.layers.0.block_sparse_moe.switch_mlp.gate_proj.weight"
+            not in weights
+        ):
+            mapping = {"w1": "gate_proj", "w2": "down_proj", "w3": "up_proj"}
+            for l in range(self.args.num_hidden_layers):
+                prefix = f"model.layers.{l}.block_sparse_moe"
+                if f"{prefix}.experts.0.w1.weight" not in weights:
+                    continue
+                for orig, new in mapping.items():
+                    stacked = mx.stack(
+                        [
+                            weights.pop(f"{prefix}.experts.{e}.{orig}.weight")
+                            for e in range(self.args.num_local_experts)
+                        ]
+                    )
+                    weights[f"{prefix}.switch_mlp.{new}.weight"] = stacked
+
+        return weights
+
+    @property
+    def layers(self):
+        return self.model.layers
+
+    @property
+    def cast_predicate(self):
+        # Keep the router correction bias in fp32.
+        return lambda k: "e_score_correction_bias" not in k
+
+    @property
+    def quant_predicate(self):
+        def predicate(path, _):
+            # Routers stay high-precision (small, sensitive to quantization).
+            if path.endswith("block_sparse_moe.gate"):
+                return {"group_size": 64, "bits": 8}
+            return True
+
+        return predicate

model-00002-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14d532266305ebb15e241966706fabac6611fdd318f3a1de7911d779f13fb4ec
+size 4338215150

model-00003-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07589e24a50538c74de9a6fc4cc48b86df544aac22990212fa6e3cfe050f7815
+size 4169791141

model-00005-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9badf36907d6a180fbd11b120724d4fec248efa9f3a725d813b4683fd8841f44
+size 4169791085

model-00006-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f82d96042372d50e32839294bc56d544c2c9e10cefe50b72f0eb9dfb99714371
+size 4169791125

model-00007-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c611bdf6835a683f8011b38b2e3542e18a89ad19b30a54215d92ba32436bf7d8
+size 4169791171

model-00009-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4709515ac8441574da5e56f7919ee8473f3b9a0b81b7f102d1b093e54fe803ee
+size 4169791127

model-00010-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e1e6d6f1c4d8482cfeca2995c741c399a8472dc6afee0abca0643d726c66767
+size 4169791181

model-00011-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:331581fbbe54bc442dbbf76e287479d872d764bd0000b2693132465cc5d8e724
+size 4169791131

model-00012-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0df71364b4612a2eb9541d08cc88c06376b5902b0f4c2223a8d754672c3377db
+size 4169791135

model-00014-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe8df3488deafe37b92a92181c57eae70065a2f56559ad6e578d39a282c03f55
+size 4169791171

model-00015-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28c27d50c3409a12ad26f40997542230ac8de0b209fea1fe49c62f650fdeae35
+size 4169791187

model-00017-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d3a5a6d40d5c250053887468d6b32de737b63c937ac9b0efc45076f0d419bc5
+size 4169791127

model-00018-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f44ddcea8eb8502314af8b929550001a7e645a6ff0ec47604feb5e015cef4061
+size 4169791143

model-00020-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f418ff7fb1e81cb46bbd56cc554d5b0f8ea54ce7278f114857a764e4cd0695d
+size 4169791139

model-00022-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e16fd1dcc30fdaab9c9a920268983f4d262401c4303d25d85759d3d6a3da1b04
+size 4169791203

model-00023-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35f5aee4b74886e647847c1f148a4e75ee60588aaaeec642a6badcc354ef32ea
+size 4169791197

model-00024-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f57bd21e76438f259ad40516df0a8f23caa3b4acba392e985129765cf6fbad6
+size 4169791179

model-00025-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ebffd2fa363c1c062b39c82d06ca1e0491200d3dbefc7a8e020aefbe096412a
+size 4169791149

model-00026-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90f45ca455bb08acb14d41b59d4f16eccd5e1b62cc496a734f214072cb5d8e04
+size 4169791149

model-00027-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca2c196cce644265f61d2d8ec8daa372e7e460a42c6ac2c5c855760b0d35bf38
+size 4169791141

model-00028-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:611ae33ba7cf532929f5869598b2d7021bd5fbe77169eabdc83c5500ea8fea40
+size 4169791169

model-00029-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:974ea6c2923cd4c155a4e018d88f92d04227e94a1ac68bf69352be8d99a1572d
+size 4169791137

model-00031-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:809a23076ea10e11f1ff9292e04b09fb96d99f819ee6c055c5f71e5b40dce9ae
+size 4169791149

model-00032-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb2348b9df14a074b1d75c0697971c349d886ee2219e262f9151673a1b830d4e
+size 4169791121

model-00034-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5abf9deba808570847ed5eb40b8603a5c749aeea3ce7194714f3dd20d99383b6
+size 4169791159

model-00035-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:591f010b487049e3f2ba3d799c0b30031b75fe7bbfe89af7f778d0db9b40a0ab
+size 4169791121

model-00037-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05136fbd18919beb91883e88462017a34f6ae302ae3a3c20c9bfcac727e5712e
+size 4169791183

model-00038-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37e5d0518591eff0f26b8b1682976ede5a6b978d2b795468edf8ae4da7c49b55
+size 4169791201

model-00041-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9234d24fdf6c4dcf713f492e5ea474c444d1c43b1a3050034603a41b9048a14
+size 4169791145

model-00042-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41f8bed1028f0aa2d99a97e63ce126cef3344f564f3ce452736f5d75df0a6a95
+size 4169791203

model-00043-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6d5648b1d645abc0fb14fdf61ccf67d6c0195ac997d3fbda43a48374184f1c8
+size 4169791169

model-00044-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94e5aed0456d28644f40674c711b2c8b99cb985c1f6c610802ade3a673446b54
+size 4169791129

model-00046-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a91c292c02c8ec228eb18a5948ffe4bc49288284deb621d672163c8a9b8180ac
+size 4169791147

model-00047-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb89b230bba8ddd42e4f06fac5c5d8b5777c74c7ce327192247e45a2fabcf263
+size 4169791123

model-00048-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f627deaab1441d08ae53962215bb6cdf41afb1196d55321cd90467136cc67fe
+size 4169791151

model-00049-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:880c976864896cc7d7ff8e41f1a193d3034c289efbb0b2388295b727ed4422e9
+size 4169791187

model-00050-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7b4e24a2568a2d6ea709da49206b7e0b9fd45267927515a8e089fd4be5b7361
+size 4169791129

model-00053-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b64706dd5c5e9f1648fbf23785de1eeedcd0e4b5611429f8acaefa316a3aa80
+size 4169791119

model-00055-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4f7b526126f09a15c29a5be4f497454831fc523c3385dfb46a393d54d19e10c
+size 4169791163

model-00056-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7933769b12573b3b6895d41d3b14068934dc26b9c461fb695c29920972cb302
+size 4169791125

model-00057-of-00057.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67b110ce8c83bc460b8af3803d475474ed79220c341109f79bc326b0c2961834
+size 4800175181

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae73e31fefce04b005cb41c6781389426fae1a8553b6e58d29f133eaa31ebfb5
+size 15524484