Instructions to use Thox-ai/ThoxNova-12B-Core with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Thox-ai/ThoxNova-12B-Core with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/gemma-4-12b-it")
model = PeftModel.from_pretrained(base_model, "Thox-ai/ThoxNova-12B-Core")

llama-cpp-python

How to use Thox-ai/ThoxNova-12B-Core with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Thox-ai/ThoxNova-12B-Core",
	filename="thoxnova-12b-core-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Thox-ai/ThoxNova-12B-Core with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Thox-ai/ThoxNova-12B-Core:Q4_K_M

Use Docker

docker model run hf.co/Thox-ai/ThoxNova-12B-Core:Q4_K_M

LM Studio
Jan

vLLM

How to use Thox-ai/ThoxNova-12B-Core with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Thox-ai/ThoxNova-12B-Core"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/ThoxNova-12B-Core",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Thox-ai/ThoxNova-12B-Core:Q4_K_M

Ollama
How to use Thox-ai/ThoxNova-12B-Core with Ollama:
```
ollama run hf.co/Thox-ai/ThoxNova-12B-Core:Q4_K_M
```

Unsloth Studio

How to use Thox-ai/ThoxNova-12B-Core with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Thox-ai/ThoxNova-12B-Core to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Thox-ai/ThoxNova-12B-Core to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Thox-ai/ThoxNova-12B-Core to start chatting

Atomic Chat new
Docker Model Runner
How to use Thox-ai/ThoxNova-12B-Core with Docker Model Runner:
```
docker model run hf.co/Thox-ai/ThoxNova-12B-Core:Q4_K_M
```

Lemonade

How to use Thox-ai/ThoxNova-12B-Core with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Thox-ai/ThoxNova-12B-Core:Q4_K_M

Run and chat with the model

lemonade run user.ThoxNova-12B-Core-Q4_K_M

List all available models

lemonade list

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

ThoxNova-12B-Core

THOX SFT LoRA adapter for the on-device THOX assistant family. Conservative chat variant for fleet defaults.

At a glance


Base model	`google/gemma-4-12b (via unsloth/gemma-4-12b-it)`
Parameters	12B
Adapter	LoRA (r=8, alpha=16, target: attention + MLP projections)
Quantization (GGUF)	Q4_K_M, Q5_K_M
Context	8192
Training data	THOX-generated instruction data, 1,565 rows (SFT)
Final train loss	0.813
License	Gemma Terms of Use (inherited from base)
Target device	ThoxNova

Intended use

On-device THOX assistant workflows — chat, code, documents, design, and device support — tied to the THOX device family. Runs locally via Ollama or llama.cpp; no cloud account required.

Not for

Medical, legal, or financial advice; direct access by minors without adult supervision; safety-critical control loops; automated content moderation without human review.

Training data

Fine-tuned on THOX-generated instruction data (1,565 rows for this SFT model). Base-model pre-training data is governed by the upstream provider's terms; THOX does not claim provenance over it.

Bias & limitations

Honest disclosure: SFT may echo base-model biases. This model is not RLHF'd for harm reduction beyond a light identity persona. Hallucinations are possible. The THOX identity persona is a light overlay and may not fully suppress base-model self-identification.

Prompt template

<bos><|turn>user\n{prompt}<turn|>\n<|turn>model\n

Stop token: <turn|>

Identity: pinned via the thox-identity.system.md system prompt (embedded as the Ollama SYSTEM directive) plus a thox-identity.filter.py post-filter that rewrites any residual base-vendor self-identification before display.

Q5_K_M is available as a file download (Ollama rejects the 8.5 GB single-blob upload): hf download Thox-ai/ThoxNova-12B-Core thoxnova-12b-core-Q5_K_M.gguf

Quick start

Ollama registry: smaller tags (Q4_K_M / IQ4_XS / Q3_K_M) only. Q5_K_M is HF-only, install locally with ollama create — instructions below.

Ollama

ollama pull thox-ai/thoxnova-core:12b-q4_k_m
ollama run thox-ai/thoxnova-core:12b-q4_k_m "Who are you?"

GGUF via HF

hf download Thox-ai/ThoxNova-12B-Core --include "*.gguf"

HF Transformers (adapter)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("google/gemma-4-12b-it", device_map="auto")
model = PeftModel.from_pretrained(base, "Thox-ai/ThoxNova-12B-Core")
tok = AutoTokenizer.from_pretrained("google/gemma-4-12b-it")

llama.cpp

llama-cli -m ThoxNova-12B-Core-Q4_K_M.gguf -p "Who are you?" -no-cnv -st

Related artifacts

HF: https://huggingface.co/Thox-ai/ThoxNova-12B-Core
Ollama: https://ollama.com/thox-ai/thoxnova-core
Factory: https://github.com/ttracx/thoxllm-factory
THOX device family: https://github.com/ttracx/thox-device-dependency-map

Changelog

v1 (2026-07-05): initial SFT release; GGUF Q4_K_M, Q5_K_M; card + legal footer.

Ollama Q5 Distribution Note

The Q5_K_M GGUF build is distributed directly through Hugging Face because the Ollama registry currently fails large single-blob uploads for this build with HTTP 416 range errors during ollama push.

Users can still run the model in Ollama locally:

ollama create thoxnova-core:12b-q5_k_m -f Modelfile.q5
ollama run thoxnova-core:12b-q5_k_m

Recommended registry tags remain available for smaller quantizations where supported.

Ollama added improved GGUF compatibility in Ollama 0.30, so the safest user path is still local GGUF + ollama create, not registry upload for this Q5 blob.

Windows install / local create

$repo="Thox-ai/ThoxNova-12B-Core"; $dir="V:\Models\ThoxNova-12B-Core"; New-Item -ItemType Directory -Force $dir | Out-Null; hf download $repo --include "*.gguf" --local-dir $dir; Set-Content "$dir\Modelfile.q5" "FROM $dir\ThoxNova-12B-Core-Q5_K_M.gguf`nPARAMETER num_ctx 24576`nPARAMETER temperature 0.7`nSYSTEM `"You are THOX Nova Core, a local-first private AI runtime for THOX.ai devices.`""; ollama create thoxnova-core:12b-q5_k_m -f "$dir\Modelfile.q5"; ollama run thoxnova-core:12b-q5_k_m

Access and distribution

Ollama (public discovery surface). The merged GGUF is freely pullable from the public Thox.ai Ollama profile (see Quick start). Ollama's registry is open, so the Ollama-hosted copy is public.
HuggingFace (gated source of weights). This repository is gated. Download of the GGUF and LoRA adapter requires manual access approval by Thox.ai LLC. See GATED_ACCESS.md for the access-request questionnaire.

License & Legal

License: This adapter is released under the Gemma Terms of Use, inherited from the base model. See Gemma Terms of Use for exact terms.

No warranty: Provided AS IS without warranty of any kind. Users assume all risk.

Privacy: Thox.ai LLC does not collect user prompts or completions from local inference. See https://thox.ai/privacy for the full policy.

Trademarks: THOX and Thox.ai are trademarks of Thox.ai LLC. Gemma is a trademark of Google LLC. Trademark use here does not imply endorsement by the respective owners.

Entity: Thox.ai LLC is an independent Texas limited liability company, and not a DBA of any other entity.

Officers: Tommy Xaypanya - Chief Technology Officer (CTO). Craig Ross - Chief Executive Officer (CEO).

No affiliation: Thox.ai LLC is independent of, and not affiliated with, endorsed by, or sponsored by the upstream base model owner.

Work product: The THOX-specific LoRA fine-tune, the merged model, and the GGUF quantization and packaging are works of Thox.ai LLC, Copyright (c) 2026 Thox.ai LLC. Thox.ai LLC claims no ownership of the upstream base model weights.

Downloads last month: 62

GGUF

Model size

12B params

Architecture

gemma4

Hardware compatibility

4-bit

5-bit