Update README.md

README.md

@@ -1,3 +1,138 @@
----
-license: apache-2.0
----
+---
+library_name: rkllm
+pipeline_tag: text-generation
+license: apache-2.0
+language:
+- en
+base_model:
+- openbmb/MiniCPM3-4B
+tags:
+  - rkllm
+  - rk3588
+  - rockchip
+  - edge-ai
+  - llm
+  - MiniCPM3
+  - text-generation-inference
+---
+# MiniCPM3-4B — RKLLM build for RK3588 boards
+
+**Author:** @jamescallander  
+**Source model:** [openbmb/MiniCPM4-0.5B · Hugging Face](https://huggingface.co/openbmb/MiniCPM4-0.5B)
+
+**Target:** Rockchip RK3588 NPU via **RKNN-LLM Runtime**
+
+> This repository hosts a **conversion** of `MiniCPM3-4B` for use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the [RKNN-LLM toolkit](https://github.com/airockchip/rknn-llm?utm_source=chatgpt.com)
+
+#### Conversion details
+
+- **RKLLM-Toolkit version:** v1.2.1
+- **NPU driver:** v0.9.8
+- **Python:** 3.12
+- **Quantization:** `w8a8_g128`
+- **Output:** single-file `.rkllm` artifact
+- **Modifications:** quantization (w8a8_g128), export to `.rkllm` format for RK3588 SBCs
+- ****Tokenizer:**** not required at runtime (UI handles prompt I/O)
+
+## Intended use
+
+- On-device lightweight inference on RK3588 SBCs.
+- MiniCPM3-4B is a **compact general-purpose model** designed for efficiency, testing, and resource-constrained scenarios. Ideal for experimentation where low memory usage and fast response matter more than deep reasoning.
+
+## Limitations
+
+- Requires 8GB free memory
+- Tested on a Radxa Rock 5B+, other devices may require different drivers/toolkit versions.
+- Quantization (`w8a8_g128`) may further reduce output fidelity.
+
+## Quick start (RK3588)
+
+### 1) Install runtime
+
+The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from [airockchip's github page](https://github.com/airockchip).
+
+Download and install the required packages as per the toolkit's instructions.
+
+### 2) Simple Flask server deployment
+
+The simplest way the deploy the `.rkllm` converted model is using an example script provided in the toolkit in this directory: `rknn-llm/examples/rkllm_server_demo`
+
+```bash
+python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
+  --rkllm_model_path <MODEL_PATH>/MiniCPM3-4B_w8a8_g128_rk3588.rkllm \
+  --target_platform rk3588
+```
+
+### 3) Sending a request
+
+A basic format for message request is:
+
+```json
+{
+    "model":"MiniCPM3-4B",
+    "messages":[{
+        "role":"user",
+        "content":"<YOUR_PROMPT_HERE>"}],
+    "stream":false
+}
+```
+
+Example request using `curl`:
+
+```bash
+curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
+    -H 'Content-Type: application/json' \
+    -d '{"model":"MiniCPM3-4B","messages":[{"role":"user","content":"In 2 or 3 sentences, who was Napoleon Bonaparte?"}],"stream":false}'
+```
+
+The response is formated in the following way:
+
+```json
+{
+    "choices":[{
+        "finish_reason":"stop",
+        "index":0,
+        "logprobs":null,
+        "message":{
+            "content":"<MODEL_REPLY_HERE">,
+            "role":"assistant"}}],
+        "created":null,
+        "id":"rkllm_chat",
+        "object":"rkllm_chat",
+        "usage":{
+            "completion_tokens":null,
+            "prompt_tokens":null,
+            "total_tokens":null}
+}
+```
+
+Example response:
+
+```json
+{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Napoleon Bonaparte (1769-1821) was a French military leader and emperor who rose to prominence during the French Revolution and went on to conquer much of Europe. His strategic brilliance and charisma earned him many followers, but his ambitions ultimately led to his downfall. After being exiled to the island of Saint Helena in the South Atlantic, he died in 1821 at the age of 51.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
+```
+
+### 4) UI compatibility
+
+This server exposes an **OpenAI-compatible Chat Completions API**.
+
+You can connect it to any OpenAI-compatible client or UI (for example: [Open WebUI](https://github.com/open-webui/open-webui?utm_source=chatgpt.com))
+
+- Configure your client with the API base: `http://<SERVER_IP_ADDRESS>:8080` and use the endpoint: `/rkllm_chat`
+- Make sure the `model` field matches the converted model’s name, for example:
+
+```json
+{
+ "model": "MiniCPM3-4B",
+ "messages": [{"role":"user","content":"Hello!"}],
+ "stream": false
+}
+```
+
+# License
+
+This conversion follows the license of the source model: [apache-2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md)
+
+- Attribution: **Built with MiniCPM3-4B (OpenBMB)**
+- Required notice: see [`NOTICE`](NOTICE)
+- Modifications: quantization (w8a8_g128), export to `.rkllm` format for RK3588 SBCs