|
|
--- |
|
|
license: gemma |
|
|
datasets: |
|
|
- databricks/databricks-dolly-15k |
|
|
base_model: google/gemma-3-270m |
|
|
library_name: peft |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- base_model:adapter:google/gemma-3-270m |
|
|
- transformers |
|
|
- google |
|
|
- gemma3 |
|
|
- prompt-tune |
|
|
- sweelol-ai |
|
|
- peft |
|
|
--- |
|
|
|
|
|
# sweelol/pt-gemma3-270m-dolly |
|
|
|
|
|
This model is part of the **Sweelol AI Hub**, a research project focused on efficient fine-tuning of modern language models on Kaggle accelerators. |
|
|
|
|
|
**Full Research Notebook & Benchmark Results:** [Coming soon] |
|
|
|
|
|
This model is part of the **Sweelol AI Hub** collection, resulting from experiments in efficient fine-tuning, optimization strategies and knowledge distillation on the Gemma-3-270m architecture using the Databricks Dolly-15k dataset on Kaggle TPUs/GPUs. |
|
|
|
|
|
This is a **LoRA-adapted** version of the `google/gemma-3-270m` model. It was fine-tuned on the Databricks Dolly-15k dataset using the **Low-Rank Adaptation (LoRA)** technique. LoRA is a parameter-efficient fine-tuning method that freezes the original model weights and injects trainable low-rank matrices into the attention layers. This allows the model to learn task-specific knowledge (instruction following) while keeping the overall number of trainable parameters low. Only the LoRA adapter weights need to be stored, making this model highly efficient to deploy. |
|
|
|
|
|
- **Developed by:** SweeLOL-ai |
|
|
- **Shared by:** SweeLOL ai |
|
|
- **Model type:** Causal Language Model |
|
|
- **Language(s) (NLP):** English |
|
|
- **License:** Apache 2.0 |
|
|
- **Base Model:** `google/gemma-3-270m` |
|
|
|
|
|
|
|
|
### Description |
|
|
|
|
|
Gemma is a family of lightweight, state-of-the-art open models from Google, |
|
|
built from the same research and technology used to create the Gemini models. |
|
|
Gemma 3 models are multimodal, handling text and image input and generating text |
|
|
output, with open weights for both pre-trained variants and instruction-tuned |
|
|
variants. Gemma 3 has a large, 128K context window, multilingual support in over |
|
|
140 languages, and is available in more sizes than previous versions. Gemma 3 |
|
|
models are well-suited for a variety of text generation and image understanding |
|
|
tasks, including question answering, summarization, and reasoning. Their |
|
|
relatively small size makes it possible to deploy them in environments with |
|
|
limited resources such as laptops, desktops or your own cloud infrastructure, |
|
|
democratizing access to state of the art AI models and helping foster innovation |
|
|
for everyone. |
|
|
|
|
|
### Usage |
|
|
|
|
|
Below, there are some code snippets on how to get quickly started with running the model. First, install the Transformers library. Gemma 3 is supported starting from transformers 4.50.0. |
|
|
|
|
|
```sh |
|
|
$ pip install -U transformers |
|
|
``` |
|
|
|
|
|
Then, copy the snippet from the section that is relevant for your use case. |
|
|
|
|
|
#### Running with the `pipeline` API |
|
|
|
|
|
```python |
|
|
from transformers import pipeline |
|
|
import torch |
|
|
|
|
|
pipe = pipeline("text-generation", model="sweelol/pt-gemma3-270m-dolly", device="cuda", torch_dtype=torch.bfloat16) |
|
|
output = pipe("Eiffel tower is located in", max_new_tokens=50) |
|
|
``` |
|
|
|
|
|
#### Running the model on a single / multi GPU |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoTokenizer, |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("sweelol/pt-gemma3-270m-dolly") |
|
|
if tokenizer.pad_token is None: |
|
|
tokenizer.pad_token = tokenizer.eos_token |
|
|
print("✅ Set tokenizer pad_token to eos_token") |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
"sweelol/lora-gemma3-270m-dolly", |
|
|
torch_dtype=torch.bfloat16 if not USE_AMP else torch.float32, |
|
|
attn_implementation='eager' |
|
|
) |
|
|
print(f"✅ Base model loaded (dtype: {model.dtype}).") |
|
|
|
|
|
|
|
|
prompt = "Eiffel tower is located in" |
|
|
model_inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
|
|
|
input_len = model_inputs["input_ids"].shape[-1] |
|
|
|
|
|
with torch.inference_mode(): |
|
|
generation = model.generate(**model_inputs, max_new_tokens=50, do_sample=False) |
|
|
generation = generation[0][input_len:] |
|
|
|
|
|
decoded = tokenizer.decode(generation, skip_special_tokens=True) |
|
|
print(decoded) |
|
|
``` |
|
|
|
|
|
|
|
|
## Evaluation Results |
|
|
|
|
|
This table compares the performance of this **Prompt-Tuned** model against the original, un-tuned `google/gemma-3-270m` base model. |
|
|
|
|
|
| Benchmark Task | Sweelol Prompt-Tune | Baseline (Gemma-3-270m) | Change | |
|
|
| :--- | :--- | :--- | :--- | |
|
|
| **Average MMLU (5 tasks)** | **25.77%** | 24.88% | **+0.89%** | |
|
|
| HellaSwag (Common Sense) | 39.00% | 43.50% | -4.50% | |
|
|
| ---------------------------------- | ---------- | ---------- | -------- | |
|
|
| *MMLU Sub-task Breakdown:* | | | | |
|
|
| MMLU - Formal Logic | **29.37%** | 25.40% | **+3.97%** | |
|
|
| MMLU - Professional Law | **27.50%** | 27.00% | **+0.50%** | |
|
|
| MMLU - High School Computer Science | **27.00%** | 24.00% | **+3.00%** | |
|
|
| MMLU - Abstract Algebra | 23.00% | 22.00% | +1.00% | |
|
|
| MMLU - High School Mathematics | 22.00% | 26.00% | -4.00% | |
|
|
|
|
|
#### Summary of Findings |
|
|
Prompt Tuning proved to be a highly effective and efficient technique. It was the **second-best performer on MMLU**, significantly boosting logical reasoning while largely preserving the model's strong common-sense abilities. |
|
|
## Citation [optional] |
|
|
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
|
|
## Model Card Contact |
|
|
|
|
|
[More Information Needed] |
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.17.1 |
|
|
|
|
|
# Gemma 3 model card |
|
|
|
|
|
**Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core) |
|
|
|
|
|
**Resources and Technical Documentation**: |
|
|
|
|
|
* [Gemma 3 Technical Report][g3-tech-report] |
|
|
* [Responsible Generative AI Toolkit][rai-toolkit] |
|
|
* [Gemma on Kaggle][kaggle-gemma] |
|
|
* [Gemma on Vertex Model Garden][vertex-mg-gemma3] |
|
|
|
|
|
**Terms of Use**: [Terms][terms] |
|
|
|
|
|
**Authors**: Google DeepMind |
|
|
|
|
|
## Model Information |
|
|
|
|
|
Summary description and brief definition of inputs and outputs. |
|
|
|
|
|
### Description |
|
|
|
|
|
Gemma is a family of lightweight, state-of-the-art open models from Google, |
|
|
built from the same research and technology used to create the Gemini models. |
|
|
Gemma 3 models are multimodal, handling text and image input and generating text |
|
|
output, with open weights for both pre-trained variants and instruction-tuned |
|
|
variants. Gemma 3 has a large, 128K context window, multilingual support in over |
|
|
140 languages, and is available in more sizes than previous versions. Gemma 3 |
|
|
models are well-suited for a variety of text generation and image understanding |
|
|
tasks, including question answering, summarization, and reasoning. Their |
|
|
relatively small size makes it possible to deploy them in environments with |
|
|
limited resources such as laptops, desktops or your own cloud infrastructure, |
|
|
democratizing access to state of the art AI models and helping foster innovation |
|
|
for everyone. |
|
|
|
|
|
### Inputs and outputs |
|
|
|
|
|
- **Input:** |
|
|
- Text string, such as a question, a prompt, or a document to be summarized |
|
|
- Images, normalized to 896 x 896 resolution and encoded to 256 tokens |
|
|
each, for the 4B, 12B, and 27B sizes. |
|
|
- Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and |
|
|
32K tokens for the 1B and 270M sizes. |
|
|
|
|
|
- **Output:** |
|
|
- Generated text in response to the input, such as an answer to a |
|
|
question, analysis of image content, or a summary of a document |
|
|
- Total output context up to 128K tokens for the 4B, 12B, and 27B sizes, |
|
|
and 32K tokens for the 1B and 270M sizes per request, subtracting the |
|
|
request input tokens |
|
|
|
|
|
### Citation |
|
|
|
|
|
```none |
|
|
@article{gemma_2025, |
|
|
title={Gemma 3}, |
|
|
url={https://arxiv.org/abs/2503.19786}, |
|
|
publisher={Google DeepMind}, |
|
|
author={Gemma Team}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Model Data |
|
|
|
|
|
Data used for model training and how the data was processed. |
|
|
|
|
|
### Training Dataset |
|
|
|
|
|
These models were trained on a dataset of text data that includes a wide variety |
|
|
of sources. The 27B model was trained with 14 trillion tokens, the 12B model was |
|
|
trained with 12 trillion tokens, 4B model was trained with 4 trillion tokens, |
|
|
the 1B with 2 trillion tokens, and the 270M with 6 trillion tokens. The |
|
|
knowledge cutoff date for the training data was August 2024. Here are the key |
|
|
components: |
|
|
|
|
|
- Web Documents: A diverse collection of web text ensures the model is |
|
|
exposed to a broad range of linguistic styles, topics, and vocabulary. The |
|
|
training dataset includes content in over 140 languages. |
|
|
- Code: Exposing the model to code helps it to learn the syntax and |
|
|
patterns of programming languages, which improves its ability to generate |
|
|
code and understand code-related questions. |
|
|
- Mathematics: Training on mathematical text helps the model learn logical |
|
|
reasoning, symbolic representation, and to address mathematical queries. |
|
|
- Images: A wide range of images enables the model to perform image |
|
|
analysis and visual data extraction tasks. |
|
|
|
|
|
The combination of these diverse data sources is crucial for training a powerful |
|
|
multimodal model that can handle a wide variety of different tasks and data |
|
|
formats. |
|
|
|
|
|
### Data Preprocessing |
|
|
|
|
|
Here are the key data cleaning and filtering methods applied to the training |
|
|
data: |
|
|
|
|
|
- CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering |
|
|
was applied at multiple stages in the data preparation process to ensure |
|
|
the exclusion of harmful and illegal content. |
|
|
- Sensitive Data Filtering: As part of making Gemma pre-trained models |
|
|
safe and reliable, automated techniques were used to filter out certain |
|
|
personal information and other sensitive data from training sets. |
|
|
- Additional methods: Filtering based on content quality and safety in |
|
|
line with [our policies][safety-policies]. |
|
|
|
|
|
## Implementation Information |
|
|
|
|
|
Details about the model internals. |
|
|
|
|
|
### Hardware |
|
|
|
|
|
Gemma was trained using [Tensor Processing Unit (TPU)][tpu] hardware (TPUv4p, |
|
|
TPUv5p and TPUv5e). Training vision-language models (VLMS) requires significant |
|
|
computational power. TPUs, designed specifically for matrix operations common in |
|
|
machine learning, offer several advantages in this domain: |
|
|
|
|
|
- Performance: TPUs are specifically designed to handle the massive |
|
|
computations involved in training VLMs. They can speed up training |
|
|
considerably compared to CPUs. |
|
|
- Memory: TPUs often come with large amounts of high-bandwidth memory, |
|
|
allowing for the handling of large models and batch sizes during training. |
|
|
This can lead to better model quality. |
|
|
- Scalability: TPU Pods (large clusters of TPUs) provide a scalable |
|
|
solution for handling the growing complexity of large foundation models. |
|
|
You can distribute training across multiple TPU devices for faster and more |
|
|
efficient processing. |
|
|
- Cost-effectiveness: In many scenarios, TPUs can provide a more |
|
|
cost-effective solution for training large models compared to CPU-based |
|
|
infrastructure, especially when considering the time and resources saved |
|
|
due to faster training. |
|
|
- These advantages are aligned with |
|
|
[Google's commitments to operate sustainably][sustainability]. |
|
|
|
|
|
### Software |
|
|
|
|
|
Training was done using [JAX][jax] and [ML Pathways][ml-pathways]. |
|
|
|
|
|
JAX allows researchers to take advantage of the latest generation of hardware, |
|
|
including TPUs, for faster and more efficient training of large models. ML |
|
|
Pathways is Google's latest effort to build artificially intelligent systems |
|
|
capable of generalizing across multiple tasks. This is specially suitable for |
|
|
foundation models, including large language models like these ones. |
|
|
|
|
|
Together, JAX and ML Pathways are used as described in the |
|
|
[paper about the Gemini family of models][gemini-2-paper]; *"the 'single |
|
|
controller' programming model of Jax and Pathways allows a single Python |
|
|
process to orchestrate the entire training run, dramatically simplifying the |
|
|
development workflow."* |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
Model evaluation metrics and results. |
|
|
|
|
|
### Benchmark Results |
|
|
|
|
|
These models were evaluated against a large collection of different datasets and |
|
|
metrics to cover different aspects of text generation. Evaluation results marked |
|
|
with **IT** are for instruction-tuned models. Evaluation results marked with |
|
|
**PT** are for pre-trained models. |
|
|
|
|
|
#### Gemma 3 270M |
|
|
|
|
|
| **Benchmark** | **n-shot** | **Gemma 3 PT 270M** | |
|
|
| :------------------------ | :-----------: | ------------------: | |
|
|
| [HellaSwag][hellaswag] | 10-shot | 40.9 | |
|
|
| [BoolQ][boolq] | 0-shot | 61.4 | |
|
|
| [PIQA][piqa] | 0-shot | 67.7 | |
|
|
| [TriviaQA][triviaqa] | 5-shot | 15.4 | |
|
|
| [ARC-c][arc] | 25-shot | 29.0 | |
|
|
| [ARC-e][arc] | 0-shot | 57.7 | |
|
|
| [WinoGrande][winogrande] | 5-shot | 52.0 | |
|
|
|
|
|
[hellaswag]: https://arxiv.org/abs/1905.07830 |
|
|
[boolq]: https://arxiv.org/abs/1905.10044 |
|
|
[piqa]: https://arxiv.org/abs/1911.11641 |
|
|
[triviaqa]: https://arxiv.org/abs/1705.03551 |
|
|
[arc]: https://arxiv.org/abs/1911.01547 |
|
|
[winogrande]: https://arxiv.org/abs/1907.10641 |
|
|
|
|
|
| **Benchmark** | **n-shot** | **Gemma 3 IT 270m** | |
|
|
| :------------------------ | :-----------: | ------------------: | |
|
|
| [HellaSwag][hellaswag] | 0-shot | 37.7 | |
|
|
| [PIQA][piqa] | 0-shot | 66.2 | |
|
|
| [ARC-c][arc] | 0-shot | 28.2 | |
|
|
| [WinoGrande][winogrande] | 0-shot | 52.3 | |
|
|
| [BIG-Bench Hard][bbh] | few-shot | 26.7 | |
|
|
| [IF Eval][ifeval] | 0-shot | 51.2 | |
|
|
|
|
|
[hellaswag]: https://arxiv.org/abs/1905.07830 |
|
|
[piqa]: https://arxiv.org/abs/1911.11641 |
|
|
[arc]: https://arxiv.org/abs/1911.01547 |
|
|
[winogrande]: https://arxiv.org/abs/1907.10641 |
|
|
[bbh]: https://paperswithcode.com/dataset/bbh |
|
|
[bbh]: https://paperswithcode.com/dataset/bbh |
|
|
[ifeval]: https://arxiv.org/abs/2311.07911 |
|
|
|
|
|
#### Gemma 3 1B, 4B, 12B & 27B |
|
|
|
|
|
##### Reasoning and factuality |
|
|
|
|
|
| Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B | |
|
|
|--------------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:| |
|
|
| [GPQA][gpqa] Diamond | 0-shot | 19.2 | 30.8 | 40.9 | 42.4 | |
|
|
| [SimpleQA][simpleqa] | 0-shot | 2.2 | 4.0 | 6.3 | 10.0 | |
|
|
| [FACTS Grounding][facts-grdg] | - | 36.4 | 70.1 | 75.8 | 74.9 | |
|
|
| [BIG-Bench Hard][bbh] | 0-shot | 39.1 | 72.2 | 85.7 | 87.6 | |
|
|
| [BIG-Bench Extra Hard][bbeh] | 0-shot | 7.2 | 11.0 | 16.3 | 19.3 | |
|
|
| [IFEval][ifeval] | 0-shot | 80.2 | 90.2 | 88.9 | 90.4 | |
|
|
|
|
|
| Benchmark | n-shot | Gemma 3 PT 1B | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B | |
|
|
| ------------------------------ |----------|:--------------:|:-------------:|:--------------:|:--------------:| |
|
|
| [HellaSwag][hellaswag] | 10-shot | 62.3 | 77.2 | 84.2 | 85.6 | |
|
|
| [BoolQ][boolq] | 0-shot | 63.2 | 72.3 | 78.8 | 82.4 | |
|
|
| [PIQA][piqa] | 0-shot | 73.8 | 79.6 | 81.8 | 83.3 | |
|
|
| [SocialIQA][socialiqa] | 0-shot | 48.9 | 51.9 | 53.4 | 54.9 | |
|
|
| [TriviaQA][triviaqa] | 5-shot | 39.8 | 65.8 | 78.2 | 85.5 | |
|
|
| [Natural Questions][naturalq] | 5-shot | 9.48 | 20.0 | 31.4 | 36.1 | |
|
|
| [ARC-c][arc] | 25-shot | 38.4 | 56.2 | 68.9 | 70.6 | |
|
|
| [ARC-e][arc] | 0-shot | 73.0 | 82.4 | 88.3 | 89.0 | |
|
|
| [WinoGrande][winogrande] | 5-shot | 58.2 | 64.7 | 74.3 | 78.8 | |
|
|
| [BIG-Bench Hard][bbh] | few-shot | 28.4 | 50.9 | 72.6 | 77.7 | |
|
|
| [DROP][drop] | 1-shot | 42.4 | 60.1 | 72.2 | 77.2 | |
|
|
|
|
|
[gpqa]: https://arxiv.org/abs/2311.12022 |
|
|
[simpleqa]: https://arxiv.org/abs/2411.04368 |
|
|
[facts-grdg]: https://goo.gle/FACTS_paper |
|
|
[bbeh]: https://github.com/google-deepmind/bbeh |
|
|
[ifeval]: https://arxiv.org/abs/2311.07911 |
|
|
[hellaswag]: https://arxiv.org/abs/1905.07830 |
|
|
[boolq]: https://arxiv.org/abs/1905.10044 |
|
|
[piqa]: https://arxiv.org/abs/1911.11641 |
|
|
[socialiqa]: https://arxiv.org/abs/1904.09728 |
|
|
[triviaqa]: https://arxiv.org/abs/1705.03551 |
|
|
[naturalq]: https://github.com/google-research-datasets/natural-questions |
|
|
[arc]: https://arxiv.org/abs/1911.01547 |
|
|
[winogrande]: https://arxiv.org/abs/1907.10641 |
|
|
[bbh]: https://paperswithcode.com/dataset/bbh |
|
|
[drop]: https://arxiv.org/abs/1903.00161 |
|
|
|
|
|
##### STEM and code |
|
|
|
|
|
| Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B | |
|
|
|----------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:| |
|
|
| [MMLU][mmlu] (Pro) | 0-shot | 14.7 | 43.6 | 60.6 | 67.5 | |
|
|
| [LiveCodeBench][lcb] | 0-shot | 1.9 | 12.6 | 24.6 | 29.7 | |
|
|
| [Bird-SQL][bird-sql] (dev) | - | 6.4 | 36.3 | 47.9 | 54.4 | |
|
|
| [Math][math] | 0-shot | 48.0 | 75.6 | 83.8 | 89.0 | |
|
|
| HiddenMath | 0-shot | 15.8 | 43.0 | 54.5 | 60.3 | |
|
|
| [MBPP][mbpp] | 3-shot | 35.2 | 63.2 | 73.0 | 74.4 | |
|
|
| [HumanEval][humaneval] | 0-shot | 41.5 | 71.3 | 85.4 | 87.8 | |
|
|
| [Natural2Code][nat2code] | 0-shot | 56.0 | 70.3 | 80.7 | 84.5 | |
|
|
| [GSM8K][gsm8k] | 0-shot | 62.8 | 89.2 | 94.4 | 95.9 | |
|
|
|
|
|
| Benchmark | n-shot | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B | |
|
|
| ------------------------------ |----------------|:-------------:|:--------------:|:--------------:| |
|
|
| [MMLU][mmlu] | 5-shot | 59.6 | 74.5 | 78.6 | |
|
|
| [MMLU][mmlu] (Pro COT) | 5-shot | 29.2 | 45.3 | 52.2 | |
|
|
| [AGIEval][agieval] | 3-5-shot | 42.1 | 57.4 | 66.2 | |
|
|
| [MATH][math] | 4-shot | 24.2 | 43.3 | 50.0 | |
|
|
| [GSM8K][gsm8k] | 8-shot | 38.4 | 71.0 | 82.6 | |
|
|
| [GPQA][gpqa] | 5-shot | 15.0 | 25.4 | 24.3 | |
|
|
| [MBPP][mbpp] | 3-shot | 46.0 | 60.4 | 65.6 | |
|
|
| [HumanEval][humaneval] | 0-shot | 36.0 | 45.7 | 48.8 | |
