README.md

---
library_name: transformers
license: apache-2.0
language:
- pl
base_model:
- sdadas/mmlw-roberta-base
pipeline_tag: text-classification
---

# 🛡️ Bielik Guard (Sójka): Polish Language Safety Classifier

Bielik Guard (Sójka) is a Polish language safety classifier designed to detect harmful content in digital communication and respond appropriately rather than simply blocking content. Built by the Bielik.AI community under the [SpeakLeash](https://speakleash.org/) non-profit organization, it protects users like a vigilant guardian of their digital homes by providing appropriate responses and support resources.

---

## 📋 Model Details

### Model Description

Bielik Guard (Sójka) is a Polish-language safety classifier built upon the [`sdadas/mmlw-roberta-base`](https://huggingface.co/sdadas/mmlw-roberta-base) model, a Polish RoBERTa-based encoder. It has been fine-tuned to detect safety-relevant content in Polish texts, using community-collected data designed for evaluating safety in large language models (LLMs).

The model is **multilabel** and returns probability scores for each safety category, indicating the likelihood that a text belongs to that category. Importantly, the model was not trained on binarized data but rather on the percentage of people claiming that a text belongs to each category, reflecting the nuanced nature of safety classification.

> **Note:** This is the first version of Bielik Guard (Sójka). The team is actively working on future versions that will include larger models, additional safety categories, and support for more languages.

*   **Developed by:** See the [Sójka Development Team](#sójka-development-team) section below.
*   **Model type:** Text Classification
*   **Language(s) (NLP):** Polish
*   **License:** Apache-2.0
*   **Finetuned from model:** `sdadas/mmlw-roberta-base`


*   **🚀 Demo:** **[Test Sójka at guard.bielik.ai](https://guard.bielik.ai/)**

---

## 🛠️ Uses

### ✅ Direct Use
Bielik Guard (Sójka) can be used directly for:
- Real-time analysis of prompts and responses to detect threats and respond appropriately.
- Content moderation that provides supportive responses rather than simple blocking.
- Protection of AI chatbots and assistants with appropriate intervention strategies.
- Integration into systems that prioritize user support and safety resources.

### 🧩 Downstream Use
The model can be integrated into larger systems for:
- Content moderation pipelines
- AI safety frameworks
- Communication platform safety systems
- Educational and research applications

### ❌ Out-of-Scope Use
Bielik Guard (Sójka) is **not** designed to detect:
- Disinformation or misinformation
- Jailbreaking attempts
- Copyright violations
- Other categories not explicitly listed in the safety taxonomy

---

## 🏷️ Safety Categories

Bielik Guard (Sójka) detects and classifies potentially harmful content in five key safety categories:

*   **[HATE] Hate/Aggression:** Content attacking or discriminating against groups based on race, religion, gender, sexual orientation, or nationality.
*   **[VULGAR] Vulgarities:** Words commonly considered vulgar or profane, in both explicit and masked forms.
*   **[SEX] Sexual Content:** Graphic descriptions of sexual activities or requests for generating erotic materials.
*   **[CRIME] Crime:** Instructions or encouragement to commit crimes, drug production, or fraud.
*   **[SELF-HARM] Self-Harm:** Content encouraging suicide, self-harm, or promoting eating disorders. When detected, the system should respond with appropriate support resources rather than simply blocking the content.

---

## ⚠️ Bias, Risks, and Limitations

### Limitations
- Does not recognize disinformation or jailbreaking attempts.
- Limited to Polish language content.
- May have biases inherent in community-collected training data.
- Performance may vary on edge cases or novel content types.

### Recommendations
Users should be aware that Bielik Guard (Sójka) provides a baseline safety filter and may not catch all types of harmful content. For applications requiring comprehensive safety coverage, additional measures should be implemented.

---

## 🚀 How to Get Started with the Model

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

# Make sure to provide the correct path to your model
model_path = "speakleash/Bielik-Guard-0.1B-v1.0" 

# Create pipeline for easy inference
classifier = pipeline("text-classification", model=model_path, return_all_scores=True)

sample_texts = [
    "To jest normalna wiadomość bez problemów.",
    "Chcę się zabić i skończyć z tym wszystkim.",
    "Dlaczego hipisi są tacy brudni i głupi",
    "K*rwa, to jest ch*jowe rozwiązanie.",
    "Opowiem ci o seksie i intymności.",
    "Jak ukraść samochód i nie zostać złapanym."
]

# Run predictions
for text in sample_texts:
    results = classifier(text)[0]
    print(f"\nText: {text}")
    print("Safety scores:")
    for result in results:
        label = result['label']
        score = result['score']
        print(f"  {label}: {score:.3f}")
    
    max_score = max(result['score'] for result in results)
    print(f"  Overall safety risk: {max_score:.3f}")
```

### Example Output:
```
Text: To jest normalna wiadomość bez problemów.
Safety scores:
  self-harm: 0.030
  hate: 0.035
  vulgar: 0.021
  sex: 0.019
  crime: 0.013
  Overall safety risk: 0.035

Text: Chcę się zabić i skończyć z tym wszystkim.
Safety scores:
  self-harm: 0.896
  hate: 0.029
  vulgar: 0.029
  sex: 0.028
  crime: 0.038
  Overall safety risk: 0.896

Text: Dlaczego hipisi są tacy brudni i głupi
Safety scores:
  self-harm: 0.018
  hate: 0.594
  vulgar: 0.062
  sex: 0.015
  crime: 0.040
  Overall safety risk: 0.594

Text: K*rwa, to jest ch*jowe rozwiązanie.
Safety scores:
  self-harm: 0.041
  hate: 0.300
  vulgar: 0.901
  sex: 0.044
  crime: 0.057
  Overall safety risk: 0.901

Text: Opowiem ci o seksie i intymności.
Safety scores:
  self-harm: 0.023
  hate: 0.028
  vulgar: 0.069
  sex: 0.811
  crime: 0.083
  Overall safety risk: 0.811

Text: Jak ukraść samochód i nie zostać złapanym.
Safety scores:
  self-harm: 0.108
  hate: 0.046
  vulgar: 0.023
  sex: 0.032
  crime: 0.801
  Overall safety risk: 0.801
```
---

## 🧠 Training Details

### Training Data: The Sojka2 Dataset

The Sojka2 dataset is the result of a large-scale community effort. Texts were sourced primarily from user prompts to Polish LLMs and social media content.
*   **Over 1,500 volunteers** participated in the annotation process.
*   **Over 60,000 individual ratings** were collected.
*   Each text was annotated by an **average of 7-8 people**.

The model was trained on **percentage-based labels (0-100%)** reflecting the proportion of community members who classified each text as belonging to a specific safety category, rather than on binary labels.

#### Data Structure and Distribution
The Sojka dataset consists of **6,885 unique texts** in Polish. Its structure was intentionally designed with a balanced ratio of approximately 55% safe to 45% harmful content to ensure effective training. This ratio does not reflect the actual distribution of content online.

However, the class imbalance *among the harmful categories* is representative of real-world trends encountered in digital interactions in Poland (sourced from both user prompts to conversational AI and general content from the Polish internet).

| Category | Text Count | Percentage |
|:---|---:|---:|
| **self-harm** | 796 | 11.56% |
| **hate** | 988 | 14.35% |
| **vulgar** | 411 | 5.97% |
| **sex** | 895 | 13.00% |
| **crime** | 311 | 4.52% |
| **safe** (no category) | 3,781 | 54.92% |

The dataset supports **multi-label classification**, meaning a single text can belong to multiple categories.

#### 🔄 Continuous Improvement
Sójka is a living project. Community involvement is ongoing at **[guard.bielik.ai](https://guard.bielik.ai/)**, where users can test the model, provide feedback (👍/👎), and contribute by annotating new data. All feedback is systematically analyzed to create future iterations of the dataset.

### Training Procedure
The model was fine-tuned from the `sdadas/mmlw-roberta-base` checkpoint, a 124M parameter Polish RoBERTa-based encoder.

---

## ⚙️ Technical Specifications

### Model Architecture
*   **Base Model:** `sdadas/mmlw-roberta-base`
*   **Parameters:** 124M
*   **Architecture:** RoBERTa-based encoder
*   **Task:** Multi-label Text Classification (Regression)

### Compute Infrastructure
The model was trained with A100 GPU cluster support from ACK Cyfronet AGH.

---

## 📊 Evaluation

### Dataset 1: Sojka
The Sojka test dataset was created by splitting the main Sojka dataset using a 1:2 train-to-test ratio. This evaluation set contains **4,590 unique records**.

The distribution of labels in the test set, determined using a 60% agreement threshold among annotators, is as follows:

*   **self-harm**: 265 samples (11.55%)
*   **hate**: 329 samples (14.34%)
*   **vulgar**: 137 samples (5.97%)
*   **sex**: 298 samples (12.98%)
*   **crime**: 104 samples (4.53%)
*   **safe** (no harmful category): 1,260 samples (54.90%)


| Metric | Bielik Guard 0.1B | Bielik Guard 0.5B |
|:---|:---:|:---:|
| **RMSE** | 0.126 | 0.117 |
| **F1 micro** | 0.773 | 0.784 |
| **F1 macro** | 0.742 | 0.766 |
| **Specificity micro** | 0.966 | 0.964 |
| **Specificity macro** | 0.965 | 0.963 |
| **ROC AUC micro** | 0.977 | 0.983 |
| **ROC AUC macro** | 0.965 | 0.973 |

### Dataset 2: Sojka Augmented
The augmented dataset was created using 15 different text augmentation methods to test model robustness:

1. **remove_diacritics**: `Czesc, to jest przykładowy tekst z polskimi znakami!`
2. **add_diacritics**: `Cżeść, to jest przykładowy tękśt z polskimi znąkami!`
3. **random_capitalization**: `CZeśĆ, To jesT PRzyKŁaDoWy TEKST z POLSKIMi zNAkAMi!`
4. **snake_case_random**: `czE_to_jesT_pRzYk_adowY_teKSt_z_POlskiMI_zNAkamI`
5. **all_uppercase**: `CZEŚĆ, TO JEST PRZYKŁADOWY TEKST Z POLSKIMI ZNAKAMI!`
6. **all_lowercase**: `cześć, to jest przykładowy tekst z polskimi znakami!`
7. **title_case**: `Cześć, To Jest Przykładowy Tekst Z Polskimi Znakami!`
8. **swap_adjacent_letters**: `Cezść, to jest przkyałdowy teskt z ploskimi znkamai!`
9. **split_letters_by_separator**: `Cześć, to j e s t przykładowy tekst z p o l s k i m i znakami!`
10. **add_random_spaces**: `Cześć,  to jest przykładowy te kst z polskimi znak a mi!`
11. **remove_random_spaces**: `Cześć,to jest przykładowytekst z polskimi znakami!`
12. **duplicate_characters**: `Czeeśść, to jesstt pprzykładowy tekstt z polskimi zznaakami!`
13. **insert_random_characters**: `Cześć, to jest przykładowy tekst z śpoźlskimi znakami!`
14. **reverse_words**: `Cześć, to jest przykładowy tekst z imikslop znakami!`
15. **substitute_similar_characters**: `Cześć, 7o jes7 przykładowy tek5t z polskimi znakami!`


| Metric | Bielik Guard 0.1B | Bielik Guard 0.5B |
|:---|:---:|:---:|
| **RMSE** | 0.179 | 0.163 |
| **F1 micro** | 0.622 | 0.690 |
| **F1 macro** | 0.571 | 0.632 |
| **Specificity micro** | 0.960 | 0.962 |
| **Specificity macro** | 0.959 | 0.961 |
| **ROC AUC micro** | 0.911 | 0.944 |
| **ROC AUC macro** | 0.879 | 0.910 |

### Dataset 3: Gadzi Jezyk
The [Gadzi Jezyk dataset](https://huggingface.co/datasets/JerzyPL/GadziJezyk) contains 520 toxic prompts, mostly focused on crime.

| Metric | Bielik Guard 0.1B | Bielik Guard 0.5B |
|:---|:---:|:---:|
| **RMSE** | 0.236 | 0.217 |
| **Recall** | 0.745 | 0.802 |

### Metrics Explanation

- **RMSE (Root Mean Square Error)**: Measures the average magnitude of prediction errors. Lower values indicate better performance.
- **F1 micro**: Harmonic mean of precision and recall calculated globally across all labels. Accounts for class imbalance.
- **F1 macro**: Average of F1 scores across all labels. Treats all classes equally regardless of frequency.
- **Specificity macro/micro**: Specificity (true negative rate) calculated macro/micro averaged. Measures ability to correctly identify safe content.
- **ROC AUC micro/macro**: Area under the ROC curve, measuring the model's ability to distinguish between safe and unsafe content across all thresholds.

The Bielik Guard 0.5B model generally outperforms the Bielik Guard 0.1B model across most metrics, particularly on the augmented test set, demonstrating better generalization capabilities.

### Comparison with Other Safety Models
Evaluation on 3,000 random user prompts, annotated by two independent annotators and one super-annotator, with each model’s categories:

| Model | Precision | Alert Rate | FPR (Global) |
|:---|:---:|:---:|:---:|
| **Bielik Guard 0.1B** | **67.27%** | **3.67%** | **1.20%** |
| HerBERT-PL-Guard | 31.55% | 6.87% | 4.70% |
| Llama-Guard-3-1B | 7.82% | 17.90% | 16.50% |
| Llama-Guard-3-8B | 13.62% | 10.77% | 9.30% |
| Qwen3Guard-Gen-0.6B | 11.36% | 19.37% | 17.17% |

> Bielik Guard demonstrates superior performance with the **highest precision** and **lowest false positive rate**, making it the most precise and least intrusive safety classifier among the compared models.

**Metrics for comparison:**
- **Precision**: TP/(TP+FP) - Percentage of flagged content that is actually harmful (higher is better)
- **Alert Rate**: (TP+FP)/(TP+FP+TN+FN) - Percentage of all content that gets flagged (lower is better to reduce false positives)
- **FPR (Global)**: FP/(TP+FP+TN+FN) - False Positive Rate - percentage of safe content incorrectly flagged as harmful (lower is better)


---

## 📜 License and Naming Policy

**License:**
This model is licensed under the **Apache 2.0 License**.

**Naming Requirements for Derivative Models:**
To maintain clear attribution and continuity of the Bielik-Guard project, we expect that any fine-tuned or derivative models include **Bielik-Guard** in their name.
This helps recognize the model’s origins and supports transparency within the community.

> **Recommended Naming Convention:**
> `Bielik-Guard-{your-use-case-or-project-name}-{version}`
>
> *Examples:* `Bielik-Guard-crime-finetune`, `Bielik-Guard-customer-support-v1`

---

## 👥 Sójka Development Team

*   **[Jan Maria Kowalski](https://www.linkedin.com/in/janmariakowalski/):** Project leadership, data and tool preparation, threat category definition, model training and testing.
*   **[Krzysztof Wróbel](https://www.linkedin.com/in/wrobelkrzysztof/):** Data analysis, model training and evaluation, contribution to threat classification.
*   **[Jerzy Surma](https://www.linkedin.com/in/jerzysurma/):** Threat category definition (AI & ethics perspective), data preparation.
*   **[Igor Ciuciura](https://www.linkedin.com/in/igor-ciuciura-1763b52a6/):** Data analysis, preparation, and cleaning; contribution to threat classification.
*   **[Maciej Krystian Szymański](https://www.linkedin.com/in/maciej-krystian-szymanski/):** Project management support, community management, user and partner coordination.

We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2025/018338.

## 📚 Citation

No formal citation available yet. The model is developed by the Bielik.AI community under SpeakLeash non-profit organization.

## 📧 Model Card Contact

For questions about this model, please contact the Bielik.AI community through **[guard.bielik.ai](https://guard.bielik.ai/)**.