Lekhansh/gemma-3n-e4b-it-asr-psychiatric-domain-kannada

Gemma-3n-E4B-It - Kannada LoRA Adapter (Psychiatric Domain)

This is a LoRA (Low-Rank Adaptation) adapter for Google's Gemma-3n-E4B-It model, fine-tuned on psychiatric interviews and therapy session transcriptions in Kannada for automatic speech recognition outputs.

⚠️ Content Warning & Gated Repository

This model is behind a gated repository for important safety reasons:

Since this model was trained on real-world clinical psychiatric conversations, there is an anticipated risk that model hallucinations may contain sensitive content related to mental health topics. This model is intended solely for research purposes and clinical applications by qualified teams.

To request access: Please contact the repository owner with:

• Your research affiliation or clinical organization
• Intended use case
• Confirmation of ethical approval for your project (if applicable)

Model Description

• Base Model: google/gemma-3n-e4b-it
• Adapter Type: LoRA (Low-Rank Adaptation)
• Language: Kannada (ಕನ್ನಡ)
• Task: Automatic Speech Recognition (Direct Audio-to-Text Transcription)
• Domain: Psychiatric interviews and therapy sessions
• PEFT Version: 0.18.0

LoRA Configuration

• Rank (r): 32
• Alpha: 64
• Dropout: 0.05
• RSLoRA: Enabled
• Target Modules: q_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Performance

The fine-tuned model shows significant improvements over the base Gemma-3n-E4B-It model on Kannada psychiatric conversation data. All improvements are statistically significant (p < 0.001).

Test Set Results (519 instances from 8 files, ~3.9 hours)

Metric	Base Model (Median)	Fine-tuned Model (Median)	Improvement
WER (normalized)	68.18%	53.49%	21.55% relative
CER	37.12%	23.25%	37.37% relative
WIP	0.1062	0.2304	+116.95%
MER	69.84%	54.55%	21.89% relative

Dev Set Results (7 files, ~3.5 hours)

Metric	Base Model (Median)	Fine-tuned Model (Median)	Improvement
WER (normalized)	70.94%	56.59%	20.23% relative
CER	41.22%	26.60%	35.47% relative

Training Set Results (390 sampled instances, ~17.6 hours)

Metric	Base Model (Median)	Fine-tuned Model (Median)	Improvement
WER (normalized)	67.69%	47.20%	30.27% relative
CER	36.69%	20.35%	44.53% relative

Transcription Prompts

This model requires a carefully crafted Kannada-specific prompt that provides detailed transcription guidelines. The prompt is available in the prompts/ directory and specifies:

Key Prompt Features:

1. Verbatim Transcription Requirements

   • Preserve all dysfluencies (filler words, repetitions, stammers, partial words)
   • Include incomplete phrases as-is
   • No grammar correction or polishing

2. Punctuation Rules

   • Allowed: full-stop, question mark, comma, ellipsis, em-dash, exclamation mark
   • Specific usage guidelines for each punctuation type

3. Special Tokens for Non-Speech Events

   • Emotional expressions: [ನಗುವಿನ ಸದ್ದು] (laughing), [ಅಳುವಿನ ಸದ್ದು] (crying), [ಕೂಗುತ್ತಾ] (shouting)
   • Unclear speech marker: [ಅಸ್ಪಷ್ಟ]

4. Code-Mixing Handling

   • Kannada prompt handles English, Hindi, Telugu, Tamil, and Malayalam code-mixing
   • All output in Kannada script (phonetically transcribed for non-Kannada words)

5. Numerical Quantities

   • Spell out all numbers as spoken (e.g., "ಎರಡು ಗಂಟೆ" not "೨ ಗಂಟೆ" or "2 ಗಂಟೆ")
   • Apply to both cardinal and ordinal numbers

Prompt File:

• prompts/transcription_prompt_chunks_gemma_kannada.txt - Kannada psychiatric interview transcription

Training Data

The model was fine-tuned on psychiatric interview transcriptions in Kannada:

Training Set (~390 sampled instances from 35 audio files, 17.56 hours total)

Dev Set (7 files, 3.50 hours)

Test Set (519 instances from 8 files, 3.89 hours)

All splits were created with a random seed of 42.

Audio Preprocessing Requirements

IMPORTANT: Before using this model for transcription, audio files must be preprocessed:

1. Chunk audio files to under 30 seconds duration using Silero VAD (Voice Activity Detection)
2. This ensures optimal performance as the model was trained on ~30 second chunks
3. Silero VAD helps create natural speech boundaries for chunking

Usage

Required Prompts

The model requires a Kannada-specific prompt for optimal performance. This prompt is included in the prompts/ directory:

• Kannada: prompts/transcription_prompt_chunks_gemma_kannada.txt

This prompt contains detailed instructions for verbatim transcription, punctuation rules, handling of dysfluencies, code-mixing, and special tokens.

Transcription Example

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
import torch

# Load base model and processor
model_id = "google/gemma-3n-e4b-it"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id).to("cuda")

# Load LoRA adapter
model = PeftModel.from_pretrained(
    model,
    "Lekhansh/gemma-3n-e4b-it-asr-psychiatric-domain-kannada"
)

# Load the Kannada prompt
with open("prompts/transcription_prompt_chunks_gemma_kannada.txt", "r") as f:
    prompt_text = f.read()

# Path to your audio file (must be chunked to <30s using Silero VAD)
audio_file_path = "path/to/your/audio_chunk.wav"

# Prepare the messages
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": prompt_text},
            {"type": "audio", "audio": audio_file_path},
        ]
    }
]

# Apply chat template and generate
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
)
input_len = inputs["input_ids"].shape[-1]
inputs = inputs.to(model.device, dtype=model.dtype)

with torch.inference_mode():
    generation = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=1.0,
        top_k=64,
        top_p=0.95,
        disable_compile=False
    )
    generation = generation[:, input_len:]

transcription = processor.batch_decode(generation, skip_special_tokens=True)[0]
print(transcription)

Merging Adapter with Base Model (Optional)

For faster inference, you can merge the adapter weights with the base model:

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
import torch

# Load base model and adapter
base_model = AutoModelForImageTextToText.from_pretrained(
    "google/gemma-3n-e4b-it",
    torch_dtype=torch.float16
)
processor = AutoProcessor.from_pretrained("google/gemma-3n-e4b-it")

model = PeftModel.from_pretrained(
    base_model,
    "Lekhansh/gemma-3n-e4b-it-asr-psychiatric-domain-kannada"
)

# Merge and unload
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged_model")
processor.save_pretrained("./merged_model")

Error Type Analysis

The fine-tuned model shows significant improvements across all error types:

Test Set Error Reduction

Error Type	Base Model (Median)	Fine-tuned Model (Median)	Improvement
Substitutions	30.0	24.0	20.0% reduction
Insertions	2.0	4.0	-100.0%
Deletions	6.0	2.0	66.7% reduction

Notable improvements:

• Substitutions reduced from 29.64 ± 8.28 to 23.91 ± 6.97 (mean ± SD)
• Insertions significantly controlled: 51.35 ± 181.28 to 12.66 ± 64.48
• Deletions dramatically reduced from 6.88 ± 7.01 to 3.92 ± 6.06

Intended Use

Primary Use Cases

• Clinical Research: Direct audio-to-text transcription of psychiatric interviews in Kannada
• Mental Health Documentation: Automated verbatim transcription of therapy sessions and psychiatric interviews
• Kannada ASR: Native Kannada automatic speech recognition for psychiatric domain

Out-of-Scope Use

• Diagnostic Tool: This model should not be used as a standalone diagnostic tool
• Replacement for Human Review: Corrected transcriptions should be reviewed by qualified mental health professionals
• Non-Psychiatric General Purpose: While it may work on general text, it's optimized for psychiatric domain

Limitations and Biases

• Domain-Specific: Optimized for psychiatric/clinical conversations; may not generalize well to other domains
• Language Coverage: Specifically tuned for Kannada; performance on other languages may vary
• Hallucination Risk: As with all LLMs, the model may hallucinate content, which in this psychiatric context could include incorrect mental health information
• Data Privacy: Trained on real clinical data; users must ensure compliance with data protection regulations (HIPAA, GDPR, etc.)
• Code-Mixing: While the model handles Indic language and English code-mixing, performance may vary with extensive code-mixing

Ethical Considerations

• This model was trained on real psychiatric conversations. Users must ensure appropriate ethical approvals and patient consent for any clinical use
• Corrected transcriptions should always be reviewed by qualified mental health professionals
• The model should not be used for surveillance or unauthorized recording of psychiatric sessions
• Proper data security and patient confidentiality must be maintained
• Outputs should not be used for clinical decision-making without human verification
• Special care must be taken given the sensitive nature of psychiatric content

Statistical Significance

All reported improvements have been validated using the Wilcoxon signed-rank test with p < 0.001, indicating highly significant improvements over the base model across all metrics.

License

This adapter is released under the Apache 2.0 license. However, users must also comply with Google's Gemma license and any applicable regulations regarding clinical and psychiatric data.

Citation

If you use this model in your research, please cite:

@misc{gemma-kannada-psychiatric-lora,
  title={Gemma-3n-E4B-It Kannada LoRA Adapter for Psychiatric ASR},
  author={Lekhansh Shukla, Prakrithi Shivaprakash},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/lekhansh/gemma-3n-E4B-it-trl-sft-kannada-only-final}}
}

Contact

For access requests or questions about this model, please contact Dr Lekhansh Shukla @ [email protected].

---

Disclaimer: This model is provided for research purposes only. Users are responsible for ensuring compliance with all applicable laws, regulations, and ethical guidelines when using this model, particularly regarding patient privacy, mental health data handling, and psychiatric care standards.