outspark/ko-gemma-2b-lora-lbox-ljp-modified

Model Card for Enhanced Language Model with LoRA

Model Description

This model, a LoRA-finetuned language model, is based on beomi/ko-gemma-2b. It was trained using the lbox/lbox_open and ljp_criminal datasets, specifically prepared by merging facts fields with ruling.text. This training approach aims to enhance the model's capability to understand and generate legal and factual text sequences. The fine-tuning was performed on two A100 GPUs.

LoRA Configuration

• LoRA Alpha: 32
• Rank (r): 16
• LoRA Dropout: 0.05%
• Bias Configuration: None
• Targeted Modules:
  • Query Projection (q_proj)
  • Key Projection (k_proj)
  • Value Projection (v_proj)
  • Output Projection (o_proj)
  • Gate Projection (gate_proj)
  • Up Projection (up_proj)
  • Down Projection (down_proj)

Training Configuration

• Training Epochs: 1
• Batch Size per Device: 2
• Optimizer: Optimized AdamW with paged 32-bit precision
• Learning Rate: 0.00005
• Max Gradient Norm: 0.3
• Learning Rate Scheduler: Constant
• Warm-up Steps: 100
• Gradient Accumulation Steps: 1

Model Training and Evaluation

The model was trained and evaluated using the SFTTrainer with the following parameters:

• Max Sequence Length: 4096
• Dataset Text Field: training_text
• Packing: Disabled

How to Get Started with the Model

Use the following code snippet to load the model with Hugging Face Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("your_model_id")
tokenizer = AutoTokenizer.from_pretrained("your_model_id")

# Example usage
inputs = tokenizer("Example input text", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))