Hercule66/mistral-7b-agriculture-lora-tpu

AgriMistral-7B-LoRA: A Specialized Agricultural LLM for Local Contexts

This model is a specialized, parameter-efficient language model fine-tuned from mistralai/Mistral-7B-v0.3. It is designed to understand and generate text related to the agricultural domain, with a specific focus on the vocabulary, practices, and challenges relevant to local French-speaking communities in West Africa.

The core innovation behind this model is the application of Low-Rank Adaptation (LoRA), a Parameter-Efficient Fine-Tuning (PEFT) technique. This approach enabled the adaptation of the powerful base model by training only a minuscule fraction of its parameters—approximately 851,000 out of 7.24 billion, or just 0.01%.

Model Details

Model Description

The Challenge: The Agricultural AI Divide

In the agricultural sector, valuable data is often scarce, dispersed, and poorly adapted to local needs. When it comes to training Large Language Models (LLMs), the resource requirements are formidable. This technological barrier risks leaving local communities as mere spectators in the ongoing AI revolution.

The Solution: Parameter-Efficient Fine-Tuning (PEFT) with LoRA

This project directly confronts that challenge by leveraging Low-Rank Adaptation (LoRA). LoRA works by injecting small, trainable low-rank matrices into the layers of a pre-trained model, freezing the vast majority of the original weights. This allows for a dramatic reduction in the number of trainable parameters.

• Developed by: Students and Researchers of the Université Nationale d'Agriculture
• Model type: Causal Language Model (Transformer-based)
• Language(s) (NLP): French (fr), with a specialized vocabulary for West African agriculture
• License: Apache 2.0
• Finetuned from model: mistralai/Mistral-7B-v0.3

Model Sources

• Repository: Hugging Face Hub
• Paper:
  • Mistral 7B by Jiang, A. Q., et al. (2023)
  • LoRA: Low-Rank Adaptation of Large Language Models by Hu, E. J., et al. (2021)
• PEFT Library: Documentation

Uses

Direct Use

This model is intended for direct use in text-generation tasks within the agricultural domain. It can function as a specialized question-answering system, a domain-specific content generator, or a conversational assistant for agricultural topics.

Example Use Cases:

• For Farmers: Answering practical queries such as "What are the signs of maize streak virus?" or "Describe organic methods for controlling pests on tomato plants."
• For Agricultural Students: Serving as an interactive study aid that can explain complex concepts like soil composition, nitrogen fixation, or crop rotation using locally relevant terminology.
• For Agricultural Extension Agents: Assisting in the drafting of advisory documents and training materials for local farming communities.

Downstream Use

The parameter-efficient nature of this model makes it an excellent foundation for further specialization:

• Further Fine-Tuning: Users can take this agriculturally-aware LoRA adapter and perform a second stage of fine-tuning on an even more specific dataset
• Application Integration: The model's efficiency makes it highly suitable for integration into mobile apps for farmers, educational chatbots, or information portals

Out-of-Scope Use

• High-Stakes Professional Advice: The model's outputs should not be used to make critical decisions without verification by a qualified human expert
• General-Purpose Chatbot: The model has been fine-tuned for agriculture and will have degraded performance on general topics
• Real-Time Information: The model cannot provide real-time information such as current weather forecasts or live market prices

Bias, Risks, and Limitations

Geographic and Cultural Bias: The model's training data was sourced from the Université Nationale d'Agriculture and Lycée SEKOU, reflecting agricultural practices of a particular region. Its advice may be suboptimal for different French-speaking agricultural regions.

Data Source Limitations: The training corpus was collected from students, which may introduce a bias towards academic knowledge over practical, hands-on farming experience.

General LLM Limitations: Like all LLMs, this model can generate factually incorrect information (hallucinations) and reproduce societal biases present in training data.

Recommendations

Users should carefully evaluate the model's suitability for their specific regional context and consider further fine-tuning on local data before deployment in different agricultural regions.

How to Get Started with the Model

Prerequisites

pip install transformers peft accelerate torch

Python Code for Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Define the identifiers for the base model and the LoRA adapter
base_model_id = "mistralai/Mistral-7B-v0.3"
adapter_id = "your-username/AgriMistral-7B-LoRA"  # Replace with actual model ID

# 1. Load the base model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# 2. Load the PEFT model by applying the adapter to the base model
model = PeftModel.from_pretrained(model, adapter_id)

# 3. Prepare a prompt in French related to agriculture
prompt = "Expliquez les étapes de la préparation du sol pour la culture du maïs en conditions sahéliennes."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# 4. Generate a response from the specialized model
print("Generating response...")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print("\n--- Prompt ---")
print(prompt)
print("\n--- Model Response ---")

Advanced Usage with Custom Parameters

from transformers import GenerationConfig

# Create a custom generation config
generation_config = GenerationConfig(
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

# Generate with custom config
outputs = model.generate(
    **inputs,
    generation_config=generation_config
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response[len(prompt):].strip())

Training Details

Training Data

The model was fine-tuned on a custom, French-language dataset focused on agriculture. The corpus was collected from students and academic materials associated with the Université Nationale d'Agriculture and Lycée SEKOU.

Data Processing:

• Anonymization: Removal of all personally identifiable information
• Data Cleaning: Correction of errors and normalization of text
• Structuring: Formatted into instruction-response format
• Deduplication: Removal of redundant entries

Training Procedure

The fine-tuning was performed using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA) technique, implemented using the Hugging Face ecosystem on Google Cloud TPUs.

Training Hyperparameters

Hyperparameter	Value	Description
base_model	mistralai/Mistral-7B-v0.3	The foundational pre-trained model
peft_method	LoRA	The specific PEFT technique used
lora_r	16	The rank of the LoRA decomposition matrices
lora_alpha	32	The scaling factor for the LoRA weights
lora_dropout	0.05	Dropout probability applied to LoRA layers
target_modules	["q_proj", "v_proj"]	Linear layers where LoRA matrices were injected
learning_rate	2e-4	Initial learning rate
num_train_epochs	3	Total number of training epochs
optimizer	AdamW	Optimization algorithm
precision	bf16	Mixed-precision training format

Speeds, Sizes, Times

• Total Parameters: 7.24 Billion (base model)
• Trainable Parameters: 851,000 (0.01% of total)
• Training Hardware: Google Cloud TPUs
• Framework: Hugging Face PEFT ecosystem

Evaluation

Testing Data, Factors & Metrics

Metrics

• Training Loss: Measures the model's error on training data
• Test Perplexity: Measures how well the model predicts unseen test data

Results

Metric	Value	Significance
Total Parameters	7.24 Billion	Full size of the Mistral-7B-v0.3 base model
Trainable Parameters	851,000 (0.01%)	Core achievement demonstrating LoRA efficiency
Training Loss	1.87	Indicates effective learning of agricultural corpus
Test Perplexity	6.99	Strong generalization ability to unseen text

These results validate the project's thesis: creating high-performing, specialized language models without massive computational infrastructure is possible.

Technical Specifications

Model Architecture and Objective

This model is an adaptation of mistralai/Mistral-7B-v0.3, featuring:

• Grouped-Query Attention (GQA): Optimization reducing KV cache size for faster inference
• Sliding Window Attention (SWA): Efficient processing of long sequences
• Extended Vocabulary: 32,768 tokens with v3 tokenizer

Compute Infrastructure

Software

• peft==0.17.0
• transformers
• torch
• accelerate

Citation

@misc{agrimistral_2024,
  author       = {Students and Researchers of the Université Nationale d'Agriculture},
  title        = {AgriMistral-7B-LoRA: A Specialized Agricultural LLM for Local Contexts},
  year         = {2024},
  publisher    = {Hugging Face},
  journal      = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/AgriMistral-7B-LoRA}},
}

@article{jiang2023mistral,
  title={Mistral 7B},
  author={Jiang, Albert Q and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Chaplot, Devendra Singh and de las Casas, Diego and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and others},
  journal={arXiv preprint arXiv:2310.06825},
  year={2023}
}

@article{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  journal={arXiv preprint arXiv:2106.09685},
  year={2021}
}

Model Card Authors

Students and Researchers of the Université Nationale d'Agriculture

Model Card Contact

For questions and collaborations regarding this model, please contact the research team at the Université Nationale d'Agriculture.

---

Framework versions

• PEFT 0.17.0