Ayansk11/Finistral-7B_lora

See config

base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizergin
is_mistral_derived_model: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: data.jsonl
    ds_type: json
    type:
      field_instruction: instruction
      field_input: input
      field_output: output
      
      format: |-
        [INST]{input}
        {instruction} [/INST] 


dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 4096
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

gradient_accumulation_steps: 1
micro_batch_size: 32
num_epochs: 4
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.0001

bf16: auto
fp16: false
tf32: false
train_on_inputs: false
group_by_length: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
save_steps:
debug:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"

Finistral-7B Financial Sentiment Analyst

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the FinGPT Sentiment dataset. It is intended to be used for sentiment analysis tasks for financial data.

Python Example

# Downloading necessary dependencies
!pip install -qU transformers peft datasets scikit-learn seaborn matplotlib pandas tqdm bitsandbytes gradio


from huggingface_hub import login
login(token="Your_HuggingFace_Access_Token")


import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Defining model identifiers
base_model = "mistralai/Mistral-7B-v0.1"
peft_model = "Ayansk11/Finistral-7B_lora"

# Loading tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model, use_auth_token=True)
tokenizer.pad_token = tokenizer.eos_token

# Loading base model with 8-bit precision
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
    load_in_8bit=True,
    trust_remote_code=True,
    use_auth_token=True
)

# Loading LoRA adapter
model = PeftModel.from_pretrained(model, peft_model)
model.eval()


def predict_sentiment(text):

    # Formatting the input text into a prompt

    prompt = f"""Instruction: What is the sentiment of this news? Please choose an answer from {{negative/neutral/positive}}
    Input: {text}
    Answer: """

    # Tokenizing the prompt
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    # Generating Answer
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=20)

    decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

    # Extracting the answer
    answer = decoded_output.split("Answer:")[-1].strip().split()[0].lower()

    # Determining the sentiment
    if "neg" in answer:
        return "Negative"
    elif "neu" in answer:
        return "Neutral"
    elif "pos" in answer:
        return "Positive"
    else:
        return "Neutral"  


import gradio as gr

custom_css = """
body, .gradio-container {
    background-color: #0d1117 !important;
    color: #e1e1e1 !important;
}
.gradio-container .input_txt,
.gradio-container .output_txt {
    background-color: #1e272e !important;
    border-radius: 0.5rem !important;
    padding: 1rem !important;
}
.gradio-container button {
    border-radius: 0.5rem !important;
    padding: 0.75rem 1.5rem !important;
    font-weight: 600 !important;
}
.gradio-container button:hover {
    background-color: #00b894 !important;
}
.gradio-container button:focus {
    outline: 2px solid #0984e3 !important;
}
"""

with gr.Blocks(
    
    title="Finistral AI: The Market Sentiment Analyst",
    theme=gr.themes.Base(),  
    css=custom_css          
) as demo:

    gr.Markdown(
        """
        <div style='text-align:center;'>
          <span style='font-size:2.5rem;'>💹</span>
          <span style='font-size:2rem; font-weight:600; color:#00b894;'>
            Finistral AI
          </span><br>
          <span style='font-size:1.25rem; color:#e1e1e1;'>
            The Market Sentiment Analyst
          </span>
        </div>
        """
    )
    gr.Markdown(
        "Paste in a financial news snippet, headline, or article "
        "to see its sentiment breakdown."
    )

    with gr.Row():
        with gr.Column(scale=2):
            input_text = gr.Textbox(
                lines=6,
                placeholder="E.g. “Tech stocks rally as Fed signals rate cut…”",
                label="Enter Financial News"
            )
            analyze_btn = gr.Button("🔍 Analyze Sentiment")
        with gr.Column(scale=1):
            output_lbl = gr.Label(
                num_top_classes=3,
                label="Top 3 Sentiment Classes"
            )

    analyze_btn.click(
        fn=predict_sentiment,
        inputs=input_text,
        outputs=output_lbl
    )

    gr.Markdown(
        "<p style='text-align:center; color:#7f8c8d; font-size:0.875rem;'>"
        "Powered by Finistral AI • Built with Gradio"
        "</p>"
    )

demo.launch(share=True)


# Output:    
# positive
# neutral
# negative

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 64
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
0.0680	1.0	1140	0.1121
0.1337	2.0	2280	0.1009
0.0499	3.0	3420	0.1147
0.0014	4.0	4560	0.1599

Frameworks

• PEFT
• Transformers
• Pytorch
• Datasets
• Tokenizers