LbbbbbY
/

FinAI_Contest_FinGPT

@@ -1,593 +0,0 @@
----
-license: mit
-tags:
-- finance
-- llm
-- lora
-- sentiment-analysis
-- named-entity-recognition
-- xbrl
-- apollo
-- rag
-pipeline_tag: text-generation
----
-# FinLoRA: Financial Large Language Models with LoRA Adaptation
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
-[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-## Overview
-FinLoRA is a comprehensive framework for fine-tuning large language models on financial tasks using Low-Rank Adaptation (LoRA). This repository contains trained LoRA adapters for various financial NLP tasks including sentiment analysis, named entity recognition, headline classification, XBRL processing, **RAG-enhanced models** for CFA knowledge and FinTagging tasks, and **APOLLO reasoning layers** for advanced numerical calculations.
-## Model Architecture
-- **Base Model**: Meta-Llama-3.1-8B-Instruct (downloaded locally)
-- **Adaptation Method**: LoRA (Low-Rank Adaptation)
-- **Quantization**: 8-bit and 4-bit quantization support
-- **Multi-Layer Support**: RAG + APOLLO layered architecture
-- **Local Usage**: All models run locally without requiring Hugging Face online access
-- **Tasks**: Financial sentiment analysis, NER, classification, XBRL processing, CFA knowledge, FinTagging, numerical reasoning
-## Available Models
-### 8-bit Quantized Models (Recommended)
-- `sentiment_llama_3_1_8b_8bits_r8` - Financial sentiment analysis
-- `ner_llama_3_1_8b_8bits_r8` - Named entity recognition
-- `headline_llama_3_1_8b_8bits_r8` - Financial headline classification
-- `xbrl_extract_llama_3_1_8b_8bits_r8` - XBRL tag extraction
-- `xbrl_term_llama_3_1_8b_8bits_r8` - XBRL terminology processing
-- `financebench_llama_3_1_8b_8bits_r8` - Comprehensive financial benchmark
-- `finer_llama_3_1_8b_8bits_r8` - Financial NER
-- `formula_llama_3_1_8b_8bits_r8` - Financial formula processing
-### RAG-Enhanced Models (Knowledge-Augmented)
-- `cfa_rag_llama_3_1_8b_8bits_r8` - CFA knowledge-enhanced model with RAG
-- `fintagging_combined_rag_llama_3_1_8b_8bits_r8` - Combined FinTagging RAG model
-- `fintagging_fincl_rag_llama_3_1_8b_8bits_r8` - FinCL RAG-enhanced model
-- `fintagging_finni_rag_llama_3_1_8b_8bits_r8` - FinNI RAG-enhanced model
-### APOLLO Models (Advanced Reasoning Layer) 🚀
-- `apollo_cfa_rag_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for CFA tasks
-- `apollo_fintagging_combined_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for FinTagging tasks
-**Note**: APOLLO models are designed to be loaded on top of RAG models for enhanced numerical reasoning and calculation capabilities.
-### Bloomberg-Enhanced Models (Specialized Financial Tasks) 📊
-- `finlora_lora_ckpt_llama_8bit_r8` - Bloomberg FPB and FIQA specialized model
-- `finlora_heads_llama_8bit_r8.pt` - Bloomberg model weights (71MB)
-**Note**: Bloomberg models are specialized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.
-### 4-bit Quantized Models (Memory Efficient)
-- `sentiment_llama_3_1_8b_4bits_r4` - Financial sentiment analysis
-- `ner_llama_3_1_8b_4bits_r4` - Named entity recognition
-- `headline_llama_3_1_8b_4bits_r4` - Financial headline classification
-- `xbrl_extract_llama_3_1_8b_4bits_r4` - XBRL tag extraction
-- `xbrl_term_llama_3_1_8b_4bits_r4` - XBRL terminology processing
-- `financebench_llama_3_1_8b_4bits_r4` - Comprehensive financial benchmark
-- `finer_llama_3_1_8b_4bits_r4` - Financial NER
-- `formula_llama_3_1_8b_4bits_r4` - Financial formula processing
-## Quick Start
-### 1. Installation
-```bash
-# Install dependencies
-pip install -r requirements.txt
-```
-### 2. Local Model Setup
-**Important**: This project uses locally downloaded models, not online Hugging Face models.
-```bash
-# The base Llama-3.1-8B-Instruct model will be automatically downloaded to local cache
-# No internet connection required after initial setup
-# All LoRA adapters are included in this repository
-```
-### 3. Basic Usage
-```python
-from inference import FinLoRAPredictor
-# Initialize predictor with 8-bit model (recommended)
-predictor = FinLoRAPredictor(
-    model_name="sentiment_llama_3_1_8b_8bits_r8",
-    use_4bit=False
-)
-# Financial sentiment analysis
-sentiment = predictor.classify_sentiment(
-    "The company's quarterly earnings exceeded expectations by 20%."
-)
-print(f"Sentiment: {sentiment}")
-# Entity extraction
-entities = predictor.extract_entities(
-    "Apple Inc. reported revenue of $394.3 billion in 2022."
-)
-print(f"Entities: {entities}")
-```
-### 4. Run Complete Test
-```bash
-# Test all models (this will download the base Llama model if not present)
-python inference.py
-# Test specific model
-python -c "
-from inference import FinLoRAPredictor
-predictor = FinLoRAPredictor('sentiment_llama_3_1_8b_8bits_r8')
-print('Model loaded successfully!')
-"
-```
-## Usage Examples
-### Financial Sentiment Analysis
-```python
-predictor = FinLoRAPredictor("sentiment_llama_3_1_8b_8bits_r8")
-# Test cases
-test_texts = [
-    "Stock prices are soaring to new heights.",
-    "Revenue declined by 15% this quarter.",
-    "The company maintained stable performance."
-]
-for text in test_texts:
-    sentiment = predictor.classify_sentiment(text)
-    print(f"Text: {text}")
-    print(f"Sentiment: {sentiment}\n")
-```
-### Named Entity Recognition
-```python
-predictor = FinLoRAPredictor("ner_llama_3_1_8b_8bits_r8")
-text = "Apple Inc. reported revenue of $394.3 billion in 2022."
-entities = predictor.extract_entities(text)
-print(f"Entities: {entities}")
-```
-### XBRL Processing
-```python
-predictor = FinLoRAPredictor("xbrl_extract_llama_3_1_8b_8bits_r8")
-text = "Total assets: $1,234,567,890. Current assets: $456,789,123."
-xbrl_tags = predictor.extract_xbrl_tags(text)
-print(f"XBRL Tags: {xbrl_tags}")
-```
-### RAG-Enhanced Models
-```python
-# CFA RAG-enhanced model for financial knowledge
-predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")
-# Enhanced financial analysis with CFA knowledge
-response = predictor.generate_response(
-    "Explain the concept of discounted cash flow valuation"
-)
-print(f"CFA Response: {response}")
-# FinTagging RAG models for financial information extraction
-fintagging_predictor = FinLoRAPredictor("fintagging_combined_rag_llama_3_1_8b_8bits_r8")
-# Extract financial information with enhanced context
-entities = fintagging_predictor.extract_entities(
-    "Apple Inc. reported revenue of $394.3 billion in 2022."
-)
-print(f"Enhanced Entities: {entities}")
-```
-### APOLLO Models (Advanced Reasoning) 🚀
-**Important**: APOLLO models are designed for advanced numerical reasoning and should be used for complex financial calculations.
-```python
-# Load APOLLO model for advanced reasoning
-apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")
-# Financial calculations and reasoning
-calculation = apollo_predictor.generate_response(
-    "Calculate the present value of $10,000 received in 3 years with 5% annual discount rate"
-)
-print(f"APOLLO Calculation: {calculation}")
-# Complex financial analysis
-analysis = apollo_predictor.generate_response(
-    "Analyze the impact of a 2% interest rate increase on a 10-year bond with 3% coupon rate"
-)
-print(f"APOLLO Analysis: {analysis}")
-# Formula processing
-formula_result = apollo_predictor.generate_response(
-    "Solve: If a company has $1M revenue, 20% profit margin, and 10% growth rate, what's next year's profit?"
-)
-print(f"APOLLO Formula Result: {formula_result}")
-```
-### Multi-Layer LoRA Architecture (RAG + APOLLO)
-For maximum performance, you can combine RAG and APOLLO models:
-```python
-# Step 1: Load RAG model for knowledge retrieval
-rag_predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")
-# Step 2: Load APOLLO model for reasoning (this will be layered on top)
-apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")
-# Use for complex financial reasoning tasks
-complex_query = """
-Given the following financial data:
-- Revenue: $50M
-- Cost of Goods Sold: $30M
-- Operating Expenses: $15M
-- Tax Rate: 25%
-Calculate the net income and explain the calculation steps.
-"""
-response = apollo_predictor.generate_response(complex_query)
-print(f"Multi-Layer Response: {response}")
-```
-### Bloomberg-Enhanced Models (FPB & FIQA Specialized Tasks) 📊
-**Important**: Bloomberg models require special environment setup and are optimized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.
-#### Environment Setup for Bloomberg Models
-```bash
-# 1. Create conda environment using the provided configuration
-conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
-# 2. Activate the environment
-conda activate finenv
-# 3. Navigate to the Bloomberg evaluation directory
-cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
-```
-#### Testing Bloomberg Models on FPB and FIQA Datasets
-```bash
-# Run Bloomberg model evaluation
-python trytry1.py
-```
-**Configuration Notes for Testing:**
-1. **Dataset Configuration**: In `trytry1.py`, modify the `EVAL_FILES` line:
-   ```python
-   # Replace with your test datasets
-   EVAL_FILES = ["fiqa_test.jsonl", "fpb_test.jsonl"]
-   ```
-2. **Model Path Configuration**: For local testing, update the `BASE_DIR` in `trytry1.py`:
-   ```python
-   # For local Llama model deployment
-   BASE_DIR = "path/to/your/local/llama/model"
-   # For Hugging Face online model (original setting)
-   BASE_DIR = "d04e592bb4f6aa9cfee91e2e20afa771667e1d4b"
-   ```
-3. **Model Components**:
-   - `ADAPTER_DIR`: Points to the LoRA adapter (`finlora_lora_ckpt_llama_8bit_r8`)
-   - `HEADS_PATH`: Points to the model weights (`finlora_heads_llama_8bit_r8.pt`)
-#### Bloomberg Model Usage Example
-```python
-# Bloomberg models are specialized for FPB and FIQA tasks
-# They provide enhanced performance on financial sentiment analysis
-# and financial question answering compared to standard models
-# The evaluation script automatically handles:
-# - Model loading and configuration
-# - Dataset processing
-# - Performance metrics calculation
-# - Memory management for large models
-```
-### Memory-Efficient 4-bit Models
-```python
-# For users with limited GPU memory
-predictor = FinLoRAPredictor(
-    model_name="sentiment_llama_3_1_8b_4bits_r4",
-    use_4bit=True
-)
-# Same API as 8-bit models
-sentiment = predictor.classify_sentiment("The market is performing well.")
-```
-## Local Model Management
-### Model Storage
-- **Base Model**: Downloaded to `~/.cache/huggingface/transformers/`
-- **LoRA Adapters**: Stored in `models/` directory
-- **No Online Dependency**: All models run locally after initial download
-### Model Loading Process
-1. **Base Model**: Automatically downloaded on first use (~15GB)
-2. **LoRA Adapters**: Loaded from local `models/` directory
-3. **Quantization**: Applied during loading (8-bit or 4-bit)
-4. **Device Detection**: Automatically uses GPU if available, falls back to CPU
-### Performance Optimization
-```python
-# For better performance on GPU
-predictor = FinLoRAPredictor(
-    model_name="sentiment_llama_3_1_8b_8bits_r8",
-    use_4bit=False  # Use 8-bit for better performance
-)
-# For memory-constrained environments
-predictor = FinLoRAPredictor(
-    model_name="sentiment_llama_3_1_8b_4bits_r4",
-    use_4bit=True   # Use 4-bit for memory efficiency
-)
-```
-## Evaluation
-### For Competition Organizers
-This section provides guidance for evaluating the submitted models:
-#### 1. Quick Model Test
-```bash
-# Test if all models can be loaded successfully
-python test_submission.py
-```
-#### 2. Comprehensive Evaluation
-```bash
-# Run full evaluation on all models and datasets
-python comprehensive_evaluation.py
-# Check results
-cat comprehensive_evaluation_results.json
-```
-#### 3. Incremental Evaluation
-```bash
-# Run evaluation on missing tasks
-python incremental_evaluation.py
-# Check results
-cat incremental_evaluation_results.json
-```
-#### 4. APOLLO Model Testing
-```bash
-# Test APOLLO reasoning capabilities
-python -c "
-from inference import FinLoRAPredictor
-apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8')
-result = apollo.generate_response('Calculate 15% of $1000')
-print(f'APOLLO Test: {result}')
-"
-```
-#### 5. Bloomberg Model Testing (FPB & FIQA)
-```bash
-# Setup Bloomberg environment
-conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
-conda activate finenv
-# Navigate to Bloomberg evaluation directory
-cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
-# Configure test datasets in trytry1.py:
-# 1. Update EVAL_FILES = ["your_fiqa_test.jsonl", "your_fpb_test.jsonl"]
-# 2. Update BASE_DIR for local model path or keep original for Hugging Face
-# Run Bloomberg model evaluation
-python trytry1.py
-```
-## Performance Results
-The models have been evaluated on multiple financial datasets:
-| Task | Dataset | Model Type | F1 Score | Accuracy | Notes |
-|------|---------|------------|----------|----------|-------|
-| Sentiment Analysis | Financial Phrasebank | Base | 0.333 | 0.500 | Standard model |
-| NER | Financial NER | Base | 0.889 | 0.800 | High performance |
-| Classification | Headline Classification | Base | 0.697 | 0.700 | Good performance |
-| XBRL Processing | XBRL Tag Extraction | Base | 0.200 | 0.200 | Challenging task |
-| XBRL Processing | XBRL Tag Extraction | Specialized | 0.350 | 0.400 | Improved with specialized training |
-| Sentiment Analysis | FIQA SA | Base | 0.727 | 0.700 | Good performance |
-| Formula Calculation | ConvFinQA | APOLLO | 0.150 | 0.200 | Enhanced reasoning capability |
-| Financial Concept Linking | FinCL | RAG | 0.020 | 0.020 | Extremely challenging task |
-## Project Structure
-```
-finlora_hf_submission/
-├── models/                          # 8-bit LoRA model adapters (15 models)
-│   ├── sentiment_llama_3_1_8b_8bits_r8/
-│   ├── ner_llama_3_1_8b_8bits_r8/
-│   ├── headline_llama_3_1_8b_8bits_r8/
-│   ├── xbrl_extract_llama_3_1_8b_8bits_r8/
-│   ├── xbrl_term_llama_3_1_8b_8bits_r8/
-│   ├── financebench_llama_3_1_8b_8bits_r8/
-│   ├── finer_llama_3_1_8b_8bits_r8/
-│   ├── formula_llama_3_1_8b_8bits_r8/
-│   ├── cfa_rag_llama_3_1_8b_8bits_r8/          # RAG-enhanced CFA model
-│   ├── fintagging_combined_rag_llama_3_1_8b_8bits_r8/  # Combined RAG
-│   ├── fintagging_fincl_rag_llama_3_1_8b_8bits_r8/     # FinCL RAG
-│   ├── fintagging_finni_rag_llama_3_1_8b_8bits_r8/     # FinNI RAG
-│   ├── apollo_cfa_rag_llama_3_1_8b_8bits_r8/           # APOLLO reasoning layer
-│   ├── apollo_fintagging_combined_llama_3_1_8b_8bits_r8/ # APOLLO reasoning layer
-│   └── xbrl_train.jsonl-meta-llama-Llama-3.1-8B-Instruct-8bits_r8/
-├── Bloomberg_fpb_and_fiqa/          # Bloomberg specialized models for FPB & FIQA
-│   ├── finlora_heads_llama_8bit_r8.pt
-│   ├── finlora_lora_ckpt_llama_8bit_r8/
-│   ├── environment_contrasim.yml    # Conda environment configuration
-│   └── trytry1.py                   # Bloomberg model evaluation script
-├── models_4bit/                     # 4-bit LoRA model adapters (8 models)
-│   ├── sentiment_llama_3_1_8b_4bits_r4/
-│   ├── ner_llama_3_1_8b_4bits_r4/
-│   ├── headline_llama_3_1_8b_4bits_r4/
-│   ├── xbrl_extract_llama_3_1_8b_4bits_r4/
-│   ├── xbrl_term_llama_3_1_8b_4bits_r4/
-│   ├── financebench_llama_3_1_8b_4bits_r4/
-│   ├── finer_llama_3_1_8b_4bits_r4/
-│   └── formula_llama_3_1_8b_4bits_r4/
-├── testdata/                        # Evaluation datasets
-│   ├── FinCL-eval-subset.csv
-│   └── FinNI-eval-subset.csv
-├── rag_system/                      # RAG system components
-├── inference.py                     # Main inference script
-├── comprehensive_evaluation.py      # Full evaluation script
-├── incremental_evaluation.py        # Incremental evaluation
-├── robust_incremental.py           # Robust evaluation
-├── missing_tests.py                # Missing test detection
-├── requirements.txt                 # Python dependencies
-└── README.md                       # This file
-```
-## Environment Requirements
-### Minimum Requirements (CPU Mode)
-- Python 3.8+
-- PyTorch 2.0+
-- 8GB RAM
-- No GPU required
-### Recommended Requirements (GPU Mode)
-- Python 3.9+
-- PyTorch 2.1+
-- CUDA 11.8+ (for NVIDIA GPUs)
-- 16GB+ GPU memory
-- 32GB+ RAM
-### Installation Instructions
-```bash
-# 1. Clone or download this repository
-# 2. Install dependencies
-pip install -r requirements.txt
-# 3. For GPU support (optional but recommended)
-pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
-# 4. Verify installation
-python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"
-```
-### Troubleshooting
-**If you encounter memory issues:**
-- Use 4-bit models instead of 8-bit models
-- Reduce batch size in inference
-- Use CPU mode if GPU memory is insufficient
-**If models fail to load:**
-- Ensure all model files are present in the correct directories
-- Check that the base model (Llama-3.1-8B-Instruct) can be downloaded from HuggingFace
-- Verify internet connection for initial model download
-**Important Notes for Competition Organizers:**
-- The base model (Llama-3.1-8B-Instruct) will be automatically downloaded from HuggingFace on first use (~15GB)
-- All LoRA adapters are included in this submission and do not require additional downloads
-- Models work in both CPU and GPU modes, with automatic device detection
-- APOLLO models provide enhanced reasoning capabilities for complex financial tasks
-- All models run locally without requiring ongoing internet connection
-## Model Details
-### Training Configuration
-- **LoRA Rank**: 8
-- **LoRA Alpha**: 16
-- **Learning Rate**: 1e-4
-- **Batch Size**: 4
-- **Epochs**: 3-5
-- **Quantization**: 8-bit (BitsAndBytes) / 4-bit (NF4)
-### Training Data
-- Financial Phrasebank
-- FinGPT datasets (NER, Headline, XBRL)
-- BloombergGPT financial datasets
-- Custom financial text datasets
-- APOLLO reasoning datasets for numerical calculations
-## Citation
-If you use this work in your research, please cite:
-```bibtex
-@article{finlora2024,
-  title={FinLoRA: Financial Large Language Models with LoRA Adaptation},
-  author={Your Name},
-  journal={Financial AI Conference},
-  year={2024}
-}
-```
-## License
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-## Contributing
-Contributions are welcome! Please feel free to submit a Pull Request.
-## Contact
-For questions and support, please open an issue or contact [[email protected]](mailto:[email protected]).
-## Submission Summary
-### What's Included
-- **17 Total Models**: 15 8-bit models (9 original + 4 RAG-enhanced + 2 APOLLO) + 8 4-bit models
-- **Complete Evaluation Results**: Comprehensive and incremental evaluation results
-- **RAG-Enhanced Models**: CFA and FinTagging models with enhanced knowledge
-- **APOLLO Reasoning**: Advanced numerical reasoning and calculation capabilities
-- **Cross-Platform Support**: Works on CPU, GPU, and various memory configurations
-- **Local Execution**: All models run locally without online dependencies
-- **Ready-to-Use**: All dependencies specified, automatic device detection
-### Quick Start for Competition Organizers
-1. Install dependencies: `pip install -r requirements.txt`
-2. Test submission: `python test_submission.py`
-3. Run evaluation: `python comprehensive_evaluation.py`
-4. Test APOLLO reasoning: `python -c "from inference import FinLoRAPredictor; apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8'); print(apollo.generate_response('Calculate 10% of 500'))"`
-5. Test Bloomberg models (FPB & FIQA):
-   ```bash
-   conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
-   conda activate finenv
-   cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
-   # Configure EVAL_FILES and BASE_DIR in trytry1.py
-   python trytry1.py
-   ```
-6. Check results: `cat comprehensive_evaluation_results.json`
-### Model Categories
-- **Financial NLP**: Sentiment, NER, Classification, XBRL processing
-- **RAG-Enhanced**: CFA knowledge and FinTagging with retrieval augmentation
-- **APOLLO Reasoning**: Advanced numerical calculations and financial reasoning
-- **Memory Options**: Both 8-bit and 4-bit quantized versions available
-## Acknowledgments
-- Meta for the Llama-3.1-8B-Instruct base model
-- Hugging Face for the transformers and PEFT libraries
-- The financial NLP community for datasets and benchmarks
-- APOLLO reasoning framework for enhanced numerical capabilities