Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/Fig1.png filter=lfs diff=lfs merge=lfs -text
+demo.wav filter=lfs diff=lfs merge=lfs -text
+hi-Priya_woman.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,207 @@
+---
+license: mit
+language:
+- hi
+- en
+base_model: vibevoice/VibeVoice-7B
+tags:
+- text-to-speech
+- hindi
+- lora
+- peft
+- audio-generation
+- tts
+pipeline_tag: text-to-speech
+---
+
+# VibeVoice-Hindi-LoRA
+
+## Model Description
+
+This repository contains **LoRA (Low-Rank Adaptation) weights** for fine-tuning the VibeVoice-7B model for **Hindi text-to-speech synthesis**. These adapter weights enable efficient fine-tuning of the base VibeVoice model to generate high-quality, natural-sounding Hindi speech without requiring a full model retrain.
+
+LoRA is a parameter-efficient fine-tuning technique that adds trainable rank decomposition matrices to the model while keeping the original pre-trained weights frozen. This results in significantly smaller model sizes and faster training times.
+
+### Base Model
+- **Base Model:** [vibevoice/VibeVoice-7B](https://huggingface.co/vibevoice/VibeVoice-7B)
+- **Architecture:** Qwen2.5-7B + Acoustic/Semantic Tokenizers + Diffusion Head
+- **Original Training:** English and Chinese speech synthesis
+
+### Fine-tuning Details
+- **Target Language:** Hindi
+- **Method:** LoRA (Low-Rank Adaptation)
+- **Fine-tuned Components:** 
+  - LLM backbone (LoRA adapters)
+  - Diffusion head (full fine-tuning)
+  - Acoustic and Semantic connectors
+
+## What's Included
+
+This repository contains:
+- `adapter_config.json` - LoRA configuration file
+- `adapter_model.safetensors` - LoRA adapter weights for the LLM backbone
+- `diffusion_head/` - Full diffusion head weights fine-tuned for Hindi
+- `acoustic_connector/` - Acoustic connector weights
+- `semantic_connector/` - Semantic connector weights
+
+## Usage
+
+### Demo and Inference Code
+
+For complete inference examples and demos, please refer to:
+
+- **Community Repository:** [vibevoice-community/VibeVoice](https://github.com/vibevoice-community/VibeVoice)
+- **ComfyUI Integration:** [Enemyx-net/VibeVoice-ComfyUI](https://github.com/Enemyx-net/VibeVoice-ComfyUI)
+
+### Hindi Inference
+
+### Using with VibeVoice Inference Pipeline
+
+```bash
+# Clone the community repository
+git clone https://github.com/vibevoice-community/VibeVoice.git
+cd VibeVoice
+
+# Install dependencies
+uv pip install -e .
+```
+
+#### With voice cloning (Recommended for Hindi models):
+```bash
+python demo/inference_from_file.py \
+--checkpoint_path "./vibevoice-hindi-lora" \
+--txt_path "./example_hindi_script.txt" \
+--model_path "vibevoice/VibeVoice-7B" \
+--speaker_names hi-Priya_woman \
+--seed 42
+```
+
+#### With multiple speakers:
+```bash
+python demo/inference_from_file.py \
+--checkpoint_path "./vibevoice-hindi-lora" \
+--txt_path "./example_hindi_script.txt" \
+--model_path "vibevoice/VibeVoice-7B" \
+--speaker_names "Speaker1" "Speaker2" \
+--cfg_scale 1.3
+```
+
+**Note**: For voice cloning, ensure you have corresponding voice files in `demo/voices/` directory. The script will automatically map speaker names to voice files.
+
+**Key points for Hindi inference:**
+- **With voice cloning**: Specify `--speaker_names` to map speakers to voice files
+- Use `--model_path "vibevoice/VibeVoice-7B"` to match your checkpoint size
+- The model will use the provided voice samples for generation
+- Voice samples are loaded and used for voice cloning
+
+**Voice Cloning Setup:**
+- Place voice sample files in `demo/voices/` directory
+- **Required file**: `hi-Priya_woman.wav` - Hindi female voice sample
+- Use descriptive filenames like `hindi-speaker1.wav`, `hindi-speaker2.wav`
+- The script will automatically map speaker names to voice files
+- Voice cloning works best with high-quality, clear voice samples
+
+**Model Architecture Compatibility:**
+- Ensure your checkpoint matches the model size (7B checkpoint requires `--model_path "vibevoice/VibeVoice-7B"`)
+
+## Hindi Inference with Gradio Demo
+
+For interactive Hindi text generation with a model:
+
+### Launch the Gradio Demo:
+```bash
+python demo/gradio_demo.py \
+--model_path "vibevoice/VibeVoice-7B" \
+--checkpoint_path "./vibevoice-hindi-lora" \
+--device cuda
+```
+
+### Using the Web Interface:
+1. Enter your Hindi script in the text area
+2. Select speakers (use `hi-Priya_woman` for Hindi voice)
+3. Click "🚀 Generate Podcast"
+
+**Key points:**
+- Voice samples are loaded and used for voice cloning
+- The model will use the provided voice samples for generation
+- Real-time streaming audio generation is supported
+- Works with both 1.5B and 7B models (ensure checkpoint matches model size)
+- Make sure `hi-Priya_woman.wav` is in the `demo/voices/` directory
+
+## Demo
+
+### Sample Output:
+<audio controls src="https://huggingface.co/tarun7r/vibevoice-hindi-lora/resolve/main/demo.wav" style="width: 100%;"></audio>
+
+**Important Note:** The quality of the generated audio depends heavily on the reference voice file you provide in the `demo/voices/` directory. For best results:
+- Use high-quality, clear voice samples
+- Ensure the reference voice matches the desired speaking style
+- Longer reference samples (10-30 seconds) generally produce better results
+- The voice characteristics of the reference sample will be transferred to the generated speech
+
+## Model Capabilities
+
+- **Text-to-Speech:** Convert Hindi text to natural-sounding speech
+- **Multi-speaker Support:** Generate speech with multiple distinct speakers
+- **Long-form Audio:** Synthesize extended audio sequences
+- **Expressive Speech:** Maintain natural prosody and intonation for Hindi
+
+## Responsible Usage
+
+### Direct intended uses
+The VibeVoice model is limited to research purpose use exploring highly realistic audio dialogue generation detailed in the tech report.
+
+### Out-of-scope uses
+Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by MIT License. Use to generate any text transcript. Furthermore, this release is not intended or licensed for any of the following scenarios:
+
+- Voice impersonation without explicit, recorded consent – cloning a real individual's voice for satire, advertising, ransom, social‑engineering, or authentication bypass.
+- Disinformation or impersonation – creating audio presented as genuine recordings of real people or events.
+- Real‑time or low‑latency voice conversion – telephone or video‑conference "live deep‑fake" applications.
+- Unsupported language – the model is trained only on English and Chinese data; outputs in other languages are unsupported and may be unintelligible or offensive.
+- Generation of background ambience, Foley, or music – VibeVoice is speech‑only and will not produce coherent non‑speech audio.
+
+## Risks and limitations
+While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model. Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content. English and Chinese only: Transcripts in language other than English or Chinese may result in unexpected audio outputs. Non-Speech Audio: The model focuses solely on speech synthesis and does not handle background noise, music, or other sound effects. Overlapping Speech: The current model does not explicitly model or generate overlapping speech segments in conversations.
+
+## Recommendations
+We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.
+
+To mitigate the risks of misuse, we have: Embedded an audible disclaimer (e.g. "This segment was generated by AI") automatically into every synthesized audio file. Added an imperceptible watermark to generated audio so third parties can verify VibeVoice provenance. Please see contact information at the end of this model card. Logged inference requests (hashed) for abuse pattern detection and publishing aggregated statistics quarterly. Users are responsible for sourcing their datasets legally and ethically. This may include securing appropriate rights and/or anonymizing data prior to use with VibeVoice. Users are reminded to be mindful of data privacy concerns.
+
+## License & Redistribution Notice
+
+This model is released under the **MIT License**, consistent with the base VibeVoice model.
+
+**Redistribution Notice:**
+This repository contains model weights derived from [microsoft/VibeVoice-Large](https://www.modelscope.cn/models/microsoft/VibeVoice-Large), which is licensed under the MIT License. The MIT License permits redistribution and derivative works.
+
+My understanding of the MIT License, which is consistent with the broader open-source community's consensus, is that it grants the right to distribute copies of the software and its derivatives. Therefore, I am lawfully exercising the right to redistribute this model.
+
+If you are a rights holder and believe this understanding of the license is incorrect, please submit a DMCA complaint to Hugging Face at [email protected]
+
+
+## Acknowledgments
+
+- **Base Model:** Microsoft Research for the original VibeVoice model
+- **Fine-tuning Code:** [vibevoice-community/VibeVoice](https://github.com/vibevoice-community/VibeVoice) for the training framework
+- **Training Infrastructure:** [Nebius](https://lightning.ai/pricing/) H100 GPU cluster
+- **Community:** Hugging Face and the open-source AI community
+- **Framework:** Built on Qwen2.5, Transformers, and PEFT libraries
+
+## Contact
+
+**Actively seeking opportunities as an ML Engineer II / Data Scientist II**
+
+For questions, issues, or collaboration:
+- **GitHub:** [tarun7r](https://github.com/tarun7r)
+- **Hugging Face:** [tarun7r](https://huggingface.co/tarun7r)
+- **Base model contact:** [email protected]
+
+### Other Key Projects
+- **[SpeechAlgo](https://github.com/tarun7r/SpeechAlgo)** - Comprehensive Speech Processing Algorithms Library
+- **[Vocal-Agent](https://github.com/tarun7r/Vocal-Agent)** - Cascading voice assistant with real-time speech recognition
+- **[Finance-Llama-8B](https://huggingface.co/tarun7r/Finance-Llama-8B)** - Financial domain fine-tuned Llama model
+
+---
+
+**Note:** This is a research model. Please use responsibly and in compliance with applicable laws and ethical guidelines.

acoustic_connector/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3339274373c7b1becac4dad05ab36fd4119bc742d9dfe3773215e9840a68c2fa
+size 26173211

adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a3f1fb3e3e9929b90e80c412e8279825597dd688b86b5ce1bb9fd4609d87d10
+size 161530840

demo.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50c26c8b6532558303ebb3c39d8ae96bf65b575b666147b0e0b73a0ee96d956b
+size 2361644

diffusion_head/config.json

@@ -0,0 +1,20 @@
+{
+  "architectures": [
+    "VibeVoiceDiffusionHead"
+  ],
+  "ddpm_batch_mul": 4,
+  "ddpm_beta_schedule": "cosine",
+  "ddpm_num_inference_steps": 20,
+  "ddpm_num_steps": 1000,
+  "diffusion_type": "ddpm",
+  "head_ffn_ratio": 3.0,
+  "head_layers": 4,
+  "hidden_size": 3584,
+  "latent_size": 64,
+  "model_type": "vibevoice_diffusion_head",
+  "prediction_type": "v_prediction",
+  "rms_norm_eps": 1e-05,
+  "speech_vae_dim": 64,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3"
+}

diffusion_head/diffusion_head_full.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc3d8f80c59ee7e89bc18751bbf124a2f46c79d040c97c3edcb8fb7aa28432ab
+size 1338678485

diffusion_head/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3f155080fead8b2a06852f3016280e80191651b779bb54b15b53c8e1d897084
+size 1338669752

diffusion_head_full.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc3d8f80c59ee7e89bc18751bbf124a2f46c79d040c97c3edcb8fb7aa28432ab
+size 1338678485

hi-Priya_woman.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:343395d24039183143b0ab79530d4f62ff87f46ec9aec72fed5164155321a73b
+size 218046

semantic_connector/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdfd59ae14d127eb099f57e78b6e5a475668672038a184195b41846ae3817ca3
+size 26631963