thumbnail
Browse files
README.md
CHANGED
|
@@ -44,59 +44,27 @@ virtualenv --python=python3 ~/.envs/.my_env
|
|
| 44 |
source ~/.envs/.my_env/bin/activate
|
| 45 |
cd shift/
|
| 46 |
pip install -r requirements.txt
|
| 47 |
-
```
|
| 48 |
-
|
| 49 |
-
Start API
|
| 50 |
|
| 51 |
-
```
|
| 52 |
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py
|
| 53 |
```
|
| 54 |
|
| 55 |
-
The following need `api.py` to be already running `on a tmux session`.
|
| 56 |
-
|
| 57 |
## Landscape 2 Soundscape
|
| 58 |
|
|
|
|
|
|
|
| 59 |
```python
|
| 60 |
# TTS & soundscape - overlay to .mp4
|
| 61 |
python landscape2soundscape.py
|
| 62 |
```
|
| 63 |
|
| 64 |
-
#
|
| 65 |
-
|
| 66 |
-
Substitute Native voice via TTS
|
| 67 |
|
| 68 |
-
[](https://www.youtube.com/watch?v=tmo2UbKYAqc)
|
| 69 |
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
Same video where Native voice is replaced with English TTS voice with similar emotion
|
| 73 |
|
| 74 |
|
| 75 |
[](https://www.youtube.com/watch?v=geI1Vqn4QpY)
|
| 76 |
|
| 77 |
-
|
| 78 |
-
<details>
|
| 79 |
-
<summary>
|
| 80 |
-
|
| 81 |
-
Video dubbing from subtitles `.srt`
|
| 82 |
-
|
| 83 |
-
</summary>
|
| 84 |
-
|
| 85 |
-
## Video Dubbing
|
| 86 |
-
|
| 87 |
-
[](https://www.youtube.com/watch?v=bpt7rOBENcQ)
|
| 88 |
-
|
| 89 |
-
Generate dubbed video:
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
```python
|
| 93 |
-
python tts.py --text assets/head_of_fortuna_en.srt --video assets/head_of_fortuna.mp4
|
| 94 |
-
|
| 95 |
-
```
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
</details>
|
| 99 |
-
|
| 100 |
## Joint Application of D3.1 & D3.2
|
| 101 |
|
| 102 |
<a href="https://youtu.be/wWC8DpOKVvQ" rel="Subtitles to Video"></a>
|
|
@@ -128,6 +96,8 @@ For SHIFT demo / Collaboration with [SMB](https://www.smb.museum/home/)
|
|
| 128 |
|
| 129 |
[](https://youtu.be/2YjxAPkdXIc)
|
| 130 |
|
|
|
|
|
|
|
| 131 |
[](https://www.youtube.com/watch?v=Y8QyYUgLaCg)
|
| 132 |
|
| 133 |
|
|
|
|
| 44 |
source ~/.envs/.my_env/bin/activate
|
| 45 |
cd shift/
|
| 46 |
pip install -r requirements.txt
|
|
|
|
|
|
|
|
|
|
| 47 |
|
|
|
|
| 48 |
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py
|
| 49 |
```
|
| 50 |
|
|
|
|
|
|
|
| 51 |
## Landscape 2 Soundscape
|
| 52 |
|
| 53 |
+
The following needs `api.py` to be already running `on a tmux session`.
|
| 54 |
+
|
| 55 |
```python
|
| 56 |
# TTS & soundscape - overlay to .mp4
|
| 57 |
python landscape2soundscape.py
|
| 58 |
```
|
| 59 |
|
| 60 |
+
# Videos / Examples
|
|
|
|
|
|
|
| 61 |
|
|
|
|
| 62 |
|
| 63 |
+
Video where Native voice is replaced with English TTS voice
|
|
|
|
|
|
|
| 64 |
|
| 65 |
|
| 66 |
[](https://www.youtube.com/watch?v=geI1Vqn4QpY)
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
## Joint Application of D3.1 & D3.2
|
| 69 |
|
| 70 |
<a href="https://youtu.be/wWC8DpOKVvQ" rel="Subtitles to Video"></a>
|
|
|
|
| 96 |
|
| 97 |
[](https://youtu.be/2YjxAPkdXIc)
|
| 98 |
|
| 99 |
+
[](https://youtu.be/BhMh02knkco)
|
| 100 |
+
|
| 101 |
[](https://www.youtube.com/watch?v=Y8QyYUgLaCg)
|
| 102 |
|
| 103 |
|
landscape2soundscape.py
CHANGED
|
@@ -62,7 +62,7 @@ DESCRIPTIONS = [
|
|
| 62 |
'03_Schinkel_WS200-002.jpg',
|
| 63 |
'03_Schinkel_WS200-002.txt',
|
| 64 |
'Arriving at the valley on galloping horses',
|
| 65 |
-
'
|
| 66 |
'en_US/hifi-tts_low#6097',
|
| 67 |
],
|
| 68 |
#
|
|
@@ -146,7 +146,7 @@ SILENT_VIDEO = '_silent_video.mp4'
|
|
| 146 |
# SILENT CLIP
|
| 147 |
|
| 148 |
|
| 149 |
-
for img, text, scene, title, voice in DESCRIPTIONS[2:
|
| 150 |
|
| 151 |
|
| 152 |
|
|
|
|
| 62 |
'03_Schinkel_WS200-002.jpg',
|
| 63 |
'03_Schinkel_WS200-002.txt',
|
| 64 |
'Arriving at the valley on galloping horses',
|
| 65 |
+
'K. Schinkel Gotische Kirche Auf Einem Felsen 1815',
|
| 66 |
'en_US/hifi-tts_low#6097',
|
| 67 |
],
|
| 68 |
#
|
|
|
|
| 146 |
# SILENT CLIP
|
| 147 |
|
| 148 |
|
| 149 |
+
for img, text, scene, title, voice in DESCRIPTIONS[2:4]:
|
| 150 |
|
| 151 |
|
| 152 |
|
uc_spk_Landscape2Soundscape_Masterpieces_pics/thumb____03_Schinkel_WS200-002.jpg
ADDED
|
|