{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "6e724932-ad86-4006-9d1b-82001b3ff929", "metadata": {}, "outputs": [], "source": [ "# !git clone https://github.com/yl4579/PL-BERT.git\n", "!cd /home/cristic/PL-BERT/" ] }, { "cell_type": "markdown", "id": "5018600f-f529-4a93-a089-5708228d2cdc", "metadata": {}, "source": [ "Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions" ] }, { "cell_type": "markdown", "id": "d21a9dc3-db0f-4828-90d0-0467f597d35e", "metadata": {}, "source": [ "Large-scale pre-trained language models have been shown to be helpful in improving the naturalness of text-to-speech (TTS) models by enabling them to produce more naturalistic prosodic patterns. However, these models are usually word-level or sup-phoneme-level and jointly trained with phonemes, making them inefficient for the downstream TTS task where only phonemes are needed. In this work, we propose a phoneme-level BERT (PL-BERT) with a pretext task of predicting the corresponding graphemes along with the regular masked phoneme predictions. Subjective evaluations show that our phoneme-level BERT encoder has significantly improved the mean opinion scores (MOS) of rated naturalness of synthesized speech compared with the state-of-the-art (SOTA) StyleTTS baseline on out-of-distribution (OOD) texts.\n" ] }, { "cell_type": "markdown", "id": "ab3ecb2e-255b-4d5d-9e40-cf326c1fa3be", "metadata": {}, "source": [ " Preprocessing Wikipedia (Romanian) dataset" ] }, { "cell_type": "code", "execution_count": 2, "id": "8f26172d-5970-4534-a146-b0668f0448d6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/cristic/PL-BERT\n" ] } ], "source": [ "cd PL-BERT" ] }, { "cell_type": "code", "execution_count": 3, "id": "af2e2f8e-d111-430a-a5e4-e151a2e35b7b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mConfigs\u001b[0m/ model.py README.md token_maps.pkl\n", "\u001b[01;34mconverters\u001b[0m/ phonemize.py simple_loader.py train.ipynb\n", "dataloader.py preprocess.ipynb text_normalize.py utils.py\n", "LICENSE \u001b[01;34m__pycache__\u001b[0m/ text_utils.py\n" ] } ], "source": [ "ls" ] }, { "cell_type": "code", "execution_count": 4, "id": "38fbd17f-e813-4823-bb6b-310fe2666f8c", "metadata": {}, "outputs": [], "source": [ "import yaml\n", "\n", "config_path = \"Configs/config.yml\" # you can change it to anything else\n", "config = yaml.safe_load(open(config_path))" ] }, { "cell_type": "code", "execution_count": 6, "id": "03ef081d-44c1-4bb5-9296-e4d580514741", "metadata": {}, "outputs": [], "source": [ "# !pip install phonemize" ] }, { "cell_type": "code", "execution_count": 6, "id": "d35f5a90-3a98-4928-849f-a2681630ee46", "metadata": {}, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'ntlk'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[6], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mntlk\u001b[39;00m\n", "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'ntlk'" ] } ], "source": [ "# import ntlk" ] }, { "cell_type": "code", "execution_count": 7, "id": "a4735b77-e056-429d-91a9-44394813476e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: nltk in /home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages (3.9.2)\n", "Requirement already satisfied: click in /home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages (from nltk) (8.1.8)\n", "Requirement already satisfied: joblib in /home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages (from nltk) (1.5.2)\n", "Requirement already satisfied: regex>=2021.8.3 in /home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages (from nltk) (2025.10.23)\n", "Requirement already satisfied: tqdm in /home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages (from nltk) (4.67.1)\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "pip install nltk" ] }, { "cell_type": "markdown", "id": "742e7584-2554-4639-87d0-1bca5a223355", "metadata": {}, "source": [ "!pip install singleton-decorator accelerate phonemizer sacremoses pebble\n", "# !pip install datasets==3.6.0" ] }, { "cell_type": "code", "execution_count": 9, "id": "d583bf4d-361a-4aa2-bcac-8bce1ab06c97", "metadata": {}, "outputs": [], "source": [ "# pip install transformers" ] }, { "cell_type": "code", "execution_count": 10, "id": "a9dd3f4a-e497-4962-8c26-8a6346b1aa1b", "metadata": {}, "outputs": [], "source": [ "from phonemize import phonemize" ] }, { "cell_type": "markdown", "id": "b94dcd8d-721f-4696-acb8-23d3c2fbfede", "metadata": {}, "source": [ "install backend on operating system" ] }, { "cell_type": "code", "execution_count": 11, "id": "83d8b337-7c44-42c2-bbc4-6bf579e31869", "metadata": {}, "outputs": [], "source": [ "import phonemizer\n", "global_phonemizer = phonemizer.backend.EspeakBackend(language='ro', preserve_punctuation=True, with_stress=True)" ] }, { "cell_type": "code", "execution_count": 12, "id": "ad27b5bd-aad7-4123-917d-36ec2b96e419", "metadata": {}, "outputs": [], "source": [ "import os\n", "# Set the environment variable to bypass the security check\n", "os.environ['TRUST_REMOTE_CODE'] = 'True'" ] }, { "cell_type": "code", "execution_count": 13, "id": "2b77d1b5-b5ed-4a4f-8a33-85394a9d4567", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/cristic/anaconda3/envs/tts_p40/lib/python3.9/site-packages/huggingface_hub/file_download.py:942: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n", " warnings.warn(\n" ] } ], "source": [ "# from transformers import TransfoXLTokenizer\n", "# tokenizer = TransfoXLTokenizer.from_pretrained(config['dataset_params']['tokenizer'])\n", "\n", "from transformers import AutoTokenizer\n", "\n", "# Recommended replacement for Transfo-XL is XLNet\n", "tokenizer_name = \"xlnet-base-cased\" \n", "# Or: \"transfo-xl-wt103\" if you need the exact vocabulary, but it will use the safer AutoTokenizer implementation.\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)" ] }, { "cell_type": "code", "execution_count": 14, "id": "7c871269-cf5f-41d6-9771-de2e66788795", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "# dataset = load_dataset(\"wikipedia\", \"20251202.ro\")['train'] # you can use other version of this dataset" ] }, { "cell_type": "code", "execution_count": 15, "id": "2e26fcdb-9b18-4ca2-ac13-f20c2951b022", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "\n", "# # Use \"20220301.ro\" or the current supported snapshot (e.g., \"20231101.ro\")\n", "# # The format is often YYYYMMDD.lang\n", "# dataset = load_dataset(\"wikipedia\", \"20231101.ro\", language=\"ro\", date=\"20231101\")['train']" ] }, { "cell_type": "code", "execution_count": 16, "id": "12b4c74e-65f0-48ef-9b15-c592bee0eab8", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "\n", "# # Use the '20220301' version, which is known to be stable and available.\n", "# # 'ro' is the name of the configuration/language.\n", "# dataset = load_dataset(\"wikipedia\", \"20220301\", language=\"ro\")['train']\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "8ae26829-ee36-4199-8551-c8800152c08b", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "\n", "# # '20220301' is the version snapshot\n", "# # 'ro' is the language/revision identifier\n", "# try:\n", "# dataset = load_dataset(\n", "# \"wikipedia\",\n", "# name=\"20220301\",\n", "# revision=\"ro\"\n", "# )['train']\n", " \n", "# print(\"Successfully loaded the Romanian Wikipedia dataset.\")\n", "# print(f\"Number of training examples: {len(dataset)}\")\n", "\n", "# except Exception as e:\n", "# print(f\"An error occurred: {e}\")" ] }, { "cell_type": "code", "execution_count": 18, "id": "26ca54ae-3e32-43b8-bacc-ec355669dffc", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "\n", "# # The language argument is critical for the new loader mechanism\n", "# try:\n", "# dataset = load_dataset(\n", "# \"wikipedia\",\n", "# name=\"20220301\",\n", "# language=\"ro\" # <-- Use the 'language' argument\n", "# )['train']\n", " \n", "# print(\"Successfully loaded the Romanian Wikipedia dataset.\")\n", "# print(f\"Number of training examples: {len(dataset)}\")\n", "# print(dataset)\n", "\n", "# except Exception as e:\n", "# print(f\"A new error occurred: {e}\")" ] }, { "cell_type": "code", "execution_count": 19, "id": "45ecd15c-fea6-46a9-9500-b37becbdbb89", "metadata": {}, "outputs": [], "source": [ "# !huggingface-cli delete-cache" ] }, { "cell_type": "code", "execution_count": 20, "id": "f89d3890-bc7d-43b7-b4f9-008ca9ddec0a", "metadata": {}, "outputs": [], "source": [ "# !hf cache delete --disable-tui" ] }, { "cell_type": "code", "execution_count": null, "id": "6ce17179-6234-4495-aeef-79706bbfd546", "metadata": {}, "outputs": [], "source": [ "# pip install datasets==2.16.1 --no-deps" ] }, { "cell_type": "code", "execution_count": 16, "id": "f0e36c3b-1f34-4dec-b29f-18ed8d9a4b1f", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading data: 100%|██████████| 41/41 [01:57<00:00, 2.87s/files]\n", "Generating train split: 100%|██████████| 6407814/6407814 [02:33<00:00, 41754.70 examples/s]\n" ] } ], "source": [ "from datasets import load_dataset\n", "\n", "ds = load_dataset(\"wikimedia/wikipedia\", \"20231101.en\")\n" ] }, { "cell_type": "code", "execution_count": 20, "id": "d2664fd4-066d-4d10-92c4-48a092b2809c", "metadata": {}, "outputs": [], "source": [ "# from datasets import load_dataset\n", "\n", "dataset = load_dataset(\"wikimedia/wikipedia\", \"20231101.ro\")[\"train\"]\n" ] }, { "cell_type": "code", "execution_count": 19, "id": "2094a652-de15-4bc6-be44-23f5b5aeb639", "metadata": {}, "outputs": [], "source": [ "root_directory = \"./wiki_phoneme\" # set up root directory for multiprocessor processing" ] }, { "cell_type": "code", "execution_count": 21, "id": "e8598554-f244-4a6a-ac4b-efa6e6dd4a5d", "metadata": {}, "outputs": [], "source": [ "import os\n", "num_shards = 50000\n", "\n", "def process_shard(i):\n", " directory = root_directory + \"/shard_\" + str(i)\n", " if os.path.exists(directory):\n", " print(\"Shard %d already exists!\" % i)\n", " return\n", " print('Processing shard %d ...' % i)\n", " shard = dataset.shard(num_shards=num_shards, index=i)\n", " processed_dataset = shard.map(lambda t: phonemize(t['text'], global_phonemizer, tokenizer), remove_columns=['text'])\n", " if not os.path.exists(directory):\n", " os.makedirs(directory)\n", " processed_dataset.save_to_disk(directory)" ] }, { "cell_type": "code", "execution_count": 22, "id": "3b34cbb0-6089-4755-9b42-b11ecc478347", "metadata": {}, "outputs": [], "source": [ "from pebble import ProcessPool\n", "from concurrent.futures import TimeoutError" ] }, { "cell_type": "markdown", "id": "36a0cbae-28db-4a79-a2ac-717d1294e79e", "metadata": {}, "source": [ "Note: You will need to run the following cell multiple times to process all shards because some will fail. Depending on how fast you process each shard, you will need to change the timeout to a longer value to make more shards processed before being killed." ] }, { "cell_type": "code", "execution_count": 23, "id": "406fc5f5-680c-49fa-98ca-1cdda5be602a", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processing shard 0 ...Processing shard 15 ...\n", "Processing shard 1 ...\n", "Processing shard 2 ...\n", "Processing shard 4 ...Processing shard 3 ...\n", "Processing shard 5 ...Processing shard 6 ...\n", "Processing shard 7 ...\n", "Processing shard 8 ...Processing shard 9 ...\n", "\n", "Processing shard 10 ...\n", "Processing shard 11 ...\n", "Processing shard 12 ...\n", "Processing shard 13 ...\n", "Processing shard 14 ...\n", "Processing shard 17 ...\n", "\n", "\n", "Processing shard 16 ...Processing shard 18 ...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Map: 0%| | 0/9 [00:00