--- base_model: - Qwen/Qwen-Image license: cc-by-4.0 tags: - lora --- # Qwen-Image-SynthSat-v1.0-warmup-stage1 The idea of the "warmup stages" is to sort of pretrain the LoRA weights so that they can be further trained with more refined data, but maybe have a boost in overall knowledge and stuff. ***This stage is 256p, which is what Qwen-Image originally pretrained with.*** ## Training Stages - ***[SynthSat-v1.0-warmup-stage1](https://huggingface.co/PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage1) is 256p only, using freshly initialized weights.*** - [SynthSat-v1.0-warmup-stage2](https://huggingface.co/PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage2) is 640p only, re-using the weights from [SynthSat-v1.0-warmup-stage1](https://huggingface.co/PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage1). - [SynthSat-v1.0-warmup-stage2.5-nsfw](https://huggingface.co/PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage2.5-nsfw) is 256p+640p, re-using the weights from [SynthSat-v1.0-warmup-stage2-step1400](https://huggingface.co/PJMixers-Images/Qwen-Image-SynthSat-v1.0-warmup-stage2/blob/main/Qwen-Image-SynthSat-v1.0-warmup-stage2-step00001400.safetensors), using only NSFW sources (which is a mix of NSFW+SFW). - Next stage is either warmup-stage3 doing 640p+1328p or v1.0 doing 640p+1328p+1536p on a more refined dataset. ## Current Stage Details ``` INFO:musubi_tuner.hv_train_network:Load dataset config from /media/xzuyn/Toshiba1/musubi-stuff/dataset_configs/SynthSat-v1.0-warmup-stage1.toml INFO:musubi_tuner.dataset.image_video_dataset:glob images in /media/xzuyn/NVMe/Datasets/Images/000_SynthSatWarmup INFO:musubi_tuner.dataset.image_video_dataset:found 26915 images INFO:musubi_tuner.dataset.config_utils:[Dataset 0] is_image_dataset: True resolution: (256, 256) batch_size: 16 num_repeats: 1 caption_extension: ".txt" enable_bucket: True bucket_no_upscale: True cache_directory: "/media/xzuyn/NVMe/LClones/musubi-tuner/dataset_cache/SynthSat-v1.0-warmup-stage1-256" debug_dataset: False image_directory: "/media/xzuyn/NVMe/Datasets/Images/000_SynthSatWarmup" image_jsonl_file: "None" fp_latent_window_size: 9 fp_1f_clean_indices: None fp_1f_target_index: None fp_1f_no_post: False flux_kontext_no_resize_control: False qwen_image_edit_no_resize_control: False qwen_image_edit_control_resolution: None INFO:musubi_tuner.dataset.image_video_dataset:bucket: (128, 512), count: 2 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (144, 448), count: 1 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (160, 400), count: 22 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (176, 368), count: 180 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (192, 336), count: 1197 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (208, 304), count: 6783 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (224, 288), count: 7671 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (240, 272), count: 554 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (256, 256), count: 1257 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (272, 240), count: 276 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (288, 224), count: 2652 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (304, 208), count: 5412 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (336, 192), count: 727 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (368, 176), count: 135 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (400, 160), count: 29 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (448, 144), count: 13 INFO:musubi_tuner.dataset.image_video_dataset:bucket: (512, 128), count: 4 INFO:musubi_tuner.dataset.image_video_dataset:total batches: 1691 INFO:musubi_tuner.hv_train_network:preparing accelerator accelerator device: cuda INFO:musubi_tuner.hv_train_network:DiT precision: torch.bfloat16, weight precision: None INFO:musubi_tuner.hv_train_network:Loading DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors INFO:musubi_tuner.qwen_image.qwen_image_model:Creating QwenImageTransformer2DModel INFO:musubi_tuner.qwen_image.qwen_image_model:Loading DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors, device=cpu INFO:musubi_tuner.utils.lora_utils:Loading model files: ['/media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors'] INFO:musubi_tuner.utils.lora_utils:Loading state dict with FP8 optimization. Dtype of weight: None, hook enabled: False Loading qwen_image_bf16.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1933/1933 [01:22<00:00, 23.42key/s] INFO:musubi_tuner.modules.fp8_optimization_utils:Number of optimized Linear layers: 840 INFO:musubi_tuner.modules.fp8_optimization_utils:Number of monkey-patched Linear layers: 840 INFO:musubi_tuner.qwen_image.qwen_image_model:Loaded DiT model from /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors, info= INFO:musubi_tuner.hv_train_network:enable swap 10 blocks to CPU from device: cuda QwenModel: Block swap enabled. Swapping 10 blocks out of 60 blocks. Supports backward: True import network module: networks.lora_qwen_image INFO:musubi_tuner.networks.lora:create LoRA network. base dim (rank): 16, alpha: 4.0 INFO:musubi_tuner.networks.lora:neuron dropout: p=0.125, rank dropout: p=0.0, module dropout: p=0.0 INFO:musubi_tuner.networks.lora:create LoRA for U-Net/DiT: 840 modules. INFO:musubi_tuner.networks.lora:enable LoRA for U-Net: 840 modules QwenModel: Gradient checkpointing enabled. Activation CPU offloading: True prepare optimizer, data loader etc. INFO:musubi_tuner.hv_train_network:use came_pytorch.CAME | {'weight_decay': 0.01, 'enable_8bit': True, 'enable_cautious': True, 'enable_stochastic_rounding': True} ==== CAME Modifications ==== - Stochastic Rounding enabled. - Cautious Masking enabled. - 8-bit enabled: block_size=2048, min_8bit_size=16384. ==== CAME Modifications ==== override steps. steps for 1 epochs is / 指定エポックまでのステップ数: 1691 INFO:musubi_tuner.hv_train_network:preparing fused backward pass stuff running training / 学習開始 num train items / 学習画像、動画数: 26915 num batches per epoch / 1epochのバッチ数: 1691 num epochs / epoch数: 1 batch size per device / バッチサイズ: 16 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 1691 INFO:musubi_tuner.hv_train_network:set DiT model name for metadata: /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/qwen_image_bf16.safetensors INFO:musubi_tuner.hv_train_network:set VAE model name for metadata: /media/xzuyn/NVMe/LClones/musubi-tuner/source_models/vae_diffusion_pytorch_model.safetensors ```