[2024-08-21 08:16:19,948] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 FLYWHEEL_USE_CUDA_DEVICES = 0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES = 0,1,2,3,4,5,6,7 local_rank = 0, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 0 Rank = 0 [2024-08-21 08:16:28,979] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. local_rank = 3, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 3 Rank = 3 [2024-08-21 08:16:30,042] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.33261704444885254 seconds  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [2024-08-21 08:16:30,558] [INFO] [comm.py:637:init_distributed] cdb=None local_rank = 4, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 4 Rank = 4 [2024-08-21 08:16:30,652] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) local_rank = 2, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 2 Rank = 2 [2024-08-21 08:16:30,721] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) local_rank = 1, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 1 Rank = 1 [2024-08-21 08:16:30,872] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) local_rank = 6, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 6 Rank = 6 [2024-08-21 08:16:30,967] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) local_rank = 5, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 5 Rank = 5 [2024-08-21 08:16:31,089] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) local_rank = 7, device_ranks = ['0', '1', '2', '3', '4', '5', '6', '7'], set device_rank = 7 Rank = 7 [2024-08-21 08:16:31,118] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.35923218727111816 seconds  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [2024-08-21 08:16:31,734] [INFO] [comm.py:637:init_distributed] cdb=None Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.3286874294281006 seconds Time to load cpu_adam op: 0.3859071731567383 seconds Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.32793354988098145 seconds Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.30348920822143555 seconds Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination Installed CUDA version 12.5 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination ninja: no work to do. Time to load cpu_adam op: 0.29896116256713867 seconds Time to load cpu_adam op: 0.3561563491821289 seconds [2024-08-21 08:16:33,112] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:33,591] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:33,592] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:33,592] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:33,592] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:33,592] [INFO] [comm.py:637:init_distributed] cdb=None [2024-08-21 08:16:49,358] [INFO] [partition_parameters.py:345:__exit__] finished initializing model - num_params = 435, num_elems = 33.74B Is prompt masked: True Is prompt masked: True Is prompt masked: True Is prompt masked: True Is prompt masked: True Is prompt masked: True Is prompt masked: True Is prompt masked: True Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 [2024-08-21 08:17:20,722] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.14.5, git-hash=unknown, git-branch=unknown Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 [2024-08-21 08:17:20,735] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2024-08-21 08:17:20,736] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer [2024-08-21 08:17:20,736] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 [2024-08-21 08:17:20,751] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam [2024-08-21 08:17:20,752] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type= [2024-08-21 08:17:20,752] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-08-21 08:17:20,752] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 Adam Optimizer #0 is created with AVX512 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1 [2024-08-21 08:17:20,985] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning [2024-08-21 08:17:20,986] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.98 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:20,986] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 112.62 GB, percent = 6.0% [2024-08-21 08:17:20,988] [INFO] [stage3.py:130:__init__] Reduce bucket size 67108864 [2024-08-21 08:17:20,988] [INFO] [stage3.py:131:__init__] Prefetch bucket size 60397977 [2024-08-21 08:17:21,151] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin] [2024-08-21 08:17:21,151] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:21,151] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 112.62 GB, percent = 6.0% Parameter Offload: Total persistent parameters: 794624 in 97 params [2024-08-21 08:17:21,435] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end] [2024-08-21 08:17:21,435] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:21,436] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 112.62 GB, percent = 6.0% [2024-08-21 08:17:21,583] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions [2024-08-21 08:17:21,584] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:21,584] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 112.63 GB, percent = 6.0% [2024-08-21 08:17:28,165] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 5 [2024-08-21 08:17:28,165] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:28,166] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 189.04 GB, percent = 10.1% [2024-08-21 08:17:28,416] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions [2024-08-21 08:17:28,417] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:28,417] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 195.22 GB, percent = 10.4% [2024-08-21 08:17:35,219] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions [2024-08-21 08:17:35,219] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:35,219] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 303.91 GB, percent = 16.3% [2024-08-21 08:17:35,354] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states [2024-08-21 08:17:35,354] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:35,355] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 308.47 GB, percent = 16.5% [2024-08-21 08:17:49,560] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states [2024-08-21 08:17:49,560] [INFO] [utils.py:782:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.98 GB Max_CA 1 GB [2024-08-21 08:17:49,560] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 442.67 GB, percent = 23.7% [2024-08-21 08:17:49,561] [INFO] [stage3.py:485:_setup_for_real_optimizer] optimizer state initialized [2024-08-21 08:17:56,875] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer [2024-08-21 08:17:56,876] [INFO] [utils.py:782:see_memory_usage] MA 0.13 GB Max_MA 1.1 GB CA 1.47 GB Max_CA 1 GB [2024-08-21 08:17:56,876] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 508.07 GB, percent = 27.2% [2024-08-21 08:17:56,876] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3 [2024-08-21 08:17:56,876] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None [2024-08-21 08:17:56,876] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2024-08-21 08:17:56,876] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[(0.9, 0.999)] [2024-08-21 08:17:56,877] [INFO] [config.py:997:print] DeepSpeedEngine configuration: [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] amp_enabled .................. False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] amp_params ................... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] bfloat16_enabled ............. True [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] bfloat16_immediate_grad_update False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] checkpoint_parallel_write_pipeline False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] checkpoint_tag_validation_enabled True [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] checkpoint_tag_validation_fail False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] comms_config ................. [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] communication_data_type ...... None [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] curriculum_enabled_legacy .... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] curriculum_params_legacy ..... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] data_efficiency_enabled ...... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] dataloader_drop_last ......... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] disable_allgather ............ False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] dump_state ................... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] dynamic_loss_scale_args ...... None [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_enabled ........... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_gas_boundary_resolution 1 [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_layer_num ......... 0 [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_max_iter .......... 100 [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_stability ......... 1e-06 [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_tol ............... 0.01 [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] eigenvalue_verbose ........... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] elasticity_enabled ........... False [2024-08-21 08:17:56,878] [INFO] [config.py:1001:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] fp16_auto_cast ............... None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] fp16_enabled ................. False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] fp16_master_weights_and_gradients False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] global_rank .................. 0 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] grad_accum_dtype ............. None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] gradient_accumulation_steps .. 1 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] gradient_clipping ............ 1.0 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] gradient_predivide_factor .... 1.0 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] graph_harvesting ............. False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] initial_dynamic_scale ........ 1 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] load_universal_checkpoint .... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] loss_scale ................... 1.0 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] memory_breakdown ............. False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] mics_hierarchial_params_gather False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] mics_shard_size .............. -1 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] optimizer_legacy_fusion ...... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] optimizer_name ............... None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] optimizer_params ............. None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] pld_enabled .................. False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] pld_params ................... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] prescale_gradients ........... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] scheduler_name ............... None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] scheduler_params ............. None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] seq_parallel_communication_data_type torch.float32 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] sparse_attention ............. None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] sparse_gradients_enabled ..... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] steps_per_print .............. inf [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] timers_config ................ enabled=True synchronized=True [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] train_batch_size ............. 128 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] train_micro_batch_size_per_gpu 16 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] use_data_before_expert_parallel_ False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] use_node_local_storage ....... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] wall_clock_breakdown ......... False [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] weight_quantization_config ... None [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] world_size ................... 8 [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] zero_allow_untested_optimizer True [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=67108864 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=60397977 param_persistence_threshold=81920 model_persistence_threshold=sys.maxsize max_live_parameters=6000000000 max_reuse_distance=6000000000 gather_16bit_weights_on_model_save=False use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] zero_enabled ................. True [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] zero_force_ds_cpu_optimizer .. True [2024-08-21 08:17:56,879] [INFO] [config.py:1001:print] zero_optimization_stage ...... 3 [2024-08-21 08:17:56,879] [INFO] [config.py:987:print_user_config] json = { "bf16": { "enabled": true }, "zero_optimization": { "stage": 3, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "offload_param": { "device": "cpu", "pin_memory": true }, "overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1.000000e+09, "reduce_bucket_size": 6.710886e+07, "stage3_prefetch_bucket_size": 6.039798e+07, "stage3_param_persistence_threshold": 8.192000e+04, "stage3_max_live_parameters": 6.000000e+09, "stage3_max_reuse_distance": 6.000000e+09, "stage3_gather_16bit_weights_on_model_save": false }, "gradient_accumulation_steps": 1, "gradient_clipping": 1.0, "steps_per_print": inf, "train_batch_size": 128, "train_micro_batch_size_per_gpu": 16, "wall_clock_breakdown": false, "fp16": { "enabled": false }, "zero_allow_untested_optimizer": true }  [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.  [WARNING]  async_io: please install the libaio-dev package with apt  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.